IT&&LINUX&&Config: apache2(ubuntu)+mod_jk+tomcat8_2nodes+memcached(2nodes) session clustering

This article shows my own first succssful configuration of apache2(ubuntu)+mod_jk+tomcat8_2nodes+memcached(2nodes) session clustering using memcached-session-manager.
Although I did something many things in some project, I did not remember all architecture and wanted to revive those configurations.

Cloud environment, such as aws amazon, does not support multicast networks, so using memcached would be wiser decision.

Tomcat8 installed by using puppet config automation framework on each nodes. One node is ubuntu14.04 server, the other node is centos7.
And two memcached nodes uses ubuntu14.04 server.

As in jsp container, OS kernel seems doesn't matter as long as those are using same jdk version. But in real production environment, using same kernel version and same distro would be wiser dicison.
I used two different distros as a just experiment,for fun and to know centos7 architecture.

First, it is a good choice install memcached nodes.
In ubuntu server, apt-get install memcached command is enough, but should do some configuration to accept requests from tomcat containers.

I use virtualbox and vagrant.
Vagrantfile configuration is as belows. You could find vagrant information from my blog or google.com.
You should do first download boxes or make your own boxes.(Please refer to vagrantup.com)

"vagrant up memcached memcached2" command will boot two memcached servers.
On each node, I had to edit /etc/memcached.conf file to accept tomcat requests.

#vagrant multi servers configuration. I am not perfect on vagrant things, so if you know something more than me, please let me know.
# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

config.vm.define "puppet" do |ps|
   ps.vm.box = "puppet"
   ps.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   ps.vm.provision "shell", inline: "echo now time to executing shell"
   ps.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   ps.vm.network "private_network",ip:"10.0.0.5",
        virtualbox__nat: false
   ps.vm.host_name = "puppet"
      ps.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id,"--memory","1024"]
      end
   end

config.vm.define "ts" do |ts|
   ts.vm.box = "ts"
   ts.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   ts.vm.provision "shell", inline: "echo now time to executing shell"
   ts.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   ts.vm.network "private_network", ip:"10.0.0.7"
   ts.vm.host_name = "ts"
     ts.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "1024"]
     end
end

config.vm.define "ts1" do |ts1|
   ts1.vm.box = "ts1"
   ts1.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   ts1.vm.provision "shell", inline: "echo now time to executing shell"
   ts1.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   ts1.vm.network "private_network", ip:"10.0.0.9"
   ts1.vm.host_name = "ts1"
     ts1.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "512"]
     end
end

config.vm.define "centos7" do |ct7|
   ct7.vm.box = "ct7"
   ct7.vm.box_url = "file:///home/whatsup/vg/centos70.box"
   ct7.vm.provision "shell", inline: "echo now time to executing shell"
   ct7.vm.provision "shell", inline: "echo timezone config; timedatectl set-timezone Asia/Seoul"
   ct7.vm.network "private_network", ip:"10.0.0.11"
   ct7.vm.host_name = "ct7"
     ct7.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "512"]
     end
end

config.vm.define "memcached" do |mc|
   mc.vm.box = "mc"
   mc.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   mc.vm.provision "shell", inline: "echo now time to executing shell"
   mc.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   mc.vm.provision "shell", inline: "apt-get -y install memcached"
   mc.vm.network "private_network", ip:"10.0.0.13"
   mc.vm.host_name = "mc"
     mc.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "128"]
     end
end

config.vm.define "memcached2" do |mc2|
   mc2.vm.box = "mc2"
   mc2.vm.box_url = "file:///home/whatsup/vg/ubuntu14.box"
   mc2.vm.provision "shell", inline: "echo now time to executing shell"
   mc2.vm.provision "shell", inline: "echo timezone config; echo 'Asia/Seoul' > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata"
   mc2.vm.provision "shell", inline: "apt-get -y install memcached"
   mc2.vm.network "private_network", ip:"10.0.0.15"
   mc2.vm.host_name = "mc2"
     mc2.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "128"]
     end
end
end

"vagrant up memcached memcached2" command will boot two memcached servers.
On each node, I had to edit /etc/memcached.conf file to accept tomcat requests.

whatsup@whatsup-To-be-filled-by-O-E-M ~/vg $ vagrant up memcached memcached2
Bringing machine 'memcached' up with 'virtualbox' provider...
Bringing machine 'memcached2' up with 'virtualbox' provider...
==> memcached: Clearing any previously set forwarded ports...
==> memcached: Fixed port collision for 22 => 2222. Now on port 2203.
==> memcached: Clearing any previously set network interfaces...
==> memcached: Preparing network interfaces based on configuration...
    memcached: Adapter 1: nat
    memcached: Adapter 2: hostonly

vagrant@mc:~$ vi /etc/memcached.conf

# memcached default config file
# 2003 - Jay Bonci <jaybonci@debian.org>
# This configuration file is read by the start-memcached script provided as
# part of the Debian GNU/Linux distribution.

# Run memcached as a daemon. This command is implied, and is not needed for the
# daemon to run. See the README.Debian that comes with this package for more
# information.
-d

# Log memcached's output to /var/log/memcached
logfile /var/log/memcached.log

# Be verbose
# -v

# Be even more verbose (print client commands as well)
# -vv

# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default
# Note that the daemon will grow to this size, but does not start out holding this much
# memory
-m 64

# Default connection port is 11211
-p 11211

# Run the daemon as root. The start-memcached will default to running as root if no
# -u command is present in this config file
-u memcache

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
#-l 127.0.0.1
# to accept from tomcat instances.
-l 10.0.0.13

# Limit the number of simultaneous incoming connections. The daemon default is 1024
# -c 1024

# Lock down all paged memory. Consult with the README and homepage before you do this
# -k

# Return error when memory is exhausted (rather than removing items)
# -M

# Maximize core file limit
# -r

vagrant@mc2:~$ vi /etc/memcached.conf

# memcached default config file
# 2003 - Jay Bonci <jaybonci@debian.org>
# This configuration file is read by the start-memcached script provided as
# part of the Debian GNU/Linux distribution.

# Run memcached as a daemon. This command is implied, and is not needed for the
# daemon to run. See the README.Debian that comes with this package for more
# information.
-d

# Log memcached's output to /var/log/memcached
logfile /var/log/memcached.log

# Be verbose
# -v

# Be even more verbose (print client commands as well)
# -vv

# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default
# Note that the daemon will grow to this size, but does not start out holding this much
# memory
-m 64

# Default connection port is 11211
-p 11211

# Run the daemon as root. The start-memcached will default to running as root if no
# -u command is present in this config file
-u memcache

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
#-l 127.0.0.1
-l 10.0.0.15

# Limit the number of simultaneous incoming connections. The daemon default is 1024
# -c 1024

# Lock down all paged memory. Consult with the README and homepage before you do this
# -k

# Return error when memory is exhausted (rather than removing items)
# -M

# Maximize core file limit
# -r

**Restart memcached with "service restart memcached"**

**Second: tomcat configuration based on https://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Configure_memcached-session-manager_as_%3CContext%3E_Manager**

#Preparation.
#You should download these files to $TOMCAT_LIB DIRECTORY. My tomcat8 lib directory is /usr/local/tomcat8/lib.
#So using below jar files into /usr/local/tomcat8/lib will be a good start.

wget http://repo1.maven.org/maven2/de/javakaffee/msm/memcached-session-manager-tc8/1.8.3/memcached-session-manager-tc8-1.8.3.jar
wget http://repo1.maven.org/maven2/de/javakaffee/msm/memcached-session-manager/1.8.3/memcached-session-manager-1.8.3.jar
wget http://repo1.maven.org/maven2/net/spy/spymemcached/2.11.1/spymemcached-2.11.1.jar
wget http://repo1.maven.org/maven2/de/javakaffee/msm/msm-kryo-serializer/1.8.3/msm-kryo-serializer-1.8.3.jar
wget http://repo1.maven.org/maven2/de/javakaffee/kryo-serializers/0.11/kryo-serializers-0.11.jar
wget http://repo1.maven.org/maven2/com/googlecode/kryo/1.04/kryo-1.04.jar
wget http://repo1.maven.org/maven2/com/googlecode/minlog/1.2/minlog-1.2.jar
wget http://repo1.maven.org/maven2/com/googlecode/reflectasm/1.01/reflectasm-1.01.jar
wget http://repo1.maven.org/maven2/asm/asm/3.2/asm-3.2.jar

# jsp test file from http://blogs.agilefaqs.com/wp-content/uploads/2009/11/session.jsp to examples/jsp directory.

First ts node configuration of context.xml.
root@ts:/usr/local/tomcat8# cat /usr/local/tomcat8/conf/context.xml
<?xml version='1.0' encoding='utf-8'?>


<Context>

    
    
    <WatchedResource>WEB-INF/web.xml</WatchedResource>
    <WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>

    
    

    
    
    <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
             memcachedNodes="n1:10.0.0.13:11211,n2:10.0.0.15:11211"
             failoverNodes="n2"
             requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
             transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
     />
             
             
      
     


</Context>

Centos7 ct node configuration is almost like ts(ubuntu) node except "failoverNodes" setting.
<?xml version='1.0' encoding='utf-8'?>


<Context>

    
    
    <WatchedResource>WEB-INF/web.xml</WatchedResource>
    <WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>

    
    

    
    
    <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
             memcachedNodes="n1:10.0.0.13:11211,n2:10.0.0.15:11211"
             failoverNodes="n1"
             requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
             transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
     />
             
             
     
     



</Context>

**Restart tomcat8 with "service tomcat8 restart on each node. It shouldn't give any errors.Check the logs/catalina.out messages if you get ones**

**Lastly configure apache2 web server. I did compile apache2.2x version on project, but using deb package is more convenient if there is no special request from clients**
Commands, such as "apt-get -y install libapache2-mod-proxy-html libxml2-dev"
* enabling modules for mod_jk.
root@ts1:~# cat a2enmod.sh
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_ajp
a2enmod rewrite
a2enmod deflate
a2enmod headers
a2enmod proxy_balancer
a2enmod proxy_connect
a2enmod proxy_html

* Execute above shell with "bash a2enmod.sh" was enough *

** mod-jk config file edit **
root@ts1:~# cat /etc/libapache2-mod-jk/httpd-jk.conf
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Configuration Example for mod_jk
# used in combination with Apache 2.2.x

<IfModule jk_module>

    # We need a workers file exactly once
    # and in the global server
    JkWorkersFile /etc/apache2/workers.properties

    # Our JK error log
    # You can (and should) use rotatelogs here
    JkLogFile /var/log/apache2/mod_jk.log

    # Our JK log level (trace,debug,info,warn,error)
    JkLogLevel info

    # Our JK shared memory file
    JkShmFile /var/log/apache2/jk-runtime-status

    # Define a new log format you can use in any CustomLog in order
    # to add mod_jk specific information to your access log.
    # LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{Cookie}i\" \"%{Set-Cookie}o\" %{pid}P %{tid}P %{JK_LB_FIRST_NAME}n %{JK_LB_LAST_NAME}n ACC %{JK_LB_LAST_ACCESSED}n ERR %{JK_LB_LAST_ERRORS}n BSY %{JK_LB_LAST_BUSY}n %{JK_LB_LAST_STATE}n %D" extended_jk

    # This option will reject all requests, which contain an
    # encoded percent sign (%25) or backslash (%5C) in the URL
    # If you are sure, that your webapp doesn't use such
    # URLs, enable the option to prevent double encoding attacks.
    # Since: 1.2.24
    # JkOptions +RejectUnsafeURI

    # After setting JkStripSession to "On", mod_jk will
    # strip all ";jsessionid=..." from request URLs it
    # does *not* forward to a backend.
    # This is useful, if all links in a webapp use
    # URLencoded session IDs and parts of the static
    # content should be delivered directly by Apache.
    # Of course you can also do it with mod_rewrite.
    # Since: 1.2.21
    # JkStripSession On

    # Start a separate thread for internal tasks like
    # idle connection probing, connection pool resizing
    # and load value decay.
    # Run these tasks every JkWatchdogInterval seconds.
    # Since: 1.2.27
    JkWatchdogInterval 60

    # Configure access to jk-status and jk-manager
    # If you want to make this available in a virtual host,
    # either move this block into the virtual host
    # or copy it logically there by including "JkMountCopy On"
    # in the virtual host.
    # Add an appropriate authentication method here!

    <Location /jk-status>
        # Inside Location we can omit the URL in JkMount
        JkMount jk-status
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1
        Allow from 10.0.0.1
    </Location>
    <Location /jk-manager>
        # Inside Location we can omit the URL in JkMount
        JkMount jk-manager
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1
        Allow from 10.0.0.1
    </Location>

    # If you want to put all mounts into an external file
    # that gets reloaded automatically after changes
    # (with a default latency of 1 minute),
    # you can define the name of the file here.
    # JkMountFile conf/extra/uriworkermap.properties

    # Example for Mounting a context to the worker "balancer"
    # The URL syntax "a|b" instantiates two mounts at once,
    # the first one is "a", the second one is "ab".
    # JkMount /myapp|/* balancer

    # Example for UnMounting requests for all workers
    # using a simple URL pattern
    # Since: 1.2.26
    # JkUnMount /myapp/static/* *

    # Example for UnMounting requests for a named worker
    # JkUnMount /myapp/images/* balancer

    # Example for UnMounting requests using regexps
    # SetEnvIf REQUEST_URI "\.(htm|html|css|gif|jpg|js)$" no-jk

    # Example for setting a reply timeout depending on the request URL
    # Since: 1.2.27
    # SetEnvIf Request_URI "/transactions/" JK_REPLY_TIMEOUT=600000

    # Example for disabling reply timeouts for certain request URLs
    # Since: 1.2.27
    # SetEnvIf Request_URI "/reports/" JK_REPLY_TIMEOUT=0

    # IMPORTANT: Mounts and virtual hosts
    # If you are using VirtualHost elements, you
    # - can put mounts only used in some virtual host into its VirtualHost element
    # - can copy all global mounts to it using "JkMountCopy On" inside the VirtualHost
    # - can copy all global mounts to all virtual hosts by putting
    #   "JkMountCopy All" into the global server
    # Since: 1.2.26

</IfModule>

** worker.properties file **
root@ts1:~# cat /etc/apache2/workers.properties
# The load balancer worker balance1 will distribute
# load to the members worker1 and worker2
# Define list of workers that will be used
# for mapping requests
worker.list=loadbalancer,status

# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=10.0.0.7
worker.node1.type=ajp13
worker.node1.ping_mode=A
worker.node1.lbfactor=1

# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8009
worker.node2.host=10.0.0.11
worker.node2.type=ajp13
worker.node2.ping_mode=A
worker.node2.lbfactor=1

# Load-balancing behavior
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=True

# Status worker for managing load balancer
worker.status.type=status

#apache2 configuration of virtualhost.
root@ts1:~# cat /etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:80>

    # The ServerName directive sets the request scheme, hostname and port that
    # the server uses to identify itself. This is used when creating
    # redirection URLs. In the context of virtual hosts, the ServerName
    # specifies what hostname must appear in the request's Host: header to
    # match this virtual host. For the default virtual host (this file) this
    # value is not decisive as it is used as a last resort host regardless.
    # However, you must set it for any further virtual host explicitly.
    #ServerName www.example.com

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html

    # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
    # error, crit, alert, emerg.
    # It is also possible to configure the loglevel for particular
    # modules, e.g.
    #LogLevel info ssl:warn

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    # For most configuration files from conf-available/, which are
    # enabled or disabled at a global level, it is possible to
    # include a line for only one particular virtual host. For example the
    # following line enables the CGI configuration for this host only
    # after it has been globally disabled with "a2disconf".
    #Include conf-available/serve-cgi-bin.conf
        JkMount / loadbalancer
        JkMount /* loadbalancer
        JkMount /jk-status status


</VirtualHost>

To sum up,
Web apache mod_jk loadbalancer is: ts1-10.0.0.9
Tomcat8 nodes are : ts-10.0.0.7, ct7-10.0.0.11
Memcached nodes are: mc-10.0.0.13, mc2-10.0.0.15

* "ts" tomcat sticky session will be directed to 10.0.0.13:11211(mc). If mc is not usable then, it will be saved 10.0.0.15(mc2).
See above tomcat configuraton. You will be easily get it. Or check https://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Configure_memcached-session-manager_as_%3CContext%3E_Manager

*Testing using web server. session id should be same all the time*

*Todo
How to get memcached information and administering:http://lzone.de/cheat-sheet/memcached explains well.
Automation with puppet? or chef?

*If anything wrong with this article, please let me know. Thanks.*

IT&&LINUX&&Config

Wednesday, April 15, 2015

apache2(ubuntu)+mod_jk+tomcat8_2nodes+memcached(2nodes) session clustering

No comments:

Post a Comment