How to setup cassandra zookeeper and rabbitmq cluster

1)
Nodes:
ct1-multinode-cont : 11.120.15.59
ct2-multinode-cont : 11.120.15.60
ct3-multinode-cont : 11.120.15.61

2)
zookeeper cluster:

a)
config:

$vim /etc/zookeeper/conf/zoo.cfg

server.59=11.120.15.59:2888:3888
server.60=11.120.15.60:2888:3888
server.61=11.120.15.61:2888:3888

b)
Check cluster status

b1)
$ /usr/share/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Mode: follower

b2)
$ /usr/share/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Mode: leader
 
b3)
From  11.120.15.61



$ /usr/share/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Mode: follower

c)
docs:
http://myjeeva.com/zookeeper-cluster-setup.html

http://hanishblogger.blogspot.in/2013/11/zookeeper-setup-in-distributed-mode.html

3)
cassandra cluster:

a)
config:

$/etc/cassandra/cassandra.yaml
cluster_name: 'contrail'

seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: 11.120.15.59,11.120.15.60,11.120.15.61
b)
Check cluster status  
 
# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens  Owns (effective)  Host ID                               Rack
UN  11.120.15.59  75.4 KB    256     100.0%            a07836a2-8c56-4da7-803d-4389f87a4854  rack1
UN  11.120.15.61  256.7 KB   256     100.0%            00c51f6a-2d08-4b27-81f9-b9754b6b0b66  rack1
UN  11.120.15.60  90.74 KB   256     100.0%            d89d9cd8-d154-43d9-97ec-6b827c9a7ced  rack1


c)
docs:
https://docs.datastax.com/en/cassandra/2.0/cassandra/initialize/initializeSingleDS.html

4)
rabbitmq cluster config:

a)
config:
http://blog.hemantthorat.com/setup-rabbitmq-clusters-on-ubuntu/#.Vm-7a5PoEc0

b)
Check cluster status  
 
$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@ct1-multinode-cont' ...
[{nodes,[{disc,['rabbit@ct1-multinode-cont','rabbit@ct2-multinode-cont',
                'rabbit@ct3-multinode-cont']}]},
 {running_nodes,['rabbit@ct2-multinode-cont','rabbit@ct3-multinode-cont',
                 'rabbit@ct1-multinode-cont']},
 {cluster_name,<<"rabbit@localhost">>},
 {partitions,[]}]

How to Apache zookeeper configure session timeout

1)
Find location of zkServer.sh script
#find / -name zkServer.sh
OR
#locate zkServer.sh

2)
#cd cd /opt/stack/contrail/third_party/zookeeper-3.4.6/bin/

saju@ubuntu:/opt/stack/contrail/third_party/zookeeper-3.4.6/bin$ ls
README.txt  zkCleanup.sh  zkCli.cmd  zkCli.sh  zkEnv.cmd  zkEnv.sh  zkServer.cmd  zkServer.sh

3)
Find zoo.cfg

saju@ubuntu:/opt/stack/contrail/third_party/zookeeper-3.4.6/bin$ ./zkServer.sh --help
JMX enabled by default
Using config: /opt/stack/contrail/third_party/zookeeper-3.4.6/bin/../conf/zoo.cfg
Usage: ./zkServer.sh {start|start-foreground|stop|restart|status|upgrade|print-cmd}

4)
Edit "zoo.cfg" and change session timeout
saju@ubuntu:/opt/stack/contrail/third_party/zookeeper-3.4.6/bin$ sudo vim ../conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=20000000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

5)
Stop zookeeper
#cd /opt/stack/contrail/third_party/zookeeper-3.4.6; ./bin/zkServer.sh stop & echo $! >/home/saju/contrail-installer/status/contrail/zk.pid; fg || echo "zk failed to start" | tee "/home/saju/contrail-installer/status/contrail/zk.failure

6)
Start zookeeper
#cd /opt/stack/contrail/third_party/zookeeper-3.4.6; ./bin/zkServer.sh start & echo $! >/home/saju/contrail-installer/status/contrail/zk.pid; fg || echo "zk failed to start" | tee "/home/saju/contrail-installer/status/contrail/zk.failure

OpenContrail Uses of ZooKeeper

In OpenContrail ZooKeeper is currently used for

1) master-election

* For schema-transformer: https://github.com/Juniper/contrail-controller/blob/master/src/config/schema-transformer/to_bgp.py#L3713

* For device-manager: https://github.com/Juniper/contrail-controller/blob/master/src/config/device-manager/device_manager/device_manager.py#L469

2) index-allocation

*IndexAllocator: https://github.com/Juniper/contrail-controller/blob/master/src/config/common/zkclient.py#L22

* In schema-transformer: https://github.com/Juniper/contrail-controller/blob/master/src/config/schema-transformer/to_bgp.py#L2526


* index-allocation is used for things like (virtual-network id for every VN, security-group id for every SG).There is a vn-id for every VN. this vn-id is sent across in bgp messages between control-node.

3) Arbitration reasons

* To convert fq-name to uuid when uuid not specified in requests.
* If there is 2 concurrent requests, api-server allocates/reserves a uuid in zk, that way only one create will succeed.

* create_fq_name_to_uuid_mapping: https://github.com/Juniper/contrail-controller/blob/master/src/config/api-server/vnc_cfg_ifmap.py#L1129

* ZookeeperClient: https://github.com/Juniper/contrail-controller/blob/master/src/config/common/zkclient.py#L181

How to use Apache ZooKeeper zkCli Command Line Interface

a)
Find zookeeper Cli
#locate zkCli

b)
Find process ID of zookeeper
#sudo ps -aux | grep zoo

 

c)
Find port of zookeeper using process ID
#sudo netstat -tuplen | grep 3876
tcp6       0      0 :::2181                 :::*                    LISTEN      1000       14046       3876/java      
tcp6       0      0 :::33040                :::*                    LISTEN      1000       13626       3876/java

* Two instance (port 2181 and 33040) of zookeeper service are running

d)
Connect to zookeeper server
#cd /opt/stack/contrail/third_party/zookeeper-3.4.6/bin/
#./zkCli.sh -server 192.168.56.102:2181

* http://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html

e)
Get zookeeper Cli commands
[zk: 192.168.56.102:2181(CONNECTED) 0] help

f)
List path
[zk: 192.168.56.102:2181(CONNECTED) 13] ls /[svc-monitor, id, schema-transformer, fq-name-to-uuid, api-server, zookeeper]

* schema-transformer creates "/id"

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /fq-name-to-uuid

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /id

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /schema-transformer

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /svc-monitor

g)
Get data from path

* Get ID of default IPAM "default-network-ipam" in the "default-project" project.
[zk: 192.168.56.102:2181(CONNECTED) 36] get /fq-name-to-uuid/network_ipam:default-domain:default-project:default-network-ipam
0bee2fa4-766d-438f-9991-0fe4b34577b8

* Get ID of IPAM "ipam333" in the "admin" project.
[zk: 192.168.56.102:2181(CONNECTED) 37] get /fq-name-to-uuid/network_ipam:default-domain:admin:ipam333
1d54fc41-8697-4747-ab1f-f85ad53ac59d

h)
Set data

* Set ID of IPAM "ipam333" in the "admin" project.
[zk: 192.168.56.102:2181(CONNECTED) 44] set /fq-name-to-uuid/network_ipam:default-domain:admin:ipam333 1d54fc41-8697-4747-ab1f-f85ad53ac599

i)
Delete Path

i1)
[zk: localhost:2181(CONNECTED) 12] get /fq-name-to-uuid/global_system_config:default-global-system-config
a9f53855-477f-4509-bd40-9297470b7a0b
cZxid = 0xa
ctime = Thu Oct 15 05:42:46 UTC 2015
mZxid = 0xa
mtime = Thu Oct 15 05:42:46 UTC 2015
pZxid = 0xa
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 36
numChildren = 0

i2)
[zk: localhost:2181(CONNECTED) 12] rmr /fq-name-to-uuid/global_system_config:default-global-system-config

i3)
[zk: localhost:2181(CONNECTED) 13] get /fq-name-to-uuid/global_system_config:default-global-system-config
Node does not exist: /fq-name-to-uuid/global_system_config:default-global-system-config

OpenContrail ZooKeeper Client

a)
Find zookeeper Cli
#locate zkCli

b)
Find process ID of zookeeper
#sudo ps -aux | grep zoo

c)
Find port of zookeeper using process ID
#sudo netstat -tuplen | grep 3876
tcp6       0      0 :::2181                 :::*                    LISTEN      1000       14046       3876/java      
tcp6       0      0 :::33040                :::*                    LISTEN      1000       13626       3876/java

* Two instance (port 2181 and 33040) of zookeeper service are running

d)
Connect to zookeeper server
#cd /opt/stack/contrail/third_party/zookeeper-3.4.6/bin/
#./zkCli.sh -server 192.168.56.102:2181

* http://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html

e)
Get zookeeper Cli commands
[zk: 192.168.56.102:2181(CONNECTED) 0] help

f)
List path
[zk: 192.168.56.102:2181(CONNECTED) 13] ls /[svc-monitor, id, schema-transformer, fq-name-to-uuid, api-server, zookeeper]

* schema-transformer creates "/id"

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /fq-name-to-uuid

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /id

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /schema-transformer

[zk: 192.168.56.102:2181(CONNECTED) 13] ls /svc-monitor

g)
Get data from path

* Get ID of default IPAM "default-network-ipam" in the "default-project" project.
[zk: 192.168.56.102:2181(CONNECTED) 36] get /fq-name-to-uuid/network_ipam:default-domain:default-project:default-network-ipam
0bee2fa4-766d-438f-9991-0fe4b34577b8

* Get ID of IPAM "ipam333" in the "admin" project.
[zk: 192.168.56.102:2181(CONNECTED) 37] get /fq-name-to-uuid/network_ipam:default-domain:admin:ipam333
1d54fc41-8697-4747-ab1f-f85ad53ac59d

h)
Set data

* Set ID of IPAM "ipam333" in the "admin" project.
[zk: 192.168.56.102:2181(CONNECTED) 44] set /fq-name-to-uuid/network_ipam:default-domain:admin:ipam333 1d54fc41-8697-4747-ab1f-f85ad53ac599