How to OpenContrail Check Compute node's peer with Controller Node

How to Contrail Check Compute node's peer with Controller Node

1)
Method-1:

Check Compute Agent's Xmpp Connection Status.
It should show an entry which contains IP of Controller node.

http://ip-of-compute-node:8085/Snh_AgentXmppConnectionStatusReq

Example:

Here IP of compute nodes 10.140.15.57 and 10.140.15.59
Here IP of controller node is 10.140.15.60

2)
Method-2:

Check Bgp Neighbor List.
It should contains entry for all Compute nodes.

http://ip-of-contrail-controller-node:8083/Snh_BgpNeighborReq

Example:

* Here IP of Compute nodes which are peered with controller is 10.140.15.57 and 10.140.15.59
* Here IP of controller node is 10.140.15.60
* I have created a VM in cp1 node and for that VM private IP assigned from "net_tempest" network and floating IP assigned from "public" network.So we can see routing instance of these two networks here (see the last picture).

3)
If you are not seeing entries in above urls, then check for "discovery server" IP in contrail-vrouter-agent.conf

$vim /etc/contrail/contrail-vrouter-agent.conf

# IP address of discovery server
server=10.140.15.78

4)

How to tweak OpenContrail supervisor

1)
Check all services run by supervisor
 
#ps -aux | grep /usr/bin/supervisord
root      1530  0.0  0.1  60712  5432 ?        S    13:44   0:01 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_analytics.conf
root      1532  0.0  0.1  60692  5312 ?        S    13:44   0:01 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_database.conf
root      1533  0.0  0.1  60924  7968 ?        S    13:44   0:01 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_support_service.conf
root      1535  0.0  0.1  60676  5088 ?        S    13:44   0:00 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_webui.conf
root      1536  0.0  0.1  60700  5268 ?        S    13:44   0:01 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_control.conf
root      1555  0.0  0.1  60684  5196 ?        S    13:45   0:00 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_vrouter.conf
root      1557  0.2  0.1  61416  7948 ?        S    13:45   0:04 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_openstack.conf
root      4287  0.0  0.2  60636  9556 ?        Ss   13:45   0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
saju      5874  0.0  0.0  11748   928 pts/2    S+   14:15   0:00 grep --color=auto /usr/bin/supervisord
root      6365  0.1  0.3  60720 14260 ?        S    13:45   0:01 /usr/bin/python /usr/bin/supervisord --nodaemon -c /etc/contrail/supervisord_config.conf

2)main contrail supervisord and its config

#sudo vim /etc/contrail/supervisord_config.conf
[include]
files = /etc/contrail/supervisord_config_files/*.ini

3)

Check the files in the "/etc/contrail/supervisord_config_files" directory. All configurations used by supervisor for "contrail config node" can find here.

#sudo ls /etc/contrail/supervisord_config_files
contrail-api.ini       contrail-device-manager.ini  contrail-nodemgr-config.ini  contrail-svc-monitor.ini
contrail-config.rules  contrail-discovery.ini        contrail-schema.ini         ifmap.ini

4)
Check the command used by supervisor to run "contrail-api" server.

#sudo vim /etc/contrail/supervisord_config_files/contrail-api.ini
[program:contrail-api]
command=/usr/bin/contrail-api --conf_file /etc/contrail/contrail-api.conf --conf_file /etc/contrail/contrail-keystone-auth.conf --listen_port 910%(process_num)01d --worker_id %(process_num)s
numprocs=1
process_name=%(process_num)s
redirect_stderr=true
stdout_logfile= /var/log/contrail/contrail-api-%(process_num)s-stdout.log
stderr_logfile=/dev/null
priority=440
autostart=true
killasgroup=true
stopsignal=KILL
exitcodes=0

5)
Check the rules used by supervisor for "contrail-config".

#sudo vim /etc/contrail/supervisord_config_files/contrail-config.rules
{ "Rules": [
    {"processname": "contrail-api", "process_state": "PROCESS_STATE_STOPPED", "action": "sudo service supervisor-config restart"},
    {"processname": "contrail-api", "process_state": "PROCESS_STATE_EXITED", "action": "sudo service supervisor-config restart"},
    {"processname": "contrail-api", "process_state": "PROCESS_STATE_FATAL", "action": "sudo service supervisor-config restart"},
    {"processname": "ifmap", "process_state": "PROCESS_STATE_STOPPED", "action": "sudo service supervisor-config restart"},
    {"processname": "ifmap", "process_state": "PROCESS_STATE_EXITED", "action": "sudo service supervisor-config restart"},
    {"processname": "ifmap", "process_state": "PROCESS_STATE_FATAL", "action": "sudo service supervisor-config restart"}
     ]
}

6)
Find the command which used to run "contrail-api" server and copy it.

#ps -aux | grep contrail-api
root      6493  0.8  1.3 324804 61400 ?        Sl   13:46   0:17 /usr/bin/python /usr/bin/contrail-api --conf_file /etc/contrail/contrail-api.conf --conf_file /etc/contrail/contrail-keystone-auth.conf --listen_port 9100 --worker_id 0

7)
Try neutron command "neutron net-list", this should work since we haven't made any change.

export OS_USERNAME=admin
export OS_PASSWORD=secret123
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://127.0.0.1:35357/v2.0

#neutron net-list

8)
Rename "contrail-api.ini" to "contrail-api.ini_bkp"

#sudo mv /etc/contrail/supervisord_config_files/contrail-api.ini /etc/contrail/supervisord_config_files/contrail-api.ini_bkp

9)
Comment first 3 lines which contains "processname": "contrail-api"

#sudo vim /etc/contrail/supervisord_config_files/contrail-config.rules
{ "Rules": [
#    {"processname": "contrail-api", "process_state": "PROCESS_STATE_STOPPED", "action": "sudo service supervisor-config restart"},
#    {"processname": "contrail-api", "process_state": "PROCESS_STATE_EXITED", "action": "sudo service supervisor-config restart"},
#    {"processname": "contrail-api", "process_state": "PROCESS_STATE_FATAL", "action": "sudo service supervisor-config restart"},
    {"processname": "ifmap", "process_state": "PROCESS_STATE_STOPPED", "action": "sudo service supervisor-config restart"},
    {"processname": "ifmap", "process_state": "PROCESS_STATE_EXITED", "action": "sudo service supervisor-config restart"},
    {"processname": "ifmap", "process_state": "PROCESS_STATE_FATAL", "action": "sudo service supervisor-config restart"}
     ]
}

10)
Restart supervisor-config

#sudo service supervisor-config restart
supervisor-config stop/waiting
supervisor-config start/running, process 21955

11)
Ensure that "contrail-api" server not running now

#ps -aux | grep contrail-api
saju     22333  0.0  0.0  11744   924 pts/2    S+   14:33   0:00 grep --color=auto contrail-api

12)
Try neutron command "neutron net-list", this would not work if "contrail-api" server is not running.

#neutron net-list
500-{u'NeutronError': {u'message': u'An unknown exception occurred.', u'type': u'NeutronException', u'detail': u''}}

13)
Manually Start "contrail-api" server

#sudo /usr/bin/python /usr/bin/contrail-api --conf_file /etc/contrail/contrail-api.conf --conf_file /etc/contrail/contrail-keystone-auth.conf --listen_port 9100 --worker_id 0

14)
Try neutron command "neutron net-list", this should work.

#neutron net-list
Works :)

Contrail add Static Routes via Host Routes

OpenContrail add Static Routes via Host Routes

1)
Static Routes
https://github.com/Juniper/contrail-controller/wiki/Static-Routes

* We can add static routes via contrail UI and Util script.
* Util script: https://github.com/Juniper/contrail-controller/blob/master/src/config/utils/provision_static_route.py
* Usinf this we can configure in a virtual-network with subnet 10.0.0.0/24, all the traffic originating from a VM in virtual-network, destined to subnet 11.1.1.0/24 can be configured to go via a service appliance (VM), using static route configured on a service VM interface.

* We can also add static route via host routes, for that we have to set "apply_subnet_host_routes=True" in  /etc/contrail/contrail-api.conf

2)
How to add static route via host routes ?
Site to Site VPN in OpenContrail:
https://github.com/numansiddique/snat_test/wiki/Site-to-Site-VPN-in-OpenContrail

3)
Patch which enable us to add static route via host routes.

https://review.opencontrail.org/#/c/1462/

https://github.com/Juniper/contrail-controller/commit/5031f59adcca7e238c1489fde2558521e2c2d81a#diff-1e8d43a6c800def7704681bd7b7827bd

4)
https://www.youtube.com/watch?v=PKJWFsyzBGw
http://fosshelp.blogspot.in/2015/06/contrail-add-static-routes-via-host_15.html

How to contrail sync keystone domain and project while starting api_server

How to opencontrail sync keystone domain and project while starting api_server
1)
Script to Start/Stop Contrail API Server
#python /usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py --conf_file /etc/contrail/contrail-api.conf --reset_config --rabbit_user guest --rabbit_password contrail123

2)
https://github.com/Juniper/contrail-controller/blob/master/src/config/api-server/vnc_cfg_api_server.py#L1038

/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py

contrail-controller/src/config/api-server/vnc_cfg_api_server.py

class VncApiServer(VncApiServerGen):

    def _db_init_entries(self):
        try:
            self._extension_mgrs['resync'].map(self._resync_domains_projects) <==1
        except Exception as e:
            pass

    def _resync_domains_projects(self, ext): <==2
        ext.obj.resync_domains_projects() <==3

3)
https://github.com/Juniper/contrail-controller/blob/master/src/config/vnc_openstack/vnc_openstack/__init__.py#L591
contrail-controller/src/config/vnc_openstack/vnc_openstack/__init__.py


class OpenstackDriver(vnc_plugin_base.Resync):

    def resync_domains_projects(self): <==4
        # add asynchronously
        self._main_glet = gevent.spawn(self._resync_domains_projects_forever)<==5
        self._worker_glets = []
        for x in range(self._resync_number_workers):
            self._worker_glets.append(gevent.spawn(self._resync_worker)) <==11

    def _resync_domains_projects_forever(self):<==6
         while True:
            retry = self._resync_all_domains() <==7
            retry = self._resync_all_projects()  <==8

    def _resync_all_projects(self):<==9
            ks_project_ids = set(
                [str(uuid.UUID(proj['id']))
                    for proj in self._ks_projects_list()])

            for ks_project_id in ks_project_ids - vnc_project_ids:
                self.q.put((Q_CREATE, 'project', ks_project_id))<==10

    def _resync_worker(self): <==12
        while True:
            oper, obj_type, obj_id = self.q.get()<==13
            try:
                if oper == Q_DELETE:
                    if obj_type == 'domain':
                        self._del_domain_from_vnc(obj_id)
                    elif obj_type == 'project':
                        self._del_project_from_vnc(obj_id)
                    else:
                        raise KeyError("An invalid obj_type was specified: %s",
                                        obj_type)
                elif oper == Q_CREATE:<==14
                    if obj_type == 'domain':
                        self._add_domain_to_vnc(obj_id)
                    elif obj_type == 'project':<==15
                        self._add_project_to_vnc(obj_id)<==16
                    else:
                        raise KeyError("An invalid obj_type was specified: %s",
                                        obj_type)
                else:
                    raise KeyError("An invalid operation was specified: %s", oper)
            except (ValueError, KeyError):
                # For an unpack error or and invalid kind.
                self.log_exception()
            finally:
                self.q.task_done()

4)
Notes
>>> import Queue
>>> q = Queue.Queue(maxsize=1000)
>>> q.put((1,2,3))
>>> q.put((2,2,3))
>>> q.put((3,2,3))
>>> q.put((4,2,3))
>>>
>>> q.get()
(1, 2, 3)
>>> q.get()
(2, 2, 3)
>>> q.get()
(3, 2, 3)
>>> q.get()
(4, 2, 3)
>>>
>>> q.queue

How to debug neutron command flow in contrail vnc_openstack

How to debug neutron command opencontrail vnc_openstack

1)
Add breakpoint or print statement
#sudo vim /usr/local/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py

def plugin_http_post_network(self):
    print "=======", context['operation']
    import pdb; pdb.set_trace()


2)
Restart API Server
#python /usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py --conf_file /etc/contrail/contrail-api.conf --reset_config --rabbit_user guest --rabbit_password contrail123 & echo $! >/home/dev-net/contrail-installer/status/contrail/apiSrv.pid; fg || echo "apiSrv failed to start" | tee "/home/dev-net/contrail-installer/status/contrail/apiSrv.failure"

3)
Fire neutron commands
#neutron net-list
#neutron net-create nw1

Contrail with OpenStack: Flow Diagram of Create a VM with Port in vRouter