Opencontrail Create Virtual Gateway and Access VM using Private/Fixed IP

Create Virtual Gateway dynamically by sending thrift messages to the vrouter agent

Notes:
* Floating-ip and simple gateway are two orthogonal features.

* Simple gateway let's the virtual machines access underlay network. The access is provided by selective leaking of routes between vrf for virtual network and underlay network. No NAT is involved here.

* Floating-ip allows a VM to borrow IP address from another network. 1:1 NAT is done by vrouter for communication between VM and the borrowed virtual network.

Imp Links: 
https://github.com/Juniper/contrail-controller/wiki/Simple-Gateway

vRouter Command Line Utilities:
https://techwiki.juniper.net/Documentation/Contrail/Contrail_Controller_Feature_Guide/Optimizing_Contrail/vRouter_Command_Line_Utilities

https://github.com/Juniper/contrail-controller/blob/master/src/vnsw/agent/README

0)
* I have created a virtual network named "nw1" with subnet "172.168.56.0/24".
* Then, Created a VM in network "nw1" and the VM got the private IP 172.168.56.3.
* Now I want to Ping and SSH to the VM's private IP 172.168.56.3 from the Host machine's network 192.168.56.0/24. For this we have to do following steps.

1)
Find VGW Provision script
$find /usr -name provision_vgw_interface.py
/usr/share/contrail-utils/provision_vgw_interface.py

$ls /usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service/

2)
login as root user
$su - root

3)
Export path
#export PYTHONPATH=/usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service

4)
Run VGW Provison script
#python /opt/contrail/utils/provision_vgw_interface.py --oper create --interface vgw1 --subnets 172.168.56.0/24 --routes 192.168.56.0/24 10.0.2.0/24 8.8.8.0/24 9.9.9.0/24 --vrf default-domain:admin:nw1:nw1

Creating virtual-gateway ...
/usr/bin/vif --create vgw1 --mac 00:01:00:5e:00:00
ifconfig vgw1 up
route add -net 172.168.56.0/24 dev vgw1
Done creating virtual-gateway...

----Points----

a) Here "172.168.56.0/24" is the CIDR of the subnet created in the virtual network nw1.
b) Here "192.168.56.0/24" is the network of interface "vhost0" in the compute node. I want to reach compute node from the VM, So i have given it in --routes option.
c) Here "10.0.2.0/24" is the network of interface "eth0" in the compute node. I want to reach compute node from the VM, So i have given it in --routes option.
d) IP of the "vhost0" interface in the compute node is "192.168.56.102"
e) IP of the "eth0" interface in the compute node is "10.0.2.15"
f) Option --subnets specifies list of subnets defined in virtual-network nw1
g) Option --routes specifies routes in public-network injected into nw1. In example above, the Virtual machines in nw1 can access subnets 8.8.8.0/24, 9.9.9.0/24 and 192.168.56.0/24 in the Host machine's network (public network).

5)
Test the setup.
Ping and ssh work from compute node (192.168.56.102) to VM using private IP (172.168.56.3) of the VM.
Ping and ssh work from VM to compute node (192.168.56.102)

6)
Delete the changes.
* This settings will lost, If the vrouter agent restarts or if the compute node reboots.
* You can manualy delete this settings with following command
$python /opt/contrail/utils/provision_vgw_interface.py --oper delete --interface vgw1 --subnets 172.168.56.0/24


7)
Infos from compute node (192.168.56.102)

a)
Configuration "vgw1" gateway interface from compute node

$ ifconfig
vgw1      Link encap:Ethernet  HWaddr 00:01:00:5e:00:00 
          inet6 addr: fe80::201:ff:fe5e:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:195 errors:0 dropped:0 overruns:0 frame:0
          TX packets:240 errors:0 dropped:5 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:22854 (22.8 KB)  TX bytes:24922 (24.9 KB)

b)
Configuration "vgw1" gateway interface from compute node

$ vif --list
vif0/5      OS: vgw1
            Type:Gateway HWaddr:00:01:00:5e:00:00 IPaddr:0
            Vrf:1 Flags:L3 MTU:1514 Ref:2
            RX packets:240  bytes:24894 errors:0
            TX packets:195  bytes:22854 errors:0

c)
Configuration "eth0", "eth1" and "vhost0" interface from compute node

$ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 08:00:27:65:a3:4e 
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe65:a34e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:607 errors:0 dropped:0 overruns:0 frame:0
          TX packets:693 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:57547 (57.5 KB)  TX bytes:60247 (60.2 KB)

$ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:22:8e:16 
          inet6 addr: fe80::a00:27ff:fe22:8e16/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19205 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17793 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1529138 (1.5 MB)  TX bytes:2711067 (2.7 MB)

$ifconfig vhost0
vhost0    Link encap:Ethernet  HWaddr 08:00:27:22:8e:16 
          inet addr:192.168.56.102  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe22:8e16/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19497 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18022 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1557182 (1.5 MB)  TX bytes:2734972 (2.7 MB)

d)
Route table form compute node

$route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    100    0        0 eth0
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.3     0.0.0.0         255.255.255.255 UH    0      0        0 vhost0
169.254.0.4     0.0.0.0         255.255.255.255 UH    0      0        0 vhost0
172.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 vgw1 <==IMP
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 vhost0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

e)
Search for IP 172.168.56.x routes table of VRF 0

$rt --dump 0  | grep 172.168.56
Kernel IP routing table 0/1/unicast
Destination            PPL        Flags        Label        Nexthop
172.168.56.0/24          24           HT            -              2

f)
Search for IP 172.168.56.3 and 172.168.56.4 in routes table of VRF 1

$ rt --dump 1  | grep 172.168.56.3
Kernel IP routing table 0/1/unicast
Destination            PPL        Flags        Label        Nexthop
172.168.56.3/32          32            H            -              8


$ rt --dump 1  | grep 172.168.56.4
Kernel IP routing table 0/1/unicast
Destination            PPL        Flags        Label        Nexthop
172.168.56.4/32          32            H            -             15

g)
Find Next hub info

$ nh --get 2
Id:002  Type:Receieve  Fmly: AF_INET  Flags:Valid, Policy(R),   Rid:0  Ref_cnt:5
    Oif:1

$ nh --get 8
Id:008  Type:Encap     Fmly: AF_INET  Flags:Valid, Policy,   Rid:0  Ref_cnt:4
    EncapFmly:0806 Oif:3 Len:14 Data:02 72 e3 89 8f 11 00 00 5e 00 01 00 08 00

$ nh --get 15
Id:015  Type:Encap     Fmly: AF_INET  Flags:Valid, Policy,   Rid:0  Ref_cnt:4
    EncapFmly:0806 Oif:4 Len:14 Data:02 5b a2 39 ed e1 00 00 5e 00 01 00 08 00

h)
Capture Flow table entries
From compute node Run "ping 172.168.56.3" in one terminal and Run "flow -l" in another terminal to capture the flows.

#flow -l
Flow table

 Index              Source:Port           Destination:Port        Proto(V)
-------------------------------------------------------------------------
185912               10.0.2.15:30857        172.168.56.3:0        1 (1)
    (K(nh):17, Action:F, S(nh):17,  Statistics:63/5292 UdpSrcPort 54744)

215132            172.168.56.3:30857           10.0.2.15:0        1 (1)
    (K(nh):7, Action:F, S(nh):7,  Statistics:63/5292 UdpSrcPort 51077)


Notes:
* Here we ping from compute node to VM (172.168.56.3).
* In Flow table, source is "10.0.2.15" that is actually the IP of "eth0" interface in the compute node.

i)
Flow table entries, when you SSH to VM (172.168.56.3) from Compute node
and ping to compute node's "vhost0" (192.168.56.102) interface from the VM (172.168.56.3).

$flow -l
Flow table

 Index              Source:Port           Destination:Port        Proto(V)
-------------------------------------------------------------------------
 12744            172.168.56.4:22              10.0.2.15:53308    6 (1)
    (K(nh):13, Action:F, S(nh):13,  Statistics:137/17902 UdpSrcPort 61994)

 29704            172.168.56.4:14593      192.168.56.102:0        1 (1)
    (K(nh):13, Action:F, S(nh):13,  Statistics:56/4704 UdpSrcPort 52697)

164716            192.168.56.102:14593        172.168.56.4:0        1 (1)
    (K(nh):17, Action:F, S(nh):17,  Statistics:56/4704 UdpSrcPort 61906)

496808               10.0.2.15:53308        172.168.56.4:22       6 (1)
    (K(nh):17, Action:F, S(nh):17,  Statistics:177/13837 UdpSrcPort 64809)

8)
Debugging

a)
Find "tap" interface in the host which connect VM to host.
* Name of the "tap" interface is constructed by using first 10 characters in the ID of port attached to the VM.
* You can find the ports using neutron command "#neutron port-list"
* For example: ID of port is "5ba239ed-e143-446a-ad1a-794869ccfbfe",
then you can find a "tap" interface named "tap5ba239ed-e1" in compute host. You can use "ifconfig" command to find it.

b)
Capture packets from interfaces using tshark

Capture packets from tap interface "tap5ba239ed-e1"
#sudo tshark -i tap5ba239ed-e1 -f "icmp"

Capture packets from gateway interface "vgw1"
#sudo tshark -i vgw1 -f "icmp"

Capture packets from "vhost0" inet-interface
#sudo tshark -i vhost0 -f "icmp"


========Code==========


1)
#vim /usr/share/contrail-utils/provision_vgw_interface.py

import os
import sys
import argparse
import ConfigParser

from thrift.Thrift import TApplicationException, TException
from thrift.transport import TTransport, TSocket
from thrift.protocol import TBinaryProtocol, TProtocol
from netaddr.ip import IPNetwork
import InstanceService
import ttypes

from vnc_api.vnc_api import *


class ProvisionVgwInterface(object):

    def __init__(self, args_str=None):
        self._args = None
        if not args_str:
            args_str = ' '.join(sys.argv[1:])
        self._parse_args(args_str)

        # Connect to thrift service
        try:
            socket = TSocket.TSocket("localhost", 9090)
            transport = TTransport.TFramedTransport(socket)
            transport.open()
            protocol = TBinaryProtocol.TBinaryProtocol(transport)
            service = InstanceService.Client(protocol) <===IMP
        except TApplicationException:
            print "Error connecting to agent thrift service"
            return

        if self._args.oper == "create":
            print "Creating virtual-gateway ..."

            with open("/proc/sys/net/ipv4/ip_forward", "w") as file:
                file.write("1") <===IMP

            vif_command = '/usr/bin/vif --create ' + self._args.interface <===IMP
            vif_command += ' --mac 00:01:00:5e:00:00'
            self.execute_command(vif_command)

            ifconfig_command = 'ifconfig ' + self._args.interface + ' up' <===IMP
            self.execute_command(ifconfig_command)

            for subnet in self._args.subnets:
                route_command = 'route add -net ' + subnet <===IMP
                route_command += ' dev ' + self._args.interface
                self.execute_command(route_command)

            subnet_list = []
            for subnet in self._args.subnets: <===IMP
                net = IPNetwork(subnet)
                subnet_list.append(ttypes.Subnet(str(net.ip), net.prefixlen))

            route_list = []
            for subnet in self._args.routes: <===IMP
                net = IPNetwork(subnet)
                route_list.append(ttypes.Subnet(str(net.ip), net.prefixlen))

            gw = ttypes.VirtualGatewayRequest(self._args.interface, self._args.vrf,
                                              subnet_list, route_list)
            gw_list = [ gw ]
            try:
                service.AddVirtualGateway(gw_list) <===IMP
            except TApplicationException:
                print "Error: Error adding VGW interface"
                return
            print "Done creating virtual-gateway..."


2)
#vim /usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py

class Client(Iface):
  def __init__(self, iprot, oprot=None):
    self._iprot = self._oprot = iprot
    if oprot is not None:
      self._oprot = oprot
    self._seqid = 0

  def AddVirtualGateway(self, vgw_list): <===IMP
    """
    Parameters:
     - vgw_list
    """
    self.send_AddVirtualGateway(vgw_list)
    return self.recv_AddVirtualGateway()

  def send_AddVirtualGateway(self, vgw_list):
    self._oprot.writeMessageBegin('AddVirtualGateway', TMessageType.CALL, self._seqid)
    args = AddVirtualGateway_args()
    args.vgw_list = vgw_list
    args.write(self._oprot)
    self._oprot.writeMessageEnd()
    self._oprot.trans.flush()

  def recv_AddVirtualGateway(self, ):
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
    if mtype == TMessageType.EXCEPTION:
      x = TApplicationException()
      x.read(self._iprot)
      self._iprot.readMessageEnd()
      raise x
    result = AddVirtualGateway_result()
    result.read(self._iprot)
    self._iprot.readMessageEnd()
    if result.success is not None:
      return result.success
    raise TApplicationException(TApplicationException.MISSING_RESULT, "AddVirtualGateway failed: unknown result");