Create Virtual Gateway dynamically by sending thrift messages to the vrouter agent
Notes:
* Floating-ip and simple gateway are two orthogonal features.
* Simple gateway let's the virtual machines access underlay network. The access is provided by selective leaking of routes between vrf for virtual network and underlay network. No NAT is involved here.
* Floating-ip allows a VM to borrow IP address from another network. 1:1 NAT is done by vrouter for communication between VM and the borrowed virtual network.
Imp Links:
https://github.com/Juniper/contrail-controller/wiki/Simple-Gateway
vRouter Command Line Utilities:
https://techwiki.juniper.net/Documentation/Contrail/Contrail_Controller_Feature_Guide/Optimizing_Contrail/vRouter_Command_Line_Utilities
https://github.com/Juniper/contrail-controller/blob/master/src/vnsw/agent/README
0)
* I have created a virtual network named "nw1" with subnet "172.168.56.0/24".
* Then, Created a VM in network "nw1" and the VM got the private IP 172.168.56.3.
* Now I want to Ping and SSH to the VM's private IP 172.168.56.3 from the Host machine's network 192.168.56.0/24. For this we have to do following steps.
1)
Find VGW Provision script
$find /usr -name provision_vgw_interface.py
/usr/share/contrail-utils/provision_vgw_interface.py
$ls /usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service/
2)
login as root user
$su - root
3)
Export path
#export PYTHONPATH=/usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service
4)
Run VGW Provison script
#python /opt/contrail/utils/provision_vgw_interface.py --oper create --interface vgw1 --subnets 172.168.56.0/24 --routes 192.168.56.0/24 10.0.2.0/24 8.8.8.0/24 9.9.9.0/24 --vrf default-domain:admin:nw1:nw1
Creating virtual-gateway ...
/usr/bin/vif --create vgw1 --mac 00:01:00:5e:00:00
ifconfig vgw1 up
route add -net 172.168.56.0/24 dev vgw1
Done creating virtual-gateway...
----Points----
a) Here "172.168.56.0/24" is the CIDR of the subnet created in the
virtual network nw1.
b) Here "192.168.56.0/24" is the network of interface "vhost0" in the compute node. I want to reach compute node from the VM, So i have given it in --routes option.
c) Here "10.0.2.0/24" is the network of interface "eth0" in the compute node. I want to reach compute node from the VM, So i have given it in --routes option.
d) IP of the "vhost0" interface in the compute node is "192.168.56.102"
e) IP of the "eth0" interface in the compute node is "10.0.2.15"
f) Option --subnets specifies list of subnets defined in virtual-network nw1
g) Option --routes specifies routes in public-network injected into nw1. In example above, the Virtual machines in nw1 can access subnets 8.8.8.0/24, 9.9.9.0/24 and 192.168.56.0/24 in the Host machine's network (public network).
5)
Test the setup.
Ping and ssh work from compute node (192.168.56.102) to VM using private IP (172.168.56.3) of the VM.
Ping and ssh work from VM to compute node (192.168.56.102)
6)
Delete the changes.
* This settings will lost, If the vrouter agent restarts or if the compute node reboots.
*
You can manualy delete this settings with following command
$python /opt/contrail/utils/provision_vgw_interface.py --oper delete --interface vgw1 --subnets 172.168.56.0/24
7)
Infos from compute node (192.168.56.102)
a)
Configuration "vgw1" gateway interface from compute node
$ ifconfig
vgw1 Link encap:Ethernet HWaddr 00:01:00:5e:00:00
inet6 addr: fe80::201:ff:fe5e:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:195 errors:0 dropped:0 overruns:0 frame:0
TX packets:240 errors:0 dropped:5 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22854 (22.8 KB) TX bytes:24922 (24.9 KB)
b)
Configuration "vgw1" gateway interface from compute node
$ vif --list
vif0/5 OS: vgw1
Type:Gateway HWaddr:00:01:00:5e:00:00 IPaddr:0
Vrf:1 Flags:L3 MTU:1514 Ref:2
RX packets:240 bytes:24894 errors:0
TX packets:195 bytes:22854 errors:0
c)
Configuration "eth0", "eth1" and "vhost0" interface from compute node
$ifconfig eth0
eth0 Link encap:Ethernet HWaddr 08:00:27:65:a3:4e
inet addr:
10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe65:a34e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:607 errors:0 dropped:0 overruns:0 frame:0
TX packets:693 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:57547 (57.5 KB) TX bytes:60247 (60.2 KB)
$ifconfig eth1
eth1 Link encap:Ethernet HWaddr 08:00:27:22:8e:16
inet6 addr: fe80::a00:27ff:fe22:8e16/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:19205 errors:0 dropped:0 overruns:0 frame:0
TX packets:17793 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1529138 (1.5 MB) TX bytes:2711067 (2.7 MB)
$ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 08:00:27:22:8e:16
inet addr:
192.168.56.102 Bcast:192.168.56.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe22:8e16/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:19497 errors:0 dropped:0 overruns:0 frame:0
TX packets:18022 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1557182 (1.5 MB) TX bytes:2734972 (2.7 MB)
d)
Route table form compute node
$route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.2.2 0.0.0.0 UG 100 0 0 eth0
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.3 0.0.0.0 255.255.255.255 UH 0 0 0 vhost0
169.254.0.4 0.0.0.0 255.255.255.255 UH 0 0 0 vhost0
172.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0 vgw1 <==IMP
192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0 vhost0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
e)
Search for IP 172.168.56.x routes table of VRF 0
$rt --dump 0 | grep 172.168.56
Kernel IP routing table 0/1/unicast
Destination PPL Flags Label Nexthop
172.168.56.0/24 24 HT - 2
f)
Search for IP 172.168.56.3 and 172.168.56.4 in routes table of VRF 1
$ rt --dump 1 | grep 172.168.56.3
Kernel IP routing table 0/1/unicast
Destination PPL Flags Label Nexthop
172.168.56.3/32 32 H - 8
$ rt --dump 1 | grep 172.168.56.4
Kernel IP routing table 0/1/unicast
Destination PPL Flags Label Nexthop
172.168.56.4/32 32 H - 15
g)
Find Next hub info
$ nh --get 2
Id:002 Type:Receieve Fmly: AF_INET Flags:Valid, Policy(R), Rid:0 Ref_cnt:5
Oif:1
$ nh --get 8
Id:008 Type:Encap Fmly: AF_INET Flags:Valid, Policy, Rid:0 Ref_cnt:4
EncapFmly:0806 Oif:3 Len:14 Data:02 72 e3 89 8f 11 00 00 5e 00 01 00 08 00
$ nh --get 15
Id:015 Type:Encap Fmly: AF_INET Flags:Valid, Policy, Rid:0 Ref_cnt:4
EncapFmly:0806 Oif:4 Len:14 Data:02 5b a2 39 ed e1 00 00 5e 00 01 00 08 00
h)
Capture Flow table entries
From compute node Run "ping 172.168.56.3" in one terminal and Run "flow -l" in another terminal to capture the flows.
#flow -l
Flow table
Index Source:Port Destination:Port Proto(V)
-------------------------------------------------------------------------
185912 10.0.2.15:30857 172.168.56.3:0 1 (1)
(K(nh):17, Action:F, S(nh):17, Statistics:63/5292 UdpSrcPort 54744)
215132 172.168.56.3:30857 10.0.2.15:0 1 (1)
(K(nh):7, Action:F, S(nh):7, Statistics:63/5292 UdpSrcPort 51077)
Notes:
* Here we ping from compute node to VM (172.168.56.3).
* In Flow table, source is "10.0.2.15" that is actually the IP of "eth0" interface in the compute node.
i)
Flow table entries, when you SSH to VM (172.168.56.3) from Compute node
and ping to compute node's "vhost0" (192.168.56.102) interface from the VM (172.168.56.3).
$flow -l
Flow table
Index Source:Port Destination:Port Proto(V)
-------------------------------------------------------------------------
12744 172.168.56.4:22 10.0.2.15:53308 6 (1)
(K(nh):13, Action:F, S(nh):13, Statistics:137/17902 UdpSrcPort 61994)
29704 172.168.56.4:14593 192.168.56.102:0 1 (1)
(K(nh):13, Action:F, S(nh):13, Statistics:56/4704 UdpSrcPort 52697)
164716 192.168.56.102:14593 172.168.56.4:0 1 (1)
(K(nh):17, Action:F, S(nh):17, Statistics:56/4704 UdpSrcPort 61906)
496808 10.0.2.15:53308 172.168.56.4:22 6 (1)
(K(nh):17, Action:F, S(nh):17, Statistics:177/13837 UdpSrcPort 64809)
8)
Debugging
a)
Find "tap" interface in the host which connect VM to host.
* Name of the "tap" interface is constructed by using first 10 characters in the ID of port attached to the VM.
* You can find the ports using neutron command "#neutron port-list"
* For example: ID of port is "5ba239ed-e143-446a-ad1a-794869ccfbfe",
then you can find a "tap" interface named "tap5ba239ed-e1" in compute host. You can use "ifconfig" command to find it.
b)
Capture packets from interfaces using tshark
Capture packets from tap interface
"tap5ba239ed-e1"
#sudo tshark -i tap5ba239ed-e1 -f "icmp"
Capture packets from gateway interface
"vgw1"
#sudo tshark -i vgw1 -f "icmp"
Capture packets from
"vhost0" inet-interface
#sudo tshark -i vhost0 -f "icmp"
========Code==========
1)
#vim /usr/share/contrail-utils/provision_vgw_interface.py
import os
import sys
import argparse
import ConfigParser
from thrift.Thrift import TApplicationException, TException
from thrift.transport import TTransport, TSocket
from thrift.protocol import TBinaryProtocol, TProtocol
from netaddr.ip import IPNetwork
import InstanceService
import ttypes
from vnc_api.vnc_api import *
class ProvisionVgwInterface(object):
def __init__(self, args_str=None):
self._args = None
if not args_str:
args_str = ' '.join(sys.argv[1:])
self._parse_args(args_str)
# Connect to thrift service
try:
socket = TSocket.TSocket("localhost", 9090)
transport = TTransport.TFramedTransport(socket)
transport.open()
protocol = TBinaryProtocol.TBinaryProtocol(transport)
service = InstanceService.Client(protocol) <===IMP
except TApplicationException:
print "Error connecting to agent thrift service"
return
if self._args.oper == "create":
print "Creating virtual-gateway ..."
with open("/proc/sys/net/ipv4/ip_forward", "w") as file:
file.write("1") <===IMP
vif_command = '/usr/bin/vif --create ' + self._args.interface <===IMP
vif_command += ' --mac 00:01:00:5e:00:00'
self.execute_command(vif_command)
ifconfig_command = 'ifconfig ' + self._args.interface + ' up' <===IMP
self.execute_command(ifconfig_command)
for subnet in self._args.subnets:
route_command = 'route add -net ' + subnet <===IMP
route_command += ' dev ' + self._args.interface
self.execute_command(route_command)
subnet_list = []
for subnet in self._args.subnets: <===IMP
net = IPNetwork(subnet)
subnet_list.append(ttypes.Subnet(str(net.ip), net.prefixlen))
route_list = []
for subnet in self._args.routes: <===IMP
net = IPNetwork(subnet)
route_list.append(ttypes.Subnet(str(net.ip), net.prefixlen))
gw = ttypes.VirtualGatewayRequest(self._args.interface, self._args.vrf,
subnet_list, route_list)
gw_list = [ gw ]
try:
service.AddVirtualGateway(gw_list) <===IMP
except TApplicationException:
print "Error: Error adding VGW interface"
return
print "Done creating virtual-gateway..."
2)
#vim /usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py
class Client(Iface):
def __init__(self, iprot, oprot=None):
self._iprot = self._oprot = iprot
if oprot is not None:
self._oprot = oprot
self._seqid = 0
def AddVirtualGateway(self, vgw_list): <===IMP
"""
Parameters:
- vgw_list
"""
self.send_AddVirtualGateway(vgw_list)
return self.recv_AddVirtualGateway()
def send_AddVirtualGateway(self, vgw_list):
self._oprot.writeMessageBegin('AddVirtualGateway', TMessageType.CALL, self._seqid)
args = AddVirtualGateway_args()
args.vgw_list = vgw_list
args.write(self._oprot)
self._oprot.writeMessageEnd()
self._oprot.trans.flush()
def recv_AddVirtualGateway(self, ):
(fname, mtype, rseqid) = self._iprot.readMessageBegin()
if mtype == TMessageType.EXCEPTION:
x = TApplicationException()
x.read(self._iprot)
self._iprot.readMessageEnd()
raise x
result = AddVirtualGateway_result()
result.read(self._iprot)
self._iprot.readMessageEnd()
if result.success is not None:
return result.success
raise TApplicationException(TApplicationException.MISSING_RESULT, "AddVirtualGateway failed: unknown result");
