debugging openstack openvswitch VM instance not getting IP address

1)
In cirros VM, run dhcp client
$sudo /sbin/cirros-dhcpc up eth0

2)
In compute node, run tcpdump on tap interface of VM.

$ sudo tcpdump -i tap29d1dea7-48
$ sudo tcpdump -e -n -l -i tap29d1dea7-48 === to get more details

20:20:05.329510 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:21:05.393333 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:22:05.454023 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

3)
In compute node, run tcpdump on linux bridge.

$ sudo tcpdump -i qbr29d1dea7-48
$ sudo tcpdump -e -n -l -i xxxxxxx === to get more details

20:24:34.662338 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:25:34.753833 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

4)
In compute node, run tcpdump on veth pair linux bridge side.

$ sudo tcpdump -i qvb29d1dea7-48
$ sudo tcpdump -e -n -l -i xxxxx === to get more details

20:26:14.651031 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:27:14.742610 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

5)
In compute node, run tcpdump on veth pair openvswitch side

$ sudo tcpdump -i qvo29d1dea7-48
$ sudo tcpdump -e -n -l -i xxxxxxx === to get more details

20:28:52.396256 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:29:52.489598 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

6)
In compute node, run tcpdump on tunnel interface

$ sudo tcpdump -i eth2
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

7)
In network node, run tcpdump on tunnel interface

$ sudo tcpdump -i eth2

IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

8)
In network node, run tcpdump on ovs port "taped5c2f1f-91" which created with dhcp namespace, when we created subnet in network.
In dhcp namespace, dhcp server (dnsmasq) is bind to this interface "taped5c2f1f-91".

$ sudo ip netns exec qdhcp-f6a5f096-4379-403f-9c1d-ee496841c601 sudo tcpdump -i taped5c2f1f-91

* BOOTP/DHCP, Request not coming here. Why ??? =====

What is Next:
From step-9, debugging why BOOTP/DHCP, Request not coming in "taped5c2f1f-91".
From tunnel interface "eth2" BOOTP/DHCP Request should reach "taped5c2f1f-91" via port "patch-int" of "br-tun".

9)
In network node,  analyze ovs bridge, port and flow. 
(From setup where VM not getting private IP address)

9a)
Take dump of br-tun flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-tun

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=6120.838s, table=0, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=drop
 cookie=0x0, duration=6120.902s, table=0, n_packets=0, n_bytes=0, idle_age=6120, priority=1,in_port=3 actions=resubmit(,2) ==== There is no port numbered "3" in br-tun.
 cookie=0x0, duration=6119.748s, table=0, n_packets=42, n_bytes=13944, idle_age=1518, priority=1,in_port=4 actions=resubmit(,4)==== There is no port numbered "4" in br-tun.
 cookie=0x0, duration=6120.777s, table=2, n_packets=0, n_bytes=0, idle_age=6120, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=6120.713s, table=2, n_packets=0, n_bytes=0, idle_age=6120, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=6120.650s, table=3, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=drop
 cookie=0x0, duration=6120.588s, table=4, n_packets=42, n_bytes=13944, idle_age=1518, priority=0 actions=drop
 cookie=0x0, duration=6120.516s, table=10, n_packets=0, n_bytes=0, idle_age=6120, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:3
 cookie=0x0, duration=6120.454s, table=20, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=6120.390s, table=22, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=drop

Issue:
There is no port numbered "3" in br-tun, So table=0 drop packets and will not submit it to table=2.
There is no port numbered "4" in br-tun, So table=0 drop packets and will not submit it to table=4.

Solution:
Change table=0 and make in_port=1, that is port 1(patch-int)
Change table=0 and make in_port=2, that is port 2(vxlan-c0a84308)

OR

Delete all VMs, routers and private/tenant networks.
Then Recreate it. That will solve this issue.

9b)
Find port number of ports in br-tun in network node.
-----------------------

$ sudo ovs-ofctl show br-tun | grep '^ [0-9]'

 1(patch-int): addr:86:6d:2d:48:69:16  ======
 2(vxlan-c0a84308): addr:5a:c7:5a:ba:42:36

9c)
Take dump of br-int flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=6156.407s, table=0, n_packets=0, n_bytes=0, idle_age=6156, priority=1 actions=NORMAL
 cookie=0x0, duration=6155.663s, table=0, n_packets=0, n_bytes=0, idle_age=6155, priority=2,in_port=14 actions=drop
 cookie=0x0, duration=6152.831s, table=0, n_packets=0, n_bytes=0, idle_age=6152, priority=3,in_port=14,vlan_tci=0x0000 actions=mod_vlan_vid:1,NORMAL
 cookie=0x0, duration=6156.347s, table=23, n_packets=0, n_bytes=0, idle_age=6156, priority=0 actions=drop

9d)
Find port number of ports in br-int in  network node
----------------------------------

$ sudo ovs-ofctl show br-int | grep '^ [0-9]'

 9(taped5c2f1f-91): addr:13:02:00:00:00:00
 10(qr-62935d23-d7): addr:16:02:00:00:00:00
 11(qg-953aa488-d5): addr:16:02:00:00:00:00
 12(int-br-ex): addr:a2:76:eb:b7:df:5d
 13(patch-tun): addr:52:b2:7f:f9:e9:6c

10)
Neutron Flow Tables

The flows are divided in different tables.
table 0 all packets enter into this table
table 1 Packets coming from VM to Outside (Ingress)
table 2 Packets coming from outside Open vSwitch (tunnel) to VM (Egress) which need to go to a VM coming from VM are directed to 20 for Unicast and 21 for Multicast
table 3 not used
table 10 Inject a rule into table 20 to cause a return Path, so when VM repond to the Packet it will work.
table 20 does unicast packet
table 21 does broadcast packet

Links:
http://www.yet.org/2014/09/openvswitch-troubleshooting/

http://dischord.org/2015/03/09/troubleshooting-openstack-neutron-networking-part-one/

http://techbackground.blogspot.in/2013/05/debugging-quantum-dhcp-and-open-vswitch.html

https://www.rdoproject.org/networking/networking-in-too-much-detail/

11)
In network node,  analyze ovs bridge, port and flow.  (From working setup)
(From setup where VM getting private IP address)

11a)
Take dump of br-tun flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-tun

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=8557.694s, table=0, n_packets=0, n_bytes=0, idle_age=8557, priority=0 actions=drop
 cookie=0x0, duration=8557.758s, table=0, n_packets=163, n_bytes=19373, idle_age=37, priority=1,in_port=3 actions=resubmit(,2) ==== port 3 is patch-int
 cookie=0x0, duration=8556.604s, table=0, n_packets=223, n_bytes=40560, idle_age=37, priority=1,in_port=4 actions=resubmit(,4)  ==== port 4 is vxlan-c0a84308
 cookie=0x0, duration=8557.633s, table=2, n_packets=125, n_bytes=16493, idle_age=37, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=8557.569s, table=2, n_packets=38, n_bytes=2880, idle_age=996, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=8557.506s, table=3, n_packets=0, n_bytes=0, idle_age=8557, priority=0 actions=drop
 cookie=0x0, duration=8557.444s, table=4, n_packets=45, n_bytes=14940, idle_age=2249, priority=0 actions=drop
 cookie=0x0, duration=1418.749s, table=4, n_packets=178, n_bytes=25620, idle_age=37, priority=1,tun_id=0x5f actions=mod_vlan_vid:2,resubmit(,10)
 cookie=0x0, duration=8557.372s, table=10, n_packets=178, n_bytes=25620, idle_age=37, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:3
 cookie=0x0, duration=8557.310s, table=20, n_packets=5, n_bytes=322, idle_age=1235, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=1126.219s, table=20, n_packets=120, n_bytes=16171, hard_timeout=300, idle_age=37, hard_age=37, priority=1,vlan_tci=0x0002/0x0fff,dl_dst=fa:16:3e:1b:4f:5d actions=load:0->NXM_OF_VLAN_TCI[],load:0x5f->NXM_NX_TUN_ID[],output:4
 cookie=0x0, duration=8557.246s, table=22, n_packets=38, n_bytes=2840, idle_age=996, priority=0 actions=drop
 cookie=0x0, duration=1418.815s, table=22, n_packets=5, n_bytes=362, idle_age=1373, dl_vlan=2 actions=strip_vlan,set_tunnel:0x5f,output:4

11b)
Find port number of ports in br-tun in network node.
-----------------------

$ sudo ovs-ofctl show br-tun | grep '^ [0-9]'

 3(patch-int): addr:c2:fa:e4:a6:ff:05 ======
 4(vxlan-c0a84308): addr:0e:76:a8:5c:5b:58

11c)
Take dump of br-int flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=8613.916s, table=0, n_packets=346, n_bytes=46181, idle_age=31, priority=1 actions=NORMAL
 cookie=0x0, duration=8613.172s, table=0, n_packets=0, n_bytes=0, idle_age=8613, priority=2,in_port=14 actions=drop
 cookie=0x0, duration=1402.281s, table=0, n_packets=9, n_bytes=602, idle_age=1290, priority=3,in_port=14,vlan_tci=0x0000 actions=mod_vlan_vid:3,NORMAL
 cookie=0x0, duration=8613.856s, table=23, n_packets=0, n_bytes=0, idle_age=8613, priority=0 actions=drop

11d)
Find port number of ports in br-int
----------------------------------

$ sudo ovs-ofctl show br-int | grep '^ [0-9]'

 14(int-br-ex): addr:d6:38:3d:2b:27:26
 15(patch-tun): addr:56:d6:30:89:a8:ca
 16(tapcb5b468b-63): addr:00:00:00:00:00:00
 17(qr-86b1f209-32): addr:00:00:00:00:00:00
 18(qg-612ce037-d7): addr:00:00:00:00:00:00

12)
- qvo: veth pair openvswitch side

- qvb: veth pair bridge side

- qbr: bridge

- qr: l3 agent managed port, router side

- qg: l3 agent managed port, gateway side

13)