debugging openstack openvswitch VM instance not getting IP address

1)
In cirros VM, run dhcp client
$sudo /sbin/cirros-dhcpc up eth0

2)
In compute node, run tcpdump on tap interface of VM.

$ sudo tcpdump -i tap29d1dea7-48
$ sudo tcpdump -e -n -l -i tap29d1dea7-48 === to get more details

20:20:05.329510 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:21:05.393333 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:22:05.454023 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

3)
In compute node, run tcpdump on linux bridge.

$ sudo tcpdump -i qbr29d1dea7-48
$ sudo tcpdump -e -n -l -i xxxxxxx === to get more details

20:24:34.662338 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:25:34.753833 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

4)
In compute node, run tcpdump on veth pair linux bridge side.

$ sudo tcpdump -i qvb29d1dea7-48
$ sudo tcpdump -e -n -l -i xxxxx === to get more details

20:26:14.651031 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:27:14.742610 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

5)
In compute node, run tcpdump on veth pair openvswitch side

$ sudo tcpdump -i qvo29d1dea7-48
$ sudo tcpdump -e -n -l -i xxxxxxx === to get more details

20:28:52.396256 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
20:29:52.489598 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

6)
In compute node, run tcpdump on tunnel interface

$ sudo tcpdump -i eth2
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

7)
In network node, run tcpdump on tunnel interface

$ sudo tcpdump -i eth2

IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:2f:19:c7 (oui Unknown), length 290

* I am seeing only BOOTP/DHCP, Request, No Reply, Why??

8)
In network node, run tcpdump on ovs port "taped5c2f1f-91" which created with dhcp namespace, when we created subnet in network.
In dhcp namespace, dhcp server (dnsmasq) is bind to this interface "taped5c2f1f-91".

$ sudo ip netns exec qdhcp-f6a5f096-4379-403f-9c1d-ee496841c601 sudo tcpdump -i taped5c2f1f-91

* BOOTP/DHCP, Request not coming here. Why ??? =====

What is Next:
From step-9, debugging why BOOTP/DHCP, Request not coming in "taped5c2f1f-91".
From tunnel interface "eth2" BOOTP/DHCP Request should reach "taped5c2f1f-91" via port "patch-int" of "br-tun".

9)
In network node,  analyze ovs bridge, port and flow. 
(From setup where VM not getting private IP address)

9a)
Take dump of br-tun flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-tun

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=6120.838s, table=0, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=drop
 cookie=0x0, duration=6120.902s, table=0, n_packets=0, n_bytes=0, idle_age=6120, priority=1,in_port=3 actions=resubmit(,2) ==== There is no port numbered "3" in br-tun.
 cookie=0x0, duration=6119.748s, table=0, n_packets=42, n_bytes=13944, idle_age=1518, priority=1,in_port=4 actions=resubmit(,4)==== There is no port numbered "4" in br-tun.
 cookie=0x0, duration=6120.777s, table=2, n_packets=0, n_bytes=0, idle_age=6120, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=6120.713s, table=2, n_packets=0, n_bytes=0, idle_age=6120, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=6120.650s, table=3, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=drop
 cookie=0x0, duration=6120.588s, table=4, n_packets=42, n_bytes=13944, idle_age=1518, priority=0 actions=drop
 cookie=0x0, duration=6120.516s, table=10, n_packets=0, n_bytes=0, idle_age=6120, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:3
 cookie=0x0, duration=6120.454s, table=20, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=6120.390s, table=22, n_packets=0, n_bytes=0, idle_age=6120, priority=0 actions=drop

Issue:
There is no port numbered "3" in br-tun, So table=0 drop packets and will not submit it to table=2.
There is no port numbered "4" in br-tun, So table=0 drop packets and will not submit it to table=4.

Solution:
Change table=0 and make in_port=1, that is port 1(patch-int)
Change table=0 and make in_port=2, that is port 2(vxlan-c0a84308)

OR

Delete all VMs, routers and private/tenant networks.
Then Recreate it. That will solve this issue.

9b)
Find port number of ports in br-tun in network node.
-----------------------

$ sudo ovs-ofctl show br-tun | grep '^ [0-9]'

 1(patch-int): addr:86:6d:2d:48:69:16  ======
 2(vxlan-c0a84308): addr:5a:c7:5a:ba:42:36

9c)
Take dump of br-int flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=6156.407s, table=0, n_packets=0, n_bytes=0, idle_age=6156, priority=1 actions=NORMAL
 cookie=0x0, duration=6155.663s, table=0, n_packets=0, n_bytes=0, idle_age=6155, priority=2,in_port=14 actions=drop
 cookie=0x0, duration=6152.831s, table=0, n_packets=0, n_bytes=0, idle_age=6152, priority=3,in_port=14,vlan_tci=0x0000 actions=mod_vlan_vid:1,NORMAL
 cookie=0x0, duration=6156.347s, table=23, n_packets=0, n_bytes=0, idle_age=6156, priority=0 actions=drop

9d)
Find port number of ports in br-int in  network node
----------------------------------

$ sudo ovs-ofctl show br-int | grep '^ [0-9]'

 9(taped5c2f1f-91): addr:13:02:00:00:00:00
 10(qr-62935d23-d7): addr:16:02:00:00:00:00
 11(qg-953aa488-d5): addr:16:02:00:00:00:00
 12(int-br-ex): addr:a2:76:eb:b7:df:5d
 13(patch-tun): addr:52:b2:7f:f9:e9:6c

10)
Neutron Flow Tables

The flows are divided in different tables.
table 0 all packets enter into this table
table 1 Packets coming from VM to Outside (Ingress)
table 2 Packets coming from outside Open vSwitch (tunnel) to VM (Egress) which need to go to a VM coming from VM are directed to 20 for Unicast and 21 for Multicast
table 3 not used
table 10 Inject a rule into table 20 to cause a return Path, so when VM repond to the Packet it will work.
table 20 does unicast packet
table 21 does broadcast packet

Links:
http://www.yet.org/2014/09/openvswitch-troubleshooting/

http://dischord.org/2015/03/09/troubleshooting-openstack-neutron-networking-part-one/

http://techbackground.blogspot.in/2013/05/debugging-quantum-dhcp-and-open-vswitch.html

https://www.rdoproject.org/networking/networking-in-too-much-detail/

11)
In network node,  analyze ovs bridge, port and flow.  (From working setup)
(From setup where VM getting private IP address)

11a)
Take dump of br-tun flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-tun

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=8557.694s, table=0, n_packets=0, n_bytes=0, idle_age=8557, priority=0 actions=drop
 cookie=0x0, duration=8557.758s, table=0, n_packets=163, n_bytes=19373, idle_age=37, priority=1,in_port=3 actions=resubmit(,2) ==== port 3 is patch-int
 cookie=0x0, duration=8556.604s, table=0, n_packets=223, n_bytes=40560, idle_age=37, priority=1,in_port=4 actions=resubmit(,4)  ==== port 4 is vxlan-c0a84308
 cookie=0x0, duration=8557.633s, table=2, n_packets=125, n_bytes=16493, idle_age=37, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=8557.569s, table=2, n_packets=38, n_bytes=2880, idle_age=996, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=8557.506s, table=3, n_packets=0, n_bytes=0, idle_age=8557, priority=0 actions=drop
 cookie=0x0, duration=8557.444s, table=4, n_packets=45, n_bytes=14940, idle_age=2249, priority=0 actions=drop
 cookie=0x0, duration=1418.749s, table=4, n_packets=178, n_bytes=25620, idle_age=37, priority=1,tun_id=0x5f actions=mod_vlan_vid:2,resubmit(,10)
 cookie=0x0, duration=8557.372s, table=10, n_packets=178, n_bytes=25620, idle_age=37, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:3
 cookie=0x0, duration=8557.310s, table=20, n_packets=5, n_bytes=322, idle_age=1235, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=1126.219s, table=20, n_packets=120, n_bytes=16171, hard_timeout=300, idle_age=37, hard_age=37, priority=1,vlan_tci=0x0002/0x0fff,dl_dst=fa:16:3e:1b:4f:5d actions=load:0->NXM_OF_VLAN_TCI[],load:0x5f->NXM_NX_TUN_ID[],output:4
 cookie=0x0, duration=8557.246s, table=22, n_packets=38, n_bytes=2840, idle_age=996, priority=0 actions=drop
 cookie=0x0, duration=1418.815s, table=22, n_packets=5, n_bytes=362, idle_age=1373, dl_vlan=2 actions=strip_vlan,set_tunnel:0x5f,output:4

11b)
Find port number of ports in br-tun in network node.
-----------------------

$ sudo ovs-ofctl show br-tun | grep '^ [0-9]'

 3(patch-int): addr:c2:fa:e4:a6:ff:05 ======
 4(vxlan-c0a84308): addr:0e:76:a8:5c:5b:58

11c)
Take dump of br-int flow from network node:
--------------------------------

$ sudo ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=8613.916s, table=0, n_packets=346, n_bytes=46181, idle_age=31, priority=1 actions=NORMAL
 cookie=0x0, duration=8613.172s, table=0, n_packets=0, n_bytes=0, idle_age=8613, priority=2,in_port=14 actions=drop
 cookie=0x0, duration=1402.281s, table=0, n_packets=9, n_bytes=602, idle_age=1290, priority=3,in_port=14,vlan_tci=0x0000 actions=mod_vlan_vid:3,NORMAL
 cookie=0x0, duration=8613.856s, table=23, n_packets=0, n_bytes=0, idle_age=8613, priority=0 actions=drop

11d)
Find port number of ports in br-int
----------------------------------

$ sudo ovs-ofctl show br-int | grep '^ [0-9]'

 14(int-br-ex): addr:d6:38:3d:2b:27:26
 15(patch-tun): addr:56:d6:30:89:a8:ca
 16(tapcb5b468b-63): addr:00:00:00:00:00:00
 17(qr-86b1f209-32): addr:00:00:00:00:00:00
 18(qg-612ce037-d7): addr:00:00:00:00:00:00

12)
- qvo: veth pair openvswitch side

- qvb: veth pair bridge side

- qbr: bridge

- qr: l3 agent managed port, router side

- qg: l3 agent managed port, gateway side

13)

How to run cirros image dhcp client

$sudo /sbin/cirros-dhcpc up eth0

$sudo /sbin/cirros-dhcpc down eth0

What is the meaning of /dev/null 2>&1

How to redirect both STDOUT and STDERROR to /dev/null

First, let me try without "2>&1"

a)
This command redirect output of command "git --version" to /dev/null
That is redirect only STDOUT to /dev/null 

$git --version >/dev/null
$

b)
This command does not have output to redirect to /dev/null. Because command
"gitxxx --version" failed with Error "gitxxx: command not found". We didn't
tell shell to redirect Error to /dev/null, So it printed error in STDOUT(1).

$gitxxx --version >/dev/null
gitxxx: command not found
$

c)
We can tell shell to redirect Error to STDOUT(1).
Here 2 means STDERROR and 1 is STDOUT.
2>&1 Means redirect STDERROR to STDOUT file descriptor.

So below command will redirect both STDOUT and STDERROR to /dev/null

$git --version >/dev/null 2>&1
$gitxxx --version >/dev/null 2>&1

How to redirect file descriptions STDIN, STDOUT, and STDERR

there are three standard file descriptions, STDIN, STDOUT, and STDERR. They are assigned to 0, 1, and 2 respectively.

a)
STDIN (0):
$git --version 0>test.txt
$cat test.txt
$gitblabla --version 0>test.txt
$cat test.txt

Example:
$git --version 0>test.txt
git version 1.9.1 === Output printed in stdout
$cat test.txt
$gitblabla --version 0>test.txt
gitblabla: command not found === Error printed in stdout
$cat test.txt
$

b)
STDOUT (1):
$git --version 1>test.txt
$cat test.txt
$gitblabla --version 1>test.txt
$cat test.txt

Example:
$git --version 1>test.txt
$cat test.txt
git version 1.9.1 === Output redirected to file
$gitblabla --version 1>test.txt
gitblabla: command not found === Error printed in stdout
$cat test.txt
$

c)
STDERR (2):
$git --version 2>test.txt
$cat test.txt
$gitblabla --version 2>test.txt
$cat test.txt

Example:
$git --version 2>test.txt
git version 1.9.1 === Output printed in stdout
$cat test.txt
$gitblabla --version 2>test.txt
$cat test.txt
gitblabla: command not found === Error redirected to file

How to tacker-db-manage generate database migration script and test upgrade and downgrade

a)
First add new column to "Vim" model.

$tacker/db/nfvo/nfvo_db.py

class Vim(model_base.BASE, models_v1.HasId, models_v1.HasTenant):
    type = sa.Column(sa.String(64), nullable=False)
    name = sa.Column(sa.String(255), nullable=False)
    description = sa.Column(sa.Text, nullable=True)
    placement_attr = sa.Column(types.Json, nullable=True)
    shared = sa.Column(sa.Boolean, default=True, server_default=sql.true(
    ), nullable=False)
    default = sa.Column(sa.Boolean, default=False, server_default=sql.false(
    ), nullable=False)
    vim_auth = orm.relationship('VimAuth')
    status = sa.Column(sa.String(255), nullable=False)

b)
Then Generate Migration script.
$ ls tacker/db/migration/alembic_migrations/versions/ | wc -l

* name of the migration script file taken from -m option
$ tacker-db-manage --config-file /etc/tacker/tacker.conf revision -m "add default to vim" --autogenerate

* Open auto generated  migration script.
$ vim tacker/db/migration/alembic_migrations/versions/d4f265e8eb9d_add_default_to_vim.py

* Then delete unwanted statements from "upgrade" and "downgrade" methods

c)
Test upgrade and downgrade:

$ tacker-db-manage --config-file /etc/tacker/tacker.conf check_migration

$ tacker-db-manage --config-file /etc/tacker/tacker.conf history
2f2e337fc6c4 -> d4f265e8eb9d (head), add default to vim

$ tacker-db-manage --config-file /etc/tacker/tacker.conf current
2f2e337fc6c4

* Upgrade to head
$ tacker-db-manage --config-file /etc/tacker/tacker.conf upgrade head
INFO  [alembic.runtime.migration] Running upgrade 2f2e337fc6c4 -> d4f265e8eb9d, add default to vim

$ tacker-db-manage --config-file /etc/tacker/tacker.conf current
d4f265e8eb9d (head)

* Downgrade to 2f2e337fc6c4
$ tacker-db-manage --config-file /etc/tacker/tacker.conf downgrade 2f2e337fc6c4
INFO  [alembic.runtime.migration] Running downgrade d4f265e8eb9d -> 2f2e337fc6c4, add default to vim

$ tacker-db-manage --config-file /etc/tacker/tacker.conf current
2f2e337fc6c4

How to openstack ec2api query instances from ec2api database

$ select * from items where project_id="xxxxxxxxxx" and id like "i-%" limit 10;

How to use tacker-db-manage to migrate database

1)
$ tacker-db-manage --config-file /etc/tacker/tacker.conf history
5f88e86b35c7 -> 22f5385a3d3f (head), Add status to vims
354de64ba129 -> 5f88e86b35c7, make VNFD/VNF/VIM name mandatory
b07673bb8654 -> 354de64ba129, set-mandatory-columns-not-null
c7cde2f45f82 -> b07673bb8654, set-status-type-tenant-id-length
6e56d4474b2a -> c7cde2f45f82, set-description-to-text
f958f58e5daa -> 6e56d4474b2a, blob-to-json-text
acf941e54075 -> f958f58e5daa, uuid consistency
5246a6bd410f -> acf941e54075, Add error_reason to device
24bec5f211c7 -> 5246a6bd410f, multisite_vim
2774a42c7163 -> 24bec5f211c7, Alter value in deviceattributes
12a57080b278 -> 2774a42c7163, remove service related
12a57080b277 -> 12a57080b278, Alter devices
5958429bcb3c -> 12a57080b277, Add Service related dbs
13c0e0661015 -> 5958429bcb3c, modify datatype of value
4c31092895b8 -> 13c0e0661015, add descrition to vnf
81ffa86020d -> 4c31092895b8, empty message
1c6b0d82afcd -> 81ffa86020d, rpc_proxy
-> 1c6b0d82afcd, add tables for tacker framework

2)

$ tacker-db-manage --config-file /etc/tacker/tacker.conf current

INFO  [alembic.runtime.migration] Context impl MySQLImpl.

INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

5f88e86b35c7

3)
$ tacker-db-manage --config-file /etc/tacker/tacker.conf revision
  Generating /opt/stack/tacker/tacker/db/migration/alembic_migrations/versions/2f2e337fc6c4_.py ... done

4)
$ tacker-db-manage --config-file /etc/tacker/tacker.conf upgrade 2f2e337fc6c4
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 5f88e86b35c7 -> 22f5385a3d3f, Add status to vims
INFO  [alembic.runtime.migration] Running upgrade 22f5385a3d3f -> 2f2e337fc6c4, empty message

5)
$ tacker-db-manage --config-file /etc/tacker/tacker.conf history
22f5385a3d3f -> 2f2e337fc6c4 (head), empty message
5f88e86b35c7 -> 22f5385a3d3f, Add status to vims
354de64ba129 -> 5f88e86b35c7, make VNFD/VNF/VIM name mandatory
b07673bb8654 -> 354de64ba129, set-mandatory-columns-not-null
c7cde2f45f82 -> b07673bb8654, set-status-type-tenant-id-length
6e56d4474b2a -> c7cde2f45f82, set-description-to-text
f958f58e5daa -> 6e56d4474b2a, blob-to-json-text
acf941e54075 -> f958f58e5daa, uuid consistency
5246a6bd410f -> acf941e54075, Add error_reason to device
24bec5f211c7 -> 5246a6bd410f, multisite_vim
2774a42c7163 -> 24bec5f211c7, Alter value in deviceattributes
12a57080b278 -> 2774a42c7163, remove service related
12a57080b277 -> 12a57080b278, Alter devices
5958429bcb3c -> 12a57080b277, Add Service related dbs
13c0e0661015 -> 5958429bcb3c, modify datatype of value
4c31092895b8 -> 13c0e0661015, add descrition to vnf
81ffa86020d -> 4c31092895b8, empty message
1c6b0d82afcd -> 81ffa86020d, rpc_proxy
-> 1c6b0d82afcd, add tables for tacker framework

6)
$ tacker-db-manage --config-file /etc/tacker/tacker.conf current
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
2f2e337fc6c4 (head)

========================
1)
Migrate to head
$ tacker-db-manage --config-file /etc/tacker/tacker.conf upgrade head