#sudo service contrail-schema status
#sudo tcpdump -i eth1 -s 0 -w dump1.pcap
#sudo tcpdump -i lo -s 0 -w dump1.pcap <=== To capture from contrail bigdebian package setup and open in wireshark.
xmpp_server_port:
tcp.port==5269
cassandra.server_port:
tcp.port==9160
ifmap_server_port:
tcp.port==8443
zk_server_port:
tcp.port==2181
bgp_port:
tcp.port==179
rabbit_port:
tcp.port==5672
To find a string within a packet, click on Edit > Find Packet. Under "Find By:" select "string" and enter your search string in the text entry box.There is a lot more information in most packets than what appears in the packet list Info column, so try "Packet details" and "Packet bytes".
################################
IFMAP Poll
=================
a)
schema-transformer poll from ifmap:
contrail-controller/src/config/schema-transformer/to_bgp.py:
b)
svc_monitor poll from ifmap:
contrail-controller/src/config/svc-monitor/svc_monitor/svc_monitor.py
c)
controller poll from ifmap:
contrail-controller/src/ifmap/client/ifmap_state_machine.cc <=== Contains ifmap client code
contrail-controller/src/ifmap/client/ifmap_channel.cc <=== contains code which set namespaces
contrail-controller/src/ifmap/client/test/ifmap_state_machine_test.cc
d)
contrail control ifmap poll, subscribe code:
contrail-controller/src/ifmap/client/ifmap_channel.cc
* Search for contrail:config-root:root
* void IFMapChannel::SendSubscribe()
* void IFMapChannel::SubscribeResponseWait()
* int IFMapChannel::ReadSubscribeResponseStr()
* void IFMapChannel::SendPollRequest()
* void IFMapChannel::PollResponseWait()
* int IFMapChannel::ReadPollResponse()
e)
Also check
contrail-controller/src/ifmap/client/ifmap_state_machine.cc
* struct SendSubscribe : sc::state<SendSubscribe, IFMapStateMachine> {
* struct SubscribeResponseWait :
* struct ArcConnect : sc::state<ArcConnect, IFMapStateMachine>
* struct ArcSslHandshake : sc::state<ArcSslHandshake, IFMapStateMachine> {
* struct SendPoll : sc::state<SendPoll, IFMapStateMachine> {
* struct PollResponseWait :
* IFMapStateMachine::IFMapStateMachine(IFMapManager *manager)
f)
contrail-controller/src/ifmap/ifmap_server_parser.cc
* bool IFMapServerParser::ParseResultItem(
* void IFMapServerParser::ParseResults(
* bool IFMapServerParser::ParseMetadata(const pugi::xml_node &node,
* static DBRequest *IFMapServerRequestClone(const DBRequest *src) {
g)
contrail-controller/src/ifmap/ifmap_graph_walker.cc
* void IFMapGraphWalker::AddNodesToWhitelist() {
* void IFMapGraphWalker::AddLinksToWhitelist() {
################################
Official repo of ifmap-python-client:
https://github.com/ITI/ifmap-python-client/
contrail patch:
https://github.com/Juniper/contrail-third-party/blob/master/ifmap-python-async.patch
########################
xmllint
a)
sudo apt-get install xmllint
xmllint --format dump1.xml > new_dump1.xml
xmllint --format dump2.xml > new_dump2.xml
vimdiff new_dump1.xml new_dump2.xml
diff -y 1.xml 2.xm
diff -y --suppress-common-lines 1.xml 2.xml
#######################
Newly added lines in ifmap on schema start after vm creation.
a)
1513 <resultItem>
1514 <identity name="contrail:virtual-machine-interface:default-domain:testtenant:6d7ceaea-04a0-4067-8e25-91c2da6c804d" type="other" other-type-definition="extended"/>
1515 <identity name="contrail:routing-instance:default-domain:testtenant:mynw1:mynw1" type="other" other-type-definition="extended"/>
1516 <metadata>
1517 <contrail:virtual-machine-interface-routing-instance xmlns:contrail="http://www.contrailsystems.com/vnc_cfg.xsd" ifmap-cardinality="singleValue" ifmap-publisher-id="api-server-1--0000000001-1" ifmap-timestamp="2015-07-08T12:09:12+00:00">
1518 <direction>both</direction>
1519 </contrail:virtual-machine-interface-routing-instance>
1520 </metadata>
1521 </resultItem>
b)
class SchemaTransformer(object):
def add_virtual_machine_interface_virtual_network(self, idents, meta):
c)
class VirtualMachineInterfaceST(DictST):
def set_virtual_network(self, vn_name):
if_obj = _vnc_lib.virtual_machine_interface_read(id=self.uuid) <===1
ri = virtual_network.get_primary_routing_instance().obj
refs = if_obj.get_routing_instance_refs()
if ri.get_fq_name() not in [r['to'] for r in (refs or [])]:
if_obj.add_routing_instance( <===2
ri, PolicyBasedForwardingRuleType(direction="both")) <===3 meta <direction>both</direction>
#######################
The publish operation associates metadata with identifiers or links between identifiers. The
search and subscribe operations use an identifier as the starting point for a query (see 3.7).
#######################
https://www.juniper.net/techpubs/en_US/release-independent/contrail/information-products/pathway-pages/api-server/vnc_cfg_api_server.ifmap.html
#######################
to_bgp.py methods which get invoke after vm creation.
idents['virtual-machine-interface'], identity
--------------------------------------------------
def add_virtual_machine_interface_virtual_network(self, idents, meta):
vmi_name = idents['virtual-machine-interface']
vn_name = idents['virtual-network']
vmi = VirtualMachineInterfaceST.locate(vmi_name)
if vmi is not None:
vmi.set_virtual_network(vn_name)
self.current_network_set |= vmi.rebake()
def add_instance_ip_virtual_machine_interface(self, idents, meta):
vmi_name = idents['virtual-machine-interface']
ip_name = idents['instance-ip']
vmi = VirtualMachineInterfaceST.locate(vmi_name)
if vmi is not None:
vmi.add_instance_ip(ip_name)
self.current_network_set |= vmi.rebake()
def add_virtual_machine_interface_properties(self, idents, meta):
vmi_name = idents['virtual-machine-interface']
prop = VirtualMachineInterfacePropertiesType()
vmi = VirtualMachineInterfaceST.locate(vmi_name)
prop.build(meta)
if vmi is not None:
vmi.set_service_interface_type(prop.get_service_interface_type())
self.current_network_set |= vmi.rebake()
vmi.set_interface_mirror(prop.get_interface_mirror())
def add_virtual_machine_interface_virtual_machine(self, idents, meta):
vmi_name = idents['virtual-machine-interface']
vm_name = idents['virtual-machine']
vmi = VirtualMachineInterfaceST.locate(vmi_name)
if vmi is not None:
vmi.set_virtual_machine(vm_name)
###############################
$zgrep eef20dbe-4a78-4a90-af05-162aaf87cba4 *.gz > somefile
#############################
CleanupInterest:
controller/src/ifmap/ifmap_graph_walker.cc
IFMapGraphWalker::IFMapGraphWalker(DBGraph *graph, IFMapExporter *exporter)
: graph_(graph),
exporter_(exporter),
work_queue_(TaskScheduler::GetInstance()->GetTaskId("db::DBTable"), 0,
boost::bind(&IFMapGraphWalker::Worker, this, _1)) {
work_queue_.SetExitCallback(
boost::bind(&IFMapGraphWalker::WorkBatchEnd, this, _1));<====1
traversal_white_list_.reset(new IFMapTypenameWhiteList());
AddNodesToWhitelist(); <====3
AddLinksToWhitelist(); <====4
}
// Cleanup all graph nodes that a bit set in the remove mask (rm_mask_) but
// where not visited by the walker.
void IFMapGraphWalker::WorkBatchEnd(bool done) { <====2
for (DBGraph::vertex_iterator iter = graph_->vertex_list_begin();
iter != graph_->vertex_list_end(); ++iter) {
DBGraphVertex *vertex = iter.operator->();
CleanupInterest(vertex);
}
rm_mask_.clear();
}
void IFMapGraphWalker::CleanupInterest(DBGraphVertex *vertex) {
// interest = interest - rm_mask_ + nmask
IFMapNode *node = static_cast<IFMapNode *>(vertex);
IFMapNodeState *state = exporter_->NodeStateLookup(node);
if (state == NULL) {
return;
}
if (!state->interest().empty() && !state->nmask().empty()) {
IFMAP_DEBUG(CleanupInterest, node->ToString(),
state->interest().ToString(), rm_mask_.ToString(),
state->nmask().ToString());
}
.... ....
.... ....
}
#############################
IFMapClientSendInfo:
controller/src/ifmap/ifmap_update_sender.cc
#############################
http://lists.opencontrail.org/pipermail/dev_lists.opencontrail.org/2014-December/001782.html
> 2014-12-05 00:56:01,783 [main] INFO - Starting irond version 0.3.2...
> 2014-12-05 00:56:05,582 [main] INFO - EventProcessor: Running with 4 workers and 2 forwarders
> 2014-12-05 00:56:05,587 [main] INFO - ActionProcessor: Running with 1 workers and 1 forwarders
> 2014-12-05 00:56:05,587 [main] INFO - ChannelAcceptor: Listening on port 8443 for incoming basic authentication connections
> 2014-12-05 00:56:05,590 [main] INFO - ChannelAcceptor: Listening on port 8444 for incoming certificate-based authentication connections
> 2014-12-05 00:56:05,595 [main] INFO - irond is running :-)
> 2014-12-05 00:56:06,060 [pool-1-thread-1] FATAL - Could not store generated publisher-id for api-server
> 2014-12-05 00:56:18,139 [pool-1-thread-1] FATAL - Could not store generated publisher-id for control
> 2014-12-05 00:56:24,442 [pool-1-thread-3] FATAL - Could not store generated publisher-id for schema-transformer
########### IMP-1 ##################
a)
All routing instances of networks in all projects get cleaned up on : 2015-07-01 Wed
#zgrep 'CleanupInterest: routing-instance:default-domain' contrail-control.log-20150702.gz
#zgrep 'CleanupInterest: routing-instance:default-domain' contrail-control.log-20150702.gz | grep jiocloud_pro
b)
Before above error (a), I can see some "IFMapXmppVmSubUnsub" errro in contrail-control.log-20150702.gz on same date 2015-07-01 Wed.
#zgrep 'IFMapXmppVmSubUnsub: VmUnsubscribe ct1-production' contrail-control.log-20150702.gz
===========
#zgrep 'Error received instead of PollResult' *.gz
###########IMP-2##################
1)
=======
a)
Down ifmap server
b)
Create a VM
* Note Id of the VM
c)
Check
#curl http://127.0.0.1:8083/Snh_IFMapPendingVmRegReq
* vm id should be there.
* In production setup, we have seen id of bugy vm in this list.
d)
* Run "#nova show <vm-id>" and find the private ip of vm.
* Run "#neutron port-list" and find the port-id and construct interface name like "tap<fisr_11_char_of_port_id>" eg:tapce6e07bc-66.
e)
Search for vm's id in /var/log/contrail/contrail-control.log
* You should see an entry like
2015-07-11 Sat 10:44:44:000.010 UTC ct1-testjenkins-puppet-rjil-gate-1659 [Thread 140043615483648, Pid 20567]: Sandesh: Send: FAILED: 1436611483999988 IFMapXMPP [SYS_DEBUG]: IFMapXmppVmSubUnsub: VmSubscribe ct1-testjenkins-puppet-rjil-gate-1659:10.0.0.70 08f221ab-72f3-46c4-b823-f7f3a2097b5d controller/src/ifmap/ifmap_xmpp.cc 216
f)
Go to compute node of the VM and check tap interface "tapce6e07bc-66"
#curl http://127.0.0.1:8085/Snh_ItfReq?x=tapce6e07bc-66
* You can see that vrf_name of interface "tapce6e07bc-66" is --ERROR--
<vrf_name type="string" identifier="4" link="VrfListReq">--ERROR--</vrf_name>
g)
Search result of bugy vm_id form production contrail-control.log-20150702.gz
g1)
* VM created on 2015-07-01 Wed 08:56
* First VmSubscribe request from agent on 2015-07-01 Wed 08:56:
2015-07-01 Wed 08:56:17:122.262 UTC ct1-production [Thread 140105121527552, Pid 48037]: Sandesh: Send: FAILED: 1435740977122235 IFMapXMPP [SYS_DEBUG]: IFMapXmppVmSubUnsub: VmSubscribe ct1-production:10.140.192.96 0d4722d7-c1d9-421c-94eb-eaf278597226 controller/src/ifmap/ifmap_xmpp.cc 216
g2)
* (after one day) Second VmSubscribe request from agent on 2015-07-02 Thu 02:48:
* I think, something got restared or fixed on 2015-07-02 Thu 02:48:
2015-07-02 Thu 02:48:26:590.027 UTC ct1-production [Thread 139899038537472, Pid 61256]: Sandesh: Send: FAILED: 1435805306590019 IFMapXMPP [SYS_DEBUG]: IFMapXmppVmSubUnsub: VmSubscribe ct1-production:10.140.192.96 0d4722d7-c1d9-421c-94eb-eaf278597226 controller/src/ifmap/ifmap_xmpp.cc 216
g3)
* Got data from ifmap on 2015-07-02 Thu 02:48:
2015-07-02 Thu 02:48:28:057.616 UTC ct1-production [Thread 139899379013376, Pid 61256]: Sandesh: Send: FAILED: 1435805308024459 IFMapPeer [SYS_DEBUG]: IFMapServerConnection: 0 bytes in reply_. 8843481 bytes in reply_str. PollResponse message is:
HTTP/1.1 200 ^M
Content-Type: application/soap+xml^M
Content-Length: 8843403^M
</resultItem><resultItem><identity name="contrail:virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226" type="other" other-type-definition="extend
2015-07-02 Thu 02:48:28:485.031 UTC ct1-production [Thread 139899383211776, Pid 61256]: Sandesh: Send: FAILED: 1435805308485025 IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226 controller/src/ifmap/ifmap_server_table.cc 89
2015-07-02 Thu 02:48:28:565.594 UTC ct1-production [Thread 139899383211776, Pid 61256]: Sandesh: Send: FAILED: 1435805308565585 IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating link <virtual-machine-interface:default-domain:openstack:eef20dbe-4a78-4a90-af05-162aaf87cba4,virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226> controller/src/ifmap/ifmap_link_table.cc 67
2015-07-02 Thu 02:48:29:253.401 UTC ct1-production [Thread 139898275215104, Pid 61256]: Sandesh: Send: FAILED: 1435805309253389 IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating link <virtual-router:default-global-system-config:cp4-production,virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226> controller/src/ifmap/ifmap_link_table.cc 67
2015-07-02 Thu 02:48:30:512.098 UTC ct1-production [Thread 139898292008704, Pid 61256]: Sandesh: Send: FAILED: 1435805310512089 IFMap [SYS_DEBUG]: LinkOper: LinkAdd virtual-router:default-global-system-config:cp4-production - virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226 , lhs: 000000000000000001 , rhs: controller/src/ifmap/ifmap_graph_walker.cc 106
2015-07-02 Thu 02:48:30:514.012 UTC ct1-production [Thread 139898292008704, Pid 61256]: Sandesh: Send: FAILED: 1435805310514003 IFMap [SYS_DEBUG]: JoinVertex: JoinVertex: virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226 controller/src/ifmap/ifmap_graph_walker.cc 81
2015-07-02 Thu 02:48:30:583.899 UTC ct1-production [Thread 139898275215104, Pid 61256]: Sandesh: Send: FAILED: 1435805310583891 IFMap [SYS_DEBUG]: IFMapClientSendInfo: Sent Update of virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226 to vRouter ct1-production:10.140.192.96 controller/src/ifmap/ifmap_update_sender.cc 315
2015-07-02 Thu 02:48:30:583.938 UTC ct1-production [Thread 139898275215104, Pid 61256]: Sandesh: Send: FAILED: 1435805310583928 IFMap [SYS_DEBUG]: IFMapClientSendInfo: Sent Update of link <virtual-router:default-global-system-config:cp4-production,virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226> to vRouter ct1-production:10.140.192.96 controller/src/ifmap/ifmap_update_sender.cc 315
2015-07-02 Thu 02:48:31:702.033 UTC ct1-production [Thread 139899097315072, Pid 61256]: Sandesh: Send: FAILED: 1435805311702025 IFMap [SYS_DEBUG]: IFMapClientSendInfo: Sent Update of link <virtual-machine-interface:default-domain:openstack:eef20dbe-4a78-4a90-af05-162aaf87cba4,virtual-machine:0d4722d7-c1d9-421c-94eb-eaf278597226> to vRouter ct1-production:10.140.192.96 controller/src/ifmap/ifmap_update_sender.cc 315
h)
* Search for "2015-07-02 Thu 02:48" in production contrail-control.log-20150702.gz
* You will get following clues
h0)
* This error you can see from date "2015-07-01 Wed 06:47" to "2015-07-02 Thu 02:48", until issue got resolved.
* -(-1) OnSessionEvent TCP Connect Failed
* -(-1):EvTcpConnectFail
* From Thread 140105745336256, Pid 48037
2015-07-02 Thu 02:46:26:114.552 UTC ct1-production [Thread 140105075345152, Pid 48037]: Connect : EvTcpConnectFail
2015-07-02 Thu 02:46:26:114.637 UTC ct1-production [Thread 140105075345152, Pid 48037]: Idle
2015-07-02 Thu 02:46:26:114.701 UTC ct1-production [Thread 140105075345152, Pid 48037]: Processing scm::EvSandeshSend in state Idle
2015-07-02 Thu 02:46:26:114.717 UTC ct1-production [Thread 140105075345152, Pid 48037]: Wrong state: Idle for event: EvSandeshSend
2015-07-02 Thu 02:46:26:114.731 UTC ct1-production [Thread 140105075345152, Pid 48037]: Processing scm::EvSandeshSend in state Idle
2015-07-02 Thu 02:46:26:114.743 UTC ct1-production [Thread 140105075345152, Pid 48037]: Wrong state: Idle for event: EvSandeshSend
2015-07-02 Thu 02:46:26:114.755 UTC ct1-production [Thread 140105075345152, Pid 48037]: Processing scm::EvTcpDeleteSession in state Idle
2015-07-02 Thu 02:46:26:114.773 UTC ct1-production [Thread 140105075345152, Pid 48037]: Processing scm::EvSandeshSend in state Idle
2015-07-02 Thu 02:46:26:114.785 UTC ct1-production [Thread 140105075345152, Pid 48037]: Wrong state: Idle for event: EvSandeshSend
2015-07-02 Thu 02:46:36:419.952 UTC ct1-production [Thread 140105121527552, Pid 48037]: Processing scm::EvSandeshSend in state Idle
2015-07-02 Thu 02:46:36:420.056 UTC ct1-production [Thread 140105121527552, Pid 48037]: Wrong state: Idle for event: EvSandeshSend
2015-07-02 Thu 02:46:44:318.227 UTC ct1-production [Thread 140105573189376, Pid 48037]: Processing scm::EvSandeshSend in state Idle
2015-07-02 Thu 02:46:44:318.320 UTC ct1-production [Thread 140105573189376, Pid 48037]: Wrong state: Idle for event: EvSandeshSend
2015-07-02 Thu 02:46:44:318.369 UTC ct1-production [Thread 140105573189376, Pid 48037]: Processing scm::EvSandeshSend in state Idle
2015-07-02 Thu 02:46:44:318.410 UTC ct1-production [Thread 140105573189376, Pid 48037]: Wrong state: Idle for event: EvSandeshSend
2015-07-02 Thu 02:46:56:114.903 UTC ct1-production [Thread 140105066948352, Pid 48037]: Processing scm::EvIdleHoldTimerExpired in state Idle
2015-07-02 Thu 02:46:56:115.232 UTC ct1-production [Thread 140105066948352, Pid 48037]: Connect : Start Connect timer 10.140.192.40:8086
2015-07-02 Thu 02:46:56:115.266 UTC ct1-production [Thread 140105745336256, Pid 48037]: -(-1) OnSessionEvent TCP Connect Failed
2015-07-02 Thu 02:46:56:115.352 UTC ct1-production [Thread 140105745336256, Pid 48037]: -(-1):EvTcpConnectFail
2015-07-02 Thu 02:46:56:115.417 UTC ct1-production [Thread 140105066948352, Pid 48037]: Processing scm::EvSandeshSend in state Connect
2015-07-02 Thu 02:46:56:115.457 UTC ct1-production [Thread 140105066948352, Pid 48037]: Wrong state: Connect for event: EvSandeshSend
2015-07-02 Thu 02:46:56:115.488 UTC ct1-production [Thread 140105066948352, Pid 48037]: Processing scm::EvTcpConnectFail in state Connect
2015-07-02 Thu 02:46:56:115.516 UTC ct1-production [Thread 140105066948352, Pid 48037]: Connect : EvTcpConnectFail
2015-07-02 Thu 02:46:56:115.790 UTC ct1-production [Thread 140105066948352, Pid 48037]: Idle
2015-07-02 Thu 02:46:56:115.932 UTC ct1-production [Thread 140105066948352, Pid 48037]: Processing scm::EvSandeshSend in state Idle
h01)
* Thread 140105745336256, Pid 48037 <=== IMP
2015-07-01 Wed 13:35:54:287.037 UTC ct1-production [Thread 140105745336256, Pid 48037]: -(-1) OnSessionEvent TCP Connect Failed
2015-07-01 Wed 13:35:54:287.124 UTC ct1-production [Thread 140105745336256, Pid 48037]: -(-1):EvTcpConnectFail
2015-07-01 Wed 13:35:59:390.062 UTC ct1-production [Thread 140105745336256, Pid 48037]: Sandesh: Send: FAILED: 1435757759390034 XMPP [SYS_DEBUG]: XmppCreateConnection: Xmpp creating dynamic channel 10.140.192.142 controller/src/xmpp/xmpp_server.cc 273
2015-07-01 Wed 13:35:59:390.211 UTC ct1-production [Thread 140105745336256, Pid 48037]: Sandesh: Send: FAILED: 1435757759390195 XMPP [SYS_INFO]: XmppConnectionCreate: Created Xmpp Server connection from ct1-production To : controller/src/xmpp/xmpp_connection.cc 449
* This error resolved with "http socket open failed" error in following thread and some service restart.
2015-07-02 Thu 02:47:47:784.263 UTC ct1-production [Thread 140105568990976, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:47:52:784.758 UTC ct1-production [Thread 140105117329152, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:47:56:946.014 UTC ct1-production [Thread 140105108932352, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:47:58:784.978 UTC ct1-production [Thread 140105092138752, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:01:946.133 UTC ct1-production [Thread 140105062749952, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:06:169.145 UTC ct1-production [Thread 140105100535552, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:06:784.386 UTC ct1-production [Thread 140105100535552, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:07:945.701 UTC ct1-production [Thread 140105573189376, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:11:169.662 UTC ct1-production [Thread 140105108932352, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:15:945.382 UTC ct1-production [Thread 140105573189376, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:17:170.720 UTC ct1-production [Thread 140105568990976, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:18:784.435 UTC ct1-production [Thread 140105071146752, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:25:171.870 UTC ct1-production [Thread 140105071146752, Pid 48037]: http socket open failed: system:24
h1)
2015-07-02 Thu 02:47:47:704.928 UTC ct1-production [Thread 140105745336256, Pid 48037]: Sandesh: Send: FAILED: 1435805267704889 XMPP [SYS_DEBUG]: XmppCreateConnection: Xmpp creating dynamic channel 10.140.192.34 controller/src/xmpp/xmpp_server.cc 273
2015-07-02 Thu 02:47:47:705.155 UTC ct1-production [Thread 140105745336256, Pid 48037]: Sandesh: Send: FAILED: 1435805267705131 XMPP [SYS_INFO]: XmppConnectionCreate: Created Xmpp Server connection from ct1-production To : controller/src/xmpp/xmpp_connection.cc 449
2015-07-02 Thu 02:47:47:784.263 UTC ct1-production [Thread 140105568990976, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:47:52:784.758 UTC ct1-production [Thread 140105117329152, Pid 48037]: http socket open failed: system:24
h2)
2015-07-02 Thu 02:48:15:945.382 UTC ct1-production [Thread 140105573189376, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:17:170.720 UTC ct1-production [Thread 140105568990976, Pid 48037]: http socket open failed: system:24
2015-07-02 Thu 02:48:18:784.435 UTC ct1-production [Thread 140105071146752, Pid 48037]: http socket open failed: system:24
h3)
* Starting Bgp Server at port 179
2015-07-02 Thu 02:48:26:173.363 UTC ct1-production [Thread 139899567953856, Pid 61256]: Starting Bgp Server at port 179
2015-07-02 Thu 02:48:26:175.422 UTC ct1-production [Thread 139899567953856, Pid 61256]: DiscoveryClientMsg: publish/ct1-production xmpp-server <xmpp-server><ip-address>10.140.192.40</ip-address><port>5269</port></xmpp-server> controller/src/discovery/client/discovery_client.cc 362
2015-07-02 Thu 02:48:26:177.925 UTC ct1-production [Thread 139899567953856, Pid 61256]: SANDESH: ROLE : Generator
2015-07-02 Thu 02:48:26:177.949 UTC ct1-production [Thread 139899567953856, Pid 61256]: SANDESH: MODULE : ControlNode
2015-07-02 Thu 02:48:26:177.959 UTC ct1-production [Thread 139899567953856, Pid 61256]: SANDESH: SOURCE : ct1-production
2015-07-02 Thu 02:48:26:177.969 UTC ct1-production [Thread 139899567953856, Pid 61256]: SANDESH: NODE TYPE : Control
2015-07-02 Thu 02:48:26:177.978 UTC ct1-production [Thread 139899567953856, Pid 61256]: SANDESH: INSTANCE ID : 0
2015-07-02 Thu 02:48:26:177.988 UTC ct1-production [Thread 139899567953856, Pid 61256]: SANDESH: HTTP SERVER PORT : 8083
2015-07-02 Thu 02:48:26:182.747 UTC ct1-production [Thread 139899567953856, Pid 61256]: HTTP Introspect Init
2015-07-02 Thu 02:48:26:183.051 UTC ct1-production [Thread 139899567953856, Pid 61256]: Sandesh: Send: No client: 1435805306183029 TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8083 Initialization complete controller/src/io/tcp_server.cc 100
2015-07-02 Thu 02:48:26:183.124 UTC ct1-production [Thread 139899567953856, Pid 61256]: Sandesh Http Server Port 8083
20
h4)
* Note event change form Idle ===> ServerResolve ===> SsrcConnect ===> SsrcSslHandshake ===> SendNewSession ===> NewSessionResponseWait
SYS_DEBUG]: IFMapSmEventMessage: Processing ifsm::EvStart in state Idle controller/src/ifmap/client/ifmap_state_machine.cc 944
2015-07-02 Thu 02:48:26:195.302 UTC ct1-production [Thread 139899088918272, Pid 61256]: Sandesh: Send: FAILED: 1435805306195291 IFMapStateMachine [SYS_DEBUG]: IFMapSmTransitionMessage: Idle ===> ServerResolve controller/src/ifmap/client/ifmap_state_machine.cc 954
2015-07-02 Thu 02:48:26:195.350 UTC ct1-production [Thread 139899088918272, Pid 61256]: Sandesh: Send: FAILED: 1435805306195340 IFMapStateMachine [SYS_DEBUG]: IFMapSmStartTimerMessage: 5 second response timer started. controller/src/ifmap/client/ifmap_state_machine.cc 734
2015-07-02 Thu 02:48:26:198.877 UTC ct1-production [Thread 139899395806976, Pid 61256]: Sandesh: Send: FAILED: 1435805306198861 IFMapStateMachine [SYS_DEBUG]: IFMapSmTransitionMessage: ServerResolve ===> SsrcConnect controller/src/ifmap/client/ifmap_state_machine.cc 954
2015-07-02 Thu 02:48:26:198.921 UTC ct1-production [Thread 139899395806976, Pid 61256]: Sandesh: Send: FAILED: 1435805306198907 IFMapStateMachine [SYS_DEBUG]: IFMapSmStartTimerMessage: 5 second response timer started. controller/src/ifmap/client/ifmap_state_machine.cc 734
2015-07-02 Thu 02:48:26:200.457 UTC ct1-production [Thread 139898279413504, Pid 61256]: Sandesh: Send: FAILED: 1435805306200445 IFMapStateMachine [SYS_DEBUG]: IFMapSmTransitionMessage: SsrcConnect ===> SsrcSslHandshake controller/src/ifmap/client/ifmap_state_machine.cc 954
2015-07-02 Thu 02:48:26:212.918 UTC ct1-production [Thread 139898283611904, Pid 61256]: Sandesh: Send: FAILED: 1435805306212906 IFMapStateMachine [SYS_DEBUG]: IFMapSmTransitionMessage: SsrcSslHandshake ===> SendNewSession controller/src/ifmap/client/ifmap_state_machine.cc 954
015-07-02 Thu 02:48:26:213.195 UTC ct1-production [Thread 139899038537472, Pid 61256]: Sandesh: Send: FAILED: 1435805306213181 IFMapStateMachine [SYS_DEBUG]: IFMapSmTransitionMessage: SendNewSession ===> NewSessionResponseWait controller/src/ifmap/client/ifmap_state_machine.cc 954
h5)
* See (h1) "2015-07-02 Thu 02:47:47:704.928 UTC" there we can see same log but "http socket open failed: system:24" error after that.
2015-07-02 Thu 02:48:26:198.933 UTC ct1-production [Thread 139899567953856, Pid 61256]: Sandesh: Send: FAILED: 1435805306198916 XMPP [SYS_DEBUG]: XmppCreateConnection: Xmpp creating dynamic channel 10.140.192.50 controller/src/xmpp/xmpp_server.cc 273
2015-07-02 Thu 02:48:26:199.037 UTC ct1-production [Thread 139899567953856, Pid 61256]: Sandesh: Send: FAILED: 1435805306199021 XMPP [SYS_INFO]: XmppConnectionCreate: Created Xmpp Server connection from ct1-production To : controller/src/xmpp/xmpp_connection.cc 449
h6)
2015-07-02 Thu 02:48:26:311.040 UTC ct1-production [Thread 139899404203776, Pid 61256]: Sandesh: Send: FAILED: 1435805306311021 IFMap [SYS_DEBUG]: IFMapServerClientRegUnreg: Register request for client default-global-system-config:cp21-production controller/src/ifmap/ifmap_server.cc 215
h7)
* VrSubscribe and VmSubscribe
2015-07-02 Thu 02:48:26:311.180 UTC ct1-production [Thread 139899404203776, Pid 61256]: Sandesh: Send: FAILED: 1435805306311168 IFMapXMPP [SYS_DEBUG]: IFMapXmppVrSubUnsub: VrSubscribe ct1-production:10.140.192.145 controller/src/ifmap/ifmap_xmpp.cc 199
2015-07-02 Thu 02:48:26:311.246 UTC ct1-production [Thread 139899404203776, Pid 61256]: Sandesh: Send: FAILED: 1435805306311233 IFMapXMPP [SYS_DEBUG]: IFMapXmppVmSubUnsub: VmSubscribe ct1-production:10.140.192.145 2ede789b-f57b-4756-bcd2-849441f53937 controller/src/ifmap/ifmap_xmpp.cc 216
2015-07-02 Thu 02:48:26:324.149 UTC ct1-production [Thread 139899063727872, Pid 61256]: Sandesh: Send: FAILED: 1435805306324132 IFMapXMPP [SYS_DEBUG]: IFMapXmppVmSubUnsub: VmSubscribe ct1-production:10.140.192.145 d34097f0-c070-495a-97c7-eccddea1cdad controller/src/ifmap/ifmap_xmpp.cc 216
... ....
h8)
* resource not found when processing subscribe
2015-07-02 Thu 02:48:26:324.552 UTC ct1-production [Thread 139899067926272, Pid 61256]: Sandesh: Send: FAILED: 1435805306324540 BGP [SYS_WARN]: XmppPeerMembershipLog: XMPP Peer ct1-production:10.140.192.145 Routing Instance default-domain:217882181541:default-net:default-net not found when processing subscribe controller/src/bgp/bgp_xmpp_channel.cc 1679
2015-07-02 Thu 02:48:26:324.659 UTC ct1-production [Thread 139899067926272, Pid 61256]: Sandesh: Send: FAILED: 1435805306324646 BGP [SYS_WARN]: XmppPeerMembershipLog: XMPP Peer ct1-production:10.140.192.145 Routing Instance default-domain:248775291331:default-net:default-net not found when processing subscribe controller/src/bgp/bgp_xmpp_channel.cc 1679
2015-07-02 Thu 02:48:26:324.834 UTC ct1-production [Thread 139899067926272, Pid 61256]: Sandesh: Send: FAILED: 1435805306324821 BGP [SYS_WARN]: XmppPeerMembershipLog: XMPP Peer ct1-production:10.140.192.145 Routing Instance default-domain:jenkins:at_private:at_private not found when processing subscribe controller/src/bgp/bgp_xmpp_channel.cc 1679
2015-07-02 Thu 02:48:26:324.952 UTC ct1-production [Thread 139899067926272, Pid 61256]: Sandesh: Send: FAILED: 1435805306324941 BGP [SYS_WARN]: XmppPeerMembershipLog: XMPP Peer ct1-production:10.140.192.145 Routing Instance default-domain:jiocloud_compute_pm:network1:network1 not found when processing subscribe controller/src/bgp/bgp_xmpp_channel.cc 1679
... ...
h9)
Search for "Too many open files" in contrail-control.log-20150702.gz to find the reason of error "http socket open failed: system:24"
http://stackoverflow.com/questions/880557/socket-accept-too-many-open-files
2015-07-02 Thu 02:47:56:117.055 UTC ct1-production [Thread 140105121527552, Pid 48037]: CreateSMSession Open FAILED: Too many open files
#############################
Custom Search
Monday, July 13, 2015
openstack contrail can't ping to private ip, controller failed to collect VM data from ifmap
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment