nova-compute, fails with Timeout: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""

============ISSUE==========

nova-compute, fails with Timeout: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""

============FIX==========

* In compute node, open /etc/oslo/matchmaker_ring.json and remove invalid nodes from "conductor": and other sections.

$cat /etc/oslo/matchmaker_ring.json

{
  "cert": [
    "oc1-test-dev-test",
    "ocdb1-test-dev-test"
  ],
  "consoleauth": [
    "oc1-test-dev-test",
    "ocdb1-test-dev-test"
  ],
  "conductor": [
    "oc1-test-dev-test",
    "ocdb1-test-dev-test"
  ],
  "cinder-scheduler": [
    "oc1-test-dev-test",
    "ocdb1-test-dev-test"
  ],
  "compute": [
    "gcp1-test-dev-test"
  ],
  "cinder-volume": [
    "oc1-test-dev-test",
    "ocdb1-test-dev-test"
  ],
  "cinder-volume:oc1-test-dev-test": [
    "oc1-test-dev-test"
  ],
  "cinder-volume:ocdb1-test-dev-test": [
    "ocdb1-test-dev-test"
  ],
  "scheduler": [
    "oc1-test-dev-test",
    "ocdb1-test-dev-test"
  ]
}

================ERROR1=============

$/var/log/nova/nova-compute.log

2015-06-11 16:11:36.901 55059 TRACE nova.servicegroup.drivers.db
2015-06-11 16:11:36.904 55059 WARNING nova.openstack.common.loopingcall [req-928ef316-ceb4-4e69-8e08-a965dcafe37f - - - - -] task > run outlasted interval by 50.01 sec
2015-06-11 16:11:36.906 55059 WARNING oslo_messaging._drivers.impl_zmq [req-928ef316-ceb4-4e69-8e08-a965dcafe37f - - - - -] rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing is deprecated. It is suggested to change the value to ring.
2015-06-11 16:11:36.923 55059 ERROR nova.servicegroup.drivers.db [req-928ef316-ceb4-4e69-8e08-a965dcafe37f - - - - -] Recovered model server connection!
2015-06-11 16:11:46.906 55059 WARNING oslo_messaging._drivers.impl_zmq [req-928ef316-ceb4-4e69-8e08-a965dcafe37f - - - - -] rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing is deprecated. It is suggested to change the value to ring.
2015-06-11 16:12:04.374 55059 INFO nova.openstack.common.service [req-ea1d6e3f-e2e1-4edf-b5a4-1103fb679032 - - - - -] Caught SIGTERM, exiting
2015-06-11 16:12:56.826 18384 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/dist-packages/nova/servicegroup/api.py:68
2015-06-11 16:12:56.866 18384 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-06-11 16:12:56.885 18384 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/dist-packages/nova/servicegroup/api.py:68
2015-06-11 16:12:56.888 18384 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2015-06-11 16:12:56.963 18384 WARNING oslo_messaging._drivers.impl_zmq [req-d4cefc49-632f-4150-8d76-51a6786a4622 - - - - -] rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing is deprecated. It is suggested to change the value to ring.
2015-06-11 16:24:40.795 58030 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/dist-packages/nova/servicegroup/api.py:68
2015-06-11 16:24:40.834 58030 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-06-11 16:24:40.853 58030 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/dist-packages/nova/servicegroup/api.py:68
2015-06-11 16:24:40.855 58030 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2015-06-11 16:24:40.930 58030 WARNING oslo_messaging._drivers.impl_zmq [req-c3c40f37-5f62-4c5a-9fce-14a87cd262e6 - - - - -] rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing is deprecated. It is suggested to change the value to ring.
2015-06-11 16:25:40.935 58030 CRITICAL nova [req-c3c40f37-5f62-4c5a-9fce-14a87cd262e6 - - - - -] Timeout: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""
2015-06-11 16:25:40.935 58030 TRACE nova Traceback (most recent call last):
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/bin/nova-compute", line 10, in
2015-06-11 16:25:40.935 58030 TRACE nova     sys.exit(main())
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/compute.py", line 72, in main
2015-06-11 16:25:40.935 58030 TRACE nova     db_allowed=CONF.conductor.use_local)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 277, in create
2015-06-11 16:25:40.935 58030 TRACE nova     db_allowed=db_allowed)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 157, in __init__
2015-06-11 16:25:40.935 58030 TRACE nova     self.conductor_api.wait_until_ready(context.get_admin_context())
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 292, in wait_until_ready
2015-06-11 16:25:40.935 58030 TRACE nova     timeout=timeout)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/baserpc.py", line 62, in ping
2015-06-11 16:25:40.935 58030 TRACE nova     return cctxt.call(context, 'ping', arg=arg_p)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-06-11 16:25:40.935 58030 TRACE nova     retry=self.retry)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-06-11 16:25:40.935 58030 TRACE nova     timeout=timeout, retry=retry)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1033, in wrap
2015-06-11 16:25:40.935 58030 TRACE nova     return func(self, *args, **kws)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1065, in send
2015-06-11 16:25:40.935 58030 TRACE nova     return self._send(target, ctxt, message, wait_for_reply, timeout)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1055, in _send
2015-06-11 16:25:40.935 58030 TRACE nova     pooled=True)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 856, in _multi_send
2015-06-11 16:25:40.935 58030 TRACE nova     envelope, allowed_remote_exmods, pooled)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 781, in _call
2015-06-11 16:25:40.935 58030 TRACE nova     msg = msg_waiter.recv()
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 228, in recv
2015-06-11 16:25:40.935 58030 TRACE nova     return self.sock.recv_multipart(**kwargs)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 359, in recv_multipart
2015-06-11 16:25:40.935 58030 TRACE nova     return _Socket_recv_multipart(self, flags, copy, track)
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/zmq/sugar/socket.py", line 265, in recv_multipart
2015-06-11 16:25:40.935 58030 TRACE nova     parts = [self.recv(flags, copy=copy, track=track)]
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 338, in recv
2015-06-11 16:25:40.935 58030 TRACE nova     self._eventlet_recv_event.block()
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 104, in block
2015-06-11 16:25:40.935 58030 TRACE nova     self._hub.switch()
2015-06-11 16:25:40.935 58030 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-06-11 16:25:40.935 58030 TRACE nova     return self.greenlet.switch()
2015-06-11 16:25:40.935 58030 TRACE nova Timeout: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""
2015-06-11 16:25:40.935 58030 TRACE nova
2015-06-11 16:26:26.905 64005 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/dist-packages/nova/servicegroup/api.py:68
2015-06-11 16:26:26.934 64005 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-06-11 16:26:26.948 64005 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/dist-packages/nova/servicegroup/api.py:68
2015-06-11 16:26:26.950 64005 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2015-06-11 16:26:27.007 64005 WARNING oslo_messaging._drivers.impl_zmq [req-c2be6b2c-2e4b-48f2-b2d2-395df63250e9 - - - - -] rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing is deprecated. It is suggested to change the value to ring.
2015-06-11 16:27:27.011 64005 CRITICAL nova [req-c2be6b2c-2e4b-48f2-b2d2-395df63250e9 - - - - -] Timeout: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""


================ERROR2=============


2015-06-11 16:09:26.889 55059 TRACE nova.servicegroup.drivers.db
2015-06-11 16:09:26.892 55059 WARNING nova.openstack.common.loopingcall [req-928ef316-ceb4-4e69-8e08-a965dcafe37f - - - - -] task > run outlasted interval by 50.01 sec
2015-06-11 16:09:26.895 55059 WARNING oslo_messaging._drivers.impl_zmq [req-928ef316-ceb4-4e69-8e08-a965dcafe37f - - - - -] rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing is deprecated. It is suggested to change the value to ring.
2015-06-11 16:10:14.236 55059 ERROR nova.openstack.common.periodic_task [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Error during ComputeManager._run_pending_deletes: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py", line 224, in run_periodic_tasks
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     task(self, context)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6543, in _run_pending_deletes
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     context, filters, expected_attrs=attrs, use_slave=True)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 161, in wrapper
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     args, kwargs)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     objver=objver, args=args, kwargs=kwargs)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     retry=self.retry)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     timeout=timeout, retry=retry)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1033, in wrap
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     return func(self, *args, **kws)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1065, in send
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     return self._send(target, ctxt, message, wait_for_reply, timeout)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1055, in _send
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     pooled=True)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 856, in _multi_send
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     envelope, allowed_remote_exmods, pooled)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 781, in _call
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     msg = msg_waiter.recv()
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 228, in recv
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     return self.sock.recv_multipart(**kwargs)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 359, in recv_multipart
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     return _Socket_recv_multipart(self, flags, copy, track)
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/zmq/sugar/socket.py", line 265, in recv_multipart
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     parts = [self.recv(flags, copy=copy, track=track)]
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 338, in recv
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     self._eventlet_recv_event.block()
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 104, in block
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     self._hub.switch()
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task     return self.greenlet.switch()
2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""


================ERROR3=============


2015-06-11 16:10:14.236 55059 TRACE nova.openstack.common.periodic_task
2015-06-11 16:10:14.240 55059 DEBUG nova.openstack.common.periodic_task [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:219
2015-06-11 16:10:14.240 55059 DEBUG nova.openstack.common.periodic_task [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Running periodic task ComputeManager._run_image_cache_manager_pass run_periodic_tasks /usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:219
2015-06-11 16:10:14.242 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Acquired file lock "/var/lib/nova/instances/locks/nova-storage-registry-lock" after waiting 0.000s acquire /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212
2015-06-11 16:10:14.242 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Lock "storage-registry-lock" acquired by "do_register_storage_use" :: waited 0.001s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
2015-06-11 16:10:14.244 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Releasing file lock "/var/lib/nova/instances/locks/nova-storage-registry-lock" after holding it for 0.002s release /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-06-11 16:10:14.244 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Lock "storage-registry-lock" released by "do_register_storage_use" :: held 0.002s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-06-11 16:10:14.245 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Acquired file lock "/var/lib/nova/instances/locks/nova-storage-registry-lock" after waiting 0.000s acquire /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212
2015-06-11 16:10:14.246 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Lock "storage-registry-lock" acquired by "do_get_storage_users" :: waited 0.001s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
2015-06-11 16:10:14.247 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Releasing file lock "/var/lib/nova/instances/locks/nova-storage-registry-lock" after holding it for 0.001s release /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-06-11 16:10:14.248 55059 DEBUG oslo_concurrency.lockutils [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] Lock "storage-registry-lock" released by "do_get_storage_users" :: held 0.002s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-06-11 16:10:14.250 55059 WARNING oslo_messaging._drivers.impl_zmq [req-db4c5598-dda4-455a-b2de-e9f3693cf03f - - - - -] rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing is deprecated. It is suggested to change the value to ring.
2015-06-11 16:10:14.252 55059 DEBUG oslo_concurrency.lockutils [-] Lock "fd97cc9e-dae2-4878-80a7-c2049ec3fcf5" released by "query_driver_power_state_and_sync" :: held 60.021s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-06-11 16:10:14.253 55059 ERROR nova.compute.manager [-] [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5] Periodic sync_power_state task had an error while processing an instance.
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5] Traceback (most recent call last):
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6042, in _sync
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     query_driver_power_state_and_sync()
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     return f(*args, **kwargs)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6039, in query_driver_power_state_and_sync
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     self._query_driver_power_state_and_sync(context, db_instance)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6079, in _query_driver_power_state_and_sync
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     use_slave=True)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6095, in _sync_instance_power_state
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     db_instance.refresh(use_slave=use_slave)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 192, in wrapper
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     self._context, self, fn.__name__, args, kwargs)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 340, in object_action
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     objmethod=objmethod, args=args, kwargs=kwargs)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     retry=self.retry)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     timeout=timeout, retry=retry)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1033, in wrap
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     return func(self, *args, **kws)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1065, in send
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     return self._send(target, ctxt, message, wait_for_reply, timeout)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 1055, in _send
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     pooled=True)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 856, in _multi_send
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     envelope, allowed_remote_exmods, pooled)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 781, in _call
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     msg = msg_waiter.recv()
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 228, in recv
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     return self.sock.recv_multipart(**kwargs)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 359, in recv_multipart
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     return _Socket_recv_multipart(self, flags, copy, track)
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/zmq/sugar/socket.py", line 265, in recv_multipart
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     parts = [self.recv(flags, copy=copy, track=track)]
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 338, in recv
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     self._eventlet_recv_event.block()
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 104, in block
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     self._hub.switch()
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5]     return self.greenlet.switch()
2015-06-11 16:10:14.253 55059 TRACE nova.compute.manager [instance: fd97cc9e-dae2-4878-80a7-c2049ec3fcf5] Timeout: Timeout while waiting on RPC response - topic: "", RPC method: "" info: ""