fosshelp: May 2014

Friday, May 30, 2014

OpenStack Devstack Matchmaker unregistered: ceilometer.ubuntu, ubuntu

1)
Getting error from following services
ceilometer-acompute
ceilometer-acentral
ceilometer-collector
ceilometer-alarm-notifier

---- Error ----

2014-05-30 19:38:35.623 12035 TRACE ceilometer.openstack.common.threadgroup     return execute(conn, stack, raise_on_error)
2014-05-30 19:38:35.623 12035 TRACE ceilometer.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/client.py", line 2070, in _execute_transaction
2014-05-30 19:38:35.623 12035 TRACE ceilometer.openstack.common.threadgroup     self.raise_first_error(commands, response)
2014-05-30 19:38:35.623 12035 TRACE ceilometer.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/client.py", line 2106, in raise_first_error
2014-05-30 19:38:35.623 12035 TRACE ceilometer.openstack.common.threadgroup     raise r
2014-05-30 19:38:35.623 12035 TRACE ceilometer.openstack.common.threadgroup ResponseError: Command # 1 (SADD ceilometer.agent.compute.ubuntu ceilometer.agent.compute.ubuntu.ubuntu) of pipeline caused error: Operation against a key holding the wrong kind of value
2014-05-30 19:38:35.623 12035 TRACE ceilometer.openstack.common.threadgroup
2014-05-30 19:38:35.629 12035 INFO ceilometer.openstack.common.rpc.matchmaker [-] Matchmaker unregistered: ceilometer.ubuntu, ubuntu

Fix
====
http://fosshelp.blogspot.com/2014/05/openstack-devstack-responseerror.html

OpenStack DevStack No such file or directory: '/etc/oslo/matchmaker_ring.json'

Context
========
Enabled "ZeroMQ"
http://fosshelp.blogspot.in/2014/05/how-to-openstack-devstack-setup-with.html
http://fosshelp.blogspot.com/2014/05/openstack-devstack-responseerror.html

1)
Getting error from following services
n-cond
n-crt
n-net
n-sch
n-cauth
h-eng

---- Error ----

2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup     _get_matchmaker().register(topic, CONF.rpc_zmq_host)
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/rpc/impl_zmq.py", line 817, in _get_matchmaker
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup     matchmaker = importutils.import_object(mm, *args, **kwargs)
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/importutils.py", line 40, in import_object
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup     return import_class(import_str)(*args, **kwargs)
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/rpc/matchmaker_ring.py", line 106, in __init__
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup     self.add_binding(mm.FanoutBinding(), FanoutRingExchange(ring))
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/rpc/matchmaker_ring.py", line 88, in __init__
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup     super(FanoutRingExchange, self).__init__(ring)
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/rpc/matchmaker_ring.py", line 57, in __init__
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup     fh = open(CONF.matchmaker_ring.ringfile, 'r')
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup IOError: [Errno 2] No such file or directory: '/etc/oslo/matchmaker_ring.json'
2014-05-30 18:59:11.648 TRACE nova.openstack.common.threadgroup

Fix
=====

Notes:
-----------
*
http://ewindisch.github.io/nova/ <=== IMP
The MatchMaker is pluggable and may also be used to enforce sending messages through a BROKER.
MatchMakerRing - loads a static hash table from a JSON file, cycles hosts per bare topic to create a directed topic.
The MatchMaker ring is a JSON file containing a hash where each key is a base topic and the values are arrays.
http://ewindisch.github.io/nova/#cmnt7 <=== IMP

a)
#sudo mkdir /etc/oslo
#sudo touch /etc/oslo/matchmaker_ring.json

b)
b1)
* Open /etc/oslo/matchmaker_ring.json and add following json string
    {
        "conductor":"conductor"
    }

b2)
Goto screen and check/restart "n-cond" services. Now "n-cond" service should start without any issue.

c)
Do the same for all other services where we getting this error of ZeroMQ

d)
Example of /etc/oslo/matchmaker_ring.json
    {
        "conductor":["controller","compute1","compute2","compute2"],
        "console":["controller","compute1","compute2","compute2"],
        "consoleauth":["controller","compute1","compute2","compute2"],
        "compute":["controller","compute1","compute2","compute2"],
        "cert":["controller","compute1","compute2","compute2"],
        "scheduler":["controller","compute1","compute2","compute3"],
        "metering": ["ceilometer1"],
        "notifications-info": ["ceilometer1"],
        "notifications-error": ["ceilometer1"]
    }

OpenStack DevStack ResponseError pipeline caused error: Operation against a key holding the wrong kind of value

Context
========
Enabled "ZeroMQ"
http://fosshelp.blogspot.in/2014/05/how-to-openstack-devstack-setup-with.html

1)
Getting error from following services
n-cond
n-crt
n-net
n-sch
n-cauth
h-eng

----- Error -----

2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/service.py", line 65, in run_service
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     service.start()
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/service.py", line 179, in start
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     self.conn.create_consumer(node_topic, rpc_dispatcher, fanout=False)
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/rpc/impl_zmq.py", line 578, in create_consumer
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     _get_matchmaker().register(topic, CONF.rpc_zmq_host)
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/rpc/matchmaker.py", line 205, in register
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     self.backend_register(key, key_host)
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/rpc/matchmaker_redis.py", line 138, in backend_register
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     pipe.execute()
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/client.py", line 2155, in execute
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     return execute(conn, stack, raise_on_error)
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/client.py", line 2070, in _execute_transaction
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     self.raise_first_error(commands, response)
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/client.py", line 2106, in raise_first_error
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup     raise r
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup ResponseError: Command # 1 (SADD scheduler.ubuntu scheduler.ubuntu.ubuntu) of pipeline caused error: Operation against a key holding the wrong kind of value
2014-05-30 18:48:45.493 TRACE nova.openstack.common.threadgroup

Fix
=========

a)
Goto vim /etc/nova/nova.conf
Then change

rpc_zmq_matchmaker = nova.openstack.common.rpc.matchmaker_redis.MatchMakerRedis

To

rpc_zmq_matchmaker = nova.openstack.common.rpc.matchmaker_ring.MatchMakerRin
b)
Don't run .unstack.sh and /stack.sh, since that remove our change from /etc/nova/nova.conf

c)
Goto screen and select one service where we getting this error (Eg:n-cond).
Then Press "Ctrl + c" to Stop that service.
Then Press "UP arrow" and press enter to Start that service again

Now you may get this error:
http://fosshelp.blogspot.com/2014/05/openstack-devstack-no-such-file-or.html

Notes:
--------
http://ewindisch.github.io/nova/ <=== IMP
The MatchMaker is pluggable and may also be used to enforce sending messages through a BROKER.
MatchMakerRing - loads a static hash table from a JSON file, cycles hosts per bare topic to create a directed topic.
The MatchMaker ring is a JSON file containing a hash where each key is a base topic and the values are arrays.
http://ewindisch.github.io/nova/#cmnt7

How to OpenStack DevStack Setup with ZeroMQ

1)
If you are using stable havana
https://github.com/openstack-dev/devstack.git -b stable/havana

a)
Add followinf lines in localrc
===============================

##### Message Queue #####
disable_service rabbit
disable_service qpid
enable_service zeromq

##Note: Remove OFFLINE=True or set OFFLINE=False

b)
Install Dependencies
#sudo easy_install redis
or
#pip install redis

http://fosshelp.blogspot.in/2014/05/openstack-devstack-importerror-failed.html

c)
run ./stack.sh

d)
Check for the sockets cretated by ZeroMQ and confirm that ZeroMQ is running.
$ ls /var/run/openstack/
zmq_topic_cert.ubuntu
zmq_topic_cinder-scheduler.ubuntu
zmq_topic_cinder-volume.ubuntu
zmq_topic_conductor.ubuntu

* Form nova.conf, rpc_zmq_ipc_dir = /var/run/openstack ==> (StrOpt) Directory for holding IPC sockets.

e)
Check and confirm that "zeromq" screen window exist.

f)
Location of ZeroMQ binary

/usr/local/bin/nova-rpc-zmq-receiver

g)
DevStack using following command to run ZeroMQ

$ cd /opt/stack/nova && /usr/local/bin/nova-rpc-zmq-receiver & echo $! >/opt/stack/status/stack/zeromq.pid; fg || echo "zeromq failed to start" | tee "/opt/stack/status/stack/zeromq.failure"

h)
Configure ZeroMQ
http://docs.openstack.org/trunk/config-reference/content/configuring-rpc.html

Openstack Devstack ImportError: Failed to import module redis

Fix
=====
#sudo easy_install redis
or
#pip install redis

a)
Error From heat engine log
======================
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup     _get_matchmaker().register(topic, CONF.rpc_zmq_host)
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup   File "/opt/stack/heat/heat/openstack/common/rpc/impl_zmq.py", line 817, in _get_matchmaker
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup     matchmaker = importutils.import_object(mm, *args, **kwargs)
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup   File "/opt/stack/heat/heat/openstack/common/importutils.py", line 40, in import_object
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup     return import_class(import_str)(*args, **kwargs)
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup   File "/opt/stack/heat/heat/openstack/common/rpc/matchmaker_redis.py", line 96, in __init__
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup     raise ImportError("Failed to import module redis.")
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup ImportError: Failed to import module redis.
2014-05-30 16:34:59.954 TRACE heat.openstack.common.threadgroup

b)
https://github.com/openstack/heat/blob/master/heat/openstack/common/rpc/matchmaker_redis.py
=========================

redis = importutils.try_import('redis')

class MatchMakerRedis(mm_common.HeartbeatMatchMakerBase):
    """MatchMaker registering and looking-up hosts with a Redis server."""
    def __init__(self):
        super(MatchMakerRedis, self).__init__()

        if not redis:
            raise ImportError("Failed to import module redis.")

c)
./stack.sh output
=========================
2014-05-30 16:32:13.059 | libzmq1 is already the newest version.
2014-05-30 16:32:13.067 | python-zmq is already the newest version.
2014-05-30 16:32:13.071 | redis-server is already the newest version.

d)
saju@ubuntu:~/openstack/devstack$ sudo apt-get install redis-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
redis-server is already the newest version.
The following packages were automatically installed and are no longer required:
libboost-filesystem1.46.1 libboost-system1.46.1 libboost-program-options1.46.1 libpcrecpp0 libboost-thread1.46.1
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 95 not upgraded.
saju@ubuntu:~/openstack/devstack$

e)
saju@ubuntu:~/openstack/devstack$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import redis
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named redis
>>>