cassandra cluster status when a node goes down

1)
When all 3 nodes in the cluster up

# nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens  Owns (effective)  Host ID                               Rack
UN  10.140.15.59  75.4 KB    256     100.0%            a07836a2-8c56-4da7-803d-4389f87a4854  rack1
UN  10.140.15.61  256.7 KB   256     100.0%            00c51f6a-2d08-4b27-81f9-b9754b6b0b66  rack1
UN  10.140.15.60  90.74 KB   256     100.0%            d89d9cd8-d154-43d9-97ec-6b827c9a7ced  rack1

2)
When the node 10.140.15.61 is down

# nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens  Owns (effective)  Host ID                               Rack
UN  10.140.15.59  75.4 KB    256     100.0%            a07836a2-8c56-4da7-803d-4389f87a4854  rack1
DN  10.140.15.61  256.7 KB   256     100.0%            00c51f6a-2d08-4b27-81f9-b9754b6b0b66  rack1
UN  10.140.15.60  90.74 KB   256     100.0%            d89d9cd8-d154-43d9-97ec-6b827c9a7ced  rack1

OpenStack Create tenant, user, and user-role

1)
$keystone tenant-create --name doc-team

2)
$keystone user-create --name docuser --tenant e232e9ee1b384873b0efeb9554997506 --pass docuser --enabled true

3)
$keystone role-list
$keystone user-role-list

4)
$keystone user-role-add --user 5902cb094e98425c8cd3c1fb877e53e1 --role 779cf32247d94bc3b3abc9adfd952b41 --tenant e232e9ee1b384873b0efeb9554997506

django How to create fixture

$python manage.py dumpdata | python -mjson.tool


$python manage.py dumpdata | python -mjson.tool > somefile.json

How to setup cassandra zookeeper and rabbitmq cluster

1)
Nodes:
ct1-multinode-cont : 11.120.15.59
ct2-multinode-cont : 11.120.15.60
ct3-multinode-cont : 11.120.15.61

2)
zookeeper cluster:

a)
config:

$vim /etc/zookeeper/conf/zoo.cfg

server.59=11.120.15.59:2888:3888
server.60=11.120.15.60:2888:3888
server.61=11.120.15.61:2888:3888

b)
Check cluster status

b1)
$ /usr/share/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Mode: follower

b2)
$ /usr/share/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Mode: leader
 
b3)
From  11.120.15.61



$ /usr/share/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Mode: follower

c)
docs:
http://myjeeva.com/zookeeper-cluster-setup.html

http://hanishblogger.blogspot.in/2013/11/zookeeper-setup-in-distributed-mode.html

3)
cassandra cluster:

a)
config:

$/etc/cassandra/cassandra.yaml
cluster_name: 'contrail'

seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: 11.120.15.59,11.120.15.60,11.120.15.61
b)
Check cluster status  
 
# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens  Owns (effective)  Host ID                               Rack
UN  11.120.15.59  75.4 KB    256     100.0%            a07836a2-8c56-4da7-803d-4389f87a4854  rack1
UN  11.120.15.61  256.7 KB   256     100.0%            00c51f6a-2d08-4b27-81f9-b9754b6b0b66  rack1
UN  11.120.15.60  90.74 KB   256     100.0%            d89d9cd8-d154-43d9-97ec-6b827c9a7ced  rack1


c)
docs:
https://docs.datastax.com/en/cassandra/2.0/cassandra/initialize/initializeSingleDS.html

4)
rabbitmq cluster config:

a)
config:
http://blog.hemantthorat.com/setup-rabbitmq-clusters-on-ubuntu/#.Vm-7a5PoEc0

b)
Check cluster status  
 
$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@ct1-multinode-cont' ...
[{nodes,[{disc,['rabbit@ct1-multinode-cont','rabbit@ct2-multinode-cont',
                'rabbit@ct3-multinode-cont']}]},
 {running_nodes,['rabbit@ct2-multinode-cont','rabbit@ct3-multinode-cont',
                 'rabbit@ct1-multinode-cont']},
 {cluster_name,<<"rabbit@localhost">>},
 {partitions,[]}]

How to install python module into /usr/lib or /usr/local/lib

$python setup.py install --prefix=/usr

$python setup.py install --prefix=/usr/local


https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme

Nova boot with glance image fails Corrupt image download. Checksum was

1)
#nova boot --flavor 2 --image 61bb862c-69f1-49cc-9d37-1dbbf3eb841e --nic net-id=146520dc-2d6b-4ba1-b803-2edd1906a6f3 vm2

#nova boot --flavor 2 --image 61bb862c-69f1-49cc-9d37-1dbbf3eb841e --nic net-id=146520dc-2d6b-4ba1-b803-2edd1906a6f3 v1 --availability-zone nova:cp1-multinode-cont

2)
Fix:
#vim /usr/lib/python2.7/dist-packages/glanceclient/common/http.py

class HTTPClient(object):

    def __init__(self, endpoint, **kwargs):
        if self.endpoint.startswith("https"):
            #compression = kwargs.get('ssl_compression', True)
            compression = True

* Comment the line "compression = kwargs.get('ssl_compression', True)", then set "compression = True"

* Restart glance-api, glance-registry, cinder-api, cinder-volume, cinder-scheduler

3)
From /var/log/upstart/nova-compute.log

2015-08-07 01:43:02.100 18716 ERROR nova.compute.manager [req-66173648-5d32-4ab4-b647-6f571e5a2b76 5f51a7daa18f4465936fbb440ad05471 76cfebd8f2974dad94fd8770da958663 - - -] [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e] Instance failed to spawn
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e] Traceback (most recent call last):
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2473, in _build_resources
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     yield resources
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2345, in _build_and_run_instance
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     block_device_info=block_device_info)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2384, in spawn
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     admin_pass=admin_password)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2782, in _create_image
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     instance, size, fallback_from_host)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5900, in _try_fetch_image_cache
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     size=size)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     *args, **kwargs)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 480, in create_image
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     prepare_template(target=base, max_size=size, *args, **kwargs)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     return f(*args, **kwargs)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     fetch_func(target=target, *args, **kwargs)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 507, in fetch_image
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     max_size=max_size)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in fetch_to_raw
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     max_size=max_size)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 77, in fetch
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     IMAGE_API.download(context, image_href, dest_path=path)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 182, in download
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     dst_path=dest_path)
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 362, in download
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     for chunk in image_chunks:
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]   File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 398, in integrity_iter
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]     (md5sum, checksum))
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e] IOError: [Errno 32] Corrupt image download. Checksum was d41d8cd98f00b204e9800998ecf8427e expected 133eae9fb1c98f45894a4e60d8736619
2015-08-07 01:43:02.100 18716 TRACE nova.compute.manager [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e]
2015-08-07 01:43:02.104 18716 INFO nova.compute.manager [req-66173648-5d32-4ab4-b647-6f571e5a2b76 5f51a7daa18f4465936fbb440ad05471 76cfebd8f2974dad94fd8770da958663 - - -] [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e] Terminating instance
2015-08-07 01:43:02.109 18716 INFO nova.virt.libvirt.driver [-] [instance: cd3fa491-fd04-4be8-87f1-27738ee8285e] During wait destroy, instance disappeared.

cinder create volume from image JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1)
# cinder create --image-id 8f980b10-b491-41c3-bd48-a84001929840 --display-name cirros1 1
ERROR: Internal Server Error (HTTP 500) (Request-ID: req-410318bc-9a39-4cbc-8401-2037ec5d1d53)

2)
Fix:
#vim /usr/lib/python2.7/dist-packages/glanceclient/common/http.py

class HTTPClient(object):

    def __init__(self, endpoint, **kwargs):
        if self.endpoint.startswith("https"):
            #compression = kwargs.get('ssl_compression', True)
            compression = True

* Comment the line "compression = kwargs.get('ssl_compression', True)", then set "compression = True"

* Restart glance-api, glance-registry, cinder-api, cinder-volume, cinder-scheduler

3)
/var/log/upstart/cinder-api.log

2015-08-06 14:23:26.296 36113 DEBUG glanceclient.common.http [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -] curl -i -X GET -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' -H 'Accept-Encoding: gzip, deflate, compress' -H 'Accept: */*' -H 'X-Auth-Token: {SHA1}e6717209a08a3b895581176f415a3b34c8264e40'  --cert None --key None glance+https://image.jiocloud.com:9292/v2/images/8f980b10-b491-41c3-bd48-a84001929840 log_curl_request /usr/lib/python2.7/dist-packages/glanceclient/common/http.py:122
2015-08-06 14:23:26.369 36113 DEBUG glanceclient.common.http [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -]
HTTP/1.1 200 OK
content-length: 699
via: 1.1 image.jiocloud.com:9292
access-control-expose-headers: Accept, Content-Type, X-Auth-Token, X-Subject-Token
server: Apache/2.4.7 (Ubuntu)
access-control-allow-headers: Accept, Content-Type, X-Auth-Token, X-Subject-Token
connection: close
date: Thu, 06 Aug 2015 14:23:26 GMT
access-control-allow-origin: *
access-control-allow-methods: GET POST OPTIONS PUT DELETE PATCH
content-type: application/json; charset=UTF-8
x-openstack-request-id: req-7670140c-8334-4148-b9f3-d3d9845c11c9
 log_http_response /usr/lib/python2.7/dist-packages/glanceclient/common/http.py:135
2015-08-06 14:23:26.371 36113 DEBUG taskflow.engines.action_engine.runner [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -] Exiting old state 'SCHEDULING' in response to event 'wait_finished' on_exit /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:201
2015-08-06 14:23:26.371 36113 DEBUG taskflow.engines.action_engine.runner [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -] Entering new state 'WAITING' in response to event 'wait_finished' on_enter /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:205
2015-08-06 14:23:26.371 36113 DEBUG taskflow.engines.action_engine.runner [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -] Exiting old state 'WAITING' in response to event 'examine_finished' on_exit /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:201
2015-08-06 14:23:26.372 36113 DEBUG taskflow.engines.action_engine.runner [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -] Entering new state 'ANALYZING' in response to event 'examine_finished' on_enter /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:205
2015-08-06 14:23:26.372 36113 WARNING cinder.volume.api [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -] Task 'cinder.volume.flows.api.create_volume.ExtractVolumeRequestTask;volume:create' (14e19665-b4a7-4f40-8ce9-1aa00d1e374a) transitioned into state 'FAILURE' from state 'RUNNING'
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api Traceback (most recent call last):
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     result = task.execute(**arguments)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/api/create_volume.py", line 417, in execute
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     self._check_image_metadata(context, image_id, size)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/api/create_volume.py", line 251, in _check_image_metadata
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     image_meta = self.image_service.show(context, image_id)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 237, in show
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     _reraise_translated_image_exception(image_id)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 235, in show
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     image = self._client.call(context, 'get', image_id)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 176, in call
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     return getattr(client.images, method)(*args, **kwargs)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 102, in get
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     resp, body = self.http_client.get(url)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 261, in get
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     return self._request('GET', url, **kwargs)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 248, in _request
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     body_iter = resp.json()
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 741, in json
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     return json.loads(self.text, **kwargs)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 488, in loads
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     return _default_decoder.decode(s)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 370, in decode
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     obj, end = self.raw_decode(s)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api   File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 389, in raw_decode
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api     return self.scan_once(s, idx=_w(s, idx).end())
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2015-08-06 14:23:26.372 36113 TRACE cinder.volume.api
2015-08-06 14:23:26.374 36113 DEBUG taskflow.engines.action_engine.runner [req-ddb757de-85f5-4a1e-9999-36bff3afae7f - - - - -] Discarding failure 'Failure: simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)' (in response to event 'executed') under completion units request during completion of node 'cinder.volume.flows.api.create_volume.ExtractVolumeRequestTask;volume:create==1.0' (intention is to REVERT) analyze /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:181

python ctypes example load librbd

# python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>>
>>>
>>> from ctypes.util import find_library
>>> librbd_path = find_library('rbd')
>>> librbd_path
'librbd.so.1'
>>>
>>>
>>> from ctypes import CDLL
>>>
>>> rbdd = CDLL(librbd_path)
>>>
>>>
>>> rbdd

>>>
>>>
>>> dir(rbdd)
['_FuncPtr', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_func_flags_', '_func_restype_', '_handle', '_name']
>>> 
>>>
>>> rbdd.rbd_write
<_funcptr 0x7f1475cff460="" at="" object="">
>>> 
>>>
>>> help(rbdd.rbd_write)
>>>
>>>

How to ceph create auth

1)
#ceph auth list   

2)   
#ceph auth get-or-create client.glance mon "allow r" osd "allow class-read object_prefix rbd_children, allow rwx pool=images"

[client.glance]
    key = AQD1mLtVi9eQCRAAgR6zX/XvWyiac6A/OyRg8w==

3)   
#ceph auth list