How to Jenkins Create a Debian Package for OpenStack python-heatclient

1)
Install Jenkins and Plugins
http://fosshelp.blogspot.in/2014/04/install-jenkins-and-github-plugin.html

2)
Get debian packaging script for python-heatclient project.
#sudo apt-get install devscripts

Goto following link and locate your project's *.dsc file
http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/

Download the packaging script and extract it
#dget http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/p/python-heatclient/python-heatclient_0.2.8-0ubuntu1~cloud0.dsc
#tar -xzf python-heatclient_0.2.8-0ubuntu1~cloud0.debian.tar.gz
#cd debian
#ls



3)
Clone the source code of python-heatclient
#git clone https://github.com/sajuptpm/python-heatclient.git
#cd python-heatclient

Create a new branch named "packaging" if not exist and switch to that branch.
#git checkout -b packaging

4)
Copy debian folder (packaging script) to "packaging" branch
#cp -r ../../debian .
#git status
#git add debian
#git commit debian
#git status

5)
Push packaging branch to remote to remote github repo
#git remote -v
#git push origin packaging

6)
Create build job in Jenkins

a)
Goto "New Item" --> type name for the new job/item --> Select "Build a free-style software project"

b)
* Under "Source Code Management" ---> Select "Git" ---> Type Repository URL (eg:https://github.com/sajuptpm/python-heatclient.git)

* Credentials --> Click on "Add" button --> Select kind "username and password" --> Give your github username and password -> Click "Add"

* Branches to build --> Type "origin/master" or "*/master"

c)
Under "Additional Behaviours" Section:
* Check out to a sub-directory --> Local subdirectory for repo : "build"

* Strategy for choosing what to build -->     Choosing strategy    --> "default"

d)
Under "Build Triggers" Section:
* Select "Build when a change is pushed to GitHub"

e)
Add following script Under "Build" Section:
* Excecute Shell --> Command
-----------------------
#!/bin/bash -xe
rm -f *.deb
cd build
if ! git checkout packaging
then
    git checkout -b packaging origin/packaging
fi
git checkout packaging
dpkg-buildpackage -b
git push origin packaging || true

f)
Under "Post-build Actions" Section:
* Archive the artifacts --> Files to archive --> "*.deb"

g)
Save

7)
Build the package

a) 
Job Workspace:
* if you need to try build with fresh copy of repo, delete the workspace and run build again.
* In workspace you can see the cloned repo, build directory and debian package
* You can also download the workspace as a zip file
http://192.168.56.101:8080/job/test1/ws/

b)
Configure the job:
http://192.168.56.101:8080/job/test1/configure

c)
Console log of build Job:
http://192.168.56.101:8080/job/test1/28/console

* Explain each line of console log when build get succeeded
Example Log:
Started by user anonymous

#Create workspace directory
Building in workspace /var/lib/jenkins/jobs/test1/workspace

#Clone the repo
Fetching changes from the remote Git repository
Fetching upstream changes from https://github.com/sajuptpm/python-heatclient.git
using .gitcredentials to set credentials
Checking out Revision 03f5d54b1fefb70850b2c0882fdb561a199a9b85 (origin/master)

#Goto workspace directory /var/lib/jenkins/jobs/test1/workspace and execute build script.
[workspace] $ /bin/bash -xe /tmp/hudson2998818138453737595.sh
+ rm -f '*.deb'
+ cd build
+ git checkout packaging
+ dpkg-buildpackage -b

8)
Dependencies of dpkg-buildpackage command

#sudo apt-get install dpkg-dev
#sudo apt-get install debhelper
#sudo apt-get install python-all

* http://ftp.br.debian.org/debian/pool/main/p/python-mox3/python-mox3_0.7.0-1_all.deb
#sudo gdebi python-mox3_0.7.0-1_all.deb 

9)
Packaging Script:
https://github.com/sajuptpm/python-heatclient-debian-packaging

How to OpenStack API and WSGI api-paste.ini Work

1)
Method-1
=========

http://docs.repoze.org/moonshining/pep333.html

a)
Open a file named "myapplication.py" and add following code.

##Our Application
def green(environ, start_response):
    """
        Simplest possible application object
    """
    status = '200 OK'
    response_headers = [('Content-Type', 'text/plain')]
    start_response(status, response_headers)
    return ['%s\n' % environ.get('GREETING', 'Hello world!')]

##Middleware-1/Filter-1
##Class as a middleware component
class Caseless:
    def __init__(self, app):
        self.app = app
    def __call__(self, environ, start_response):
        for chunk in self.app(environ, start_response):
            yield chunk.lower()          
          
##Middleware-2/Filter-2
##closure or nested function or decorator as a middleware component
def greeting_setter(app):
    def _curried(environ, start_response):
        environ['GREETING'] = 'Bonjour, le monde!'
        return app(environ, start_response)
    return _curried

##Run your application "green" under the "Paste" webserver.
if __name__ == '__main__':
    from paste import httpserver
    from caseless import Caseless
    httpserver.serve(Caseless(greeting_setter(green)),
                     host='127.0.0.1', port='8080')

b)
#easy_install Paste

c)
##Run the application
#python myapplication.py

2)

Method-2 (OpenStack Using this method for API service)

=================================

http://docs.repoze.org/moonshining/tools/paste.html

a)
Open a file named "configuration.ini" and add following code

[app:green]
paste.app_factory = myapplication:app_factory

[filter:caseless]
paste.filter_app_factory = myapplication:filter_caseless_factory

[filter:greetingsetter]
paste.filter_app_factory = myapplication:filter_greeting_setter_factory

[pipeline:main]
pipeline = caseless greetingsetter green

[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 8080

b)
Open a file named "myapplication.py" and add following code.

##Our Application
def green(environ, start_response):
    """
        Simplest possible application object
    """
    status = '200 OK'
    response_headers = [('Content-Type', 'text/plain')]
    start_response(status, response_headers)
    return ['%s\n' % environ.get('GREETING', 'Hello world!')]

##Middleware-1/Filter-1
##Class as a middleware component
class Caseless:
    def __init__(self, app):
        self.app = app
    def __call__(self, environ, start_response):
        for chunk in self.app(environ, start_response):
            yield chunk.lower()
          
##Middleware-2/Filter-2
##closure or nested function or decorator as a middleware component
def greeting_setter(app):
    def _curried(environ, start_response):
        environ['GREETING'] = 'Bonjour, le monde!'
        return app(environ, start_response)
    return _curried

##Factory to get middleware "Caseless" warapped application instance
def filter_caseless_factory(app, global_config):
    return Caseless(app)

##Factory to get middleware "greeting_setter" warapped application instance
def filter_greeting_setter_factory(app, global_config):
    return greeting_setter(app)

c)
#easy_install Paste
#easy_install PasteDeploy
#easy_install PasteScript

d)
##Run the application
#paster serve configuration.ini

3)

Example from OpenStack

=======================

a)
heat-api Service:
heat/heat/bin/heat-api
https://github.com/openstack/heat/blob/master/bin/heat-api

from heat.common import config
from heat.common import wsgi

if __name__ == '__main__':
    ##This line will return all middleware/filter wrapped apllication instance based on pipeine in the .ini file.
    app = config.load_paste_app() <=== IMP
    ##Following line run your application using "eventlet green thread"
    ##See https://github.com/openstack/heat/blob/master/heat/common/wsgi.py
    server = wsgi.Server()
    server.start(app, cfg.CONF.heat_api, default_port=port)
    server.wait()

b)
heat/heat/common/config.py
https://github.com/openstack/heat/blob/master/heat/common/config.py

from heat.common import wsgi

def load_paste_app(app_name=None):
    """
        Builds and returns a WSGI app from a paste config file.
        conf_file = /etc/heat/api-paste.ini
        app_name = heat-api
    """
    app = wsgi.paste_deploy_app(conf_file, app_name, cfg.CONF)
    return app

c)
heat/heat/common/wsgi.py
https://github.com/openstack/heat/blob/master/heat/common/wsgi.py

from paste import deploy

def paste_deploy_app(paste_config_file, app_name, conf):
    setup_paste_factories(conf) <===
    try:
        return deploy.loadapp("config:%s" % paste_config_file, name=app_name)
    finally:
        teardown_paste_factories()

def setup_paste_factories(conf):
    global app_factory, filter_factory <===
    app_factory = AppFactory(conf) <===
    filter_factory = FilterFactory(conf) <===

class AppFactory(BasePasteFactory):
    KEY = 'heat.app_factory' <===
    def __call__(self, global_conf, **local_conf):
        """
            The actual paste.app_factory protocol method.
        """
        factory = self._import_factory(local_conf) <===
        ##Example of factory (app factory) : "heat.api.openstack.v1.API"
        return factory(self.conf, **local_conf)

class FilterFactory(AppFactory):
    KEY = 'heat.filter_factory'
    def __call__(self, global_conf, **local_conf):
        """
            The actual paste.filter_factory protocol method.
        """
        factory = self._import_factory(local_conf) <===
        ##Example of factory (filter factory) : "heat.api.openstack.faultwrap_filter"
        def filter(app): <===
            return factory(app, self.conf, **local_conf)
        return filter
      
class BasePasteFactory(object):
        """
            Import an app/filter class.
            Lookup the KEY from the PasteDeploy local conf and import the
            class named there. This class can then be used as an app or
            filter factory.
        """      
    KEY = None

    def __init__(self, conf):
        self.conf = conf

    def __call__(self, global_conf, **local_conf):
        raise NotImplementedError      

    def _import_factory(self, local_conf):
        class_name = local_conf[self.KEY].replace(':', '.').strip()
        ##Example of class_name : "heat.api.openstack.v1.API", "heat.api.openstack.faultwrap_filter"
        return importutils.import_class(class_name)

4)
api-paste.ini
===========

https://github.com/openstack/heat/blob/master/etc/heat/api-paste.ini

a)
[server:main]
--------------------
* In /etc/heat/api-paste.ini you can not find the section [server:main], because OpenStack explicitly using "paste.deploy.loadapp" to get the Middlewares/Filters wrapped WSGI application instance.

* You can check the following lines in the file https://github.com/openstack/heat/blob/master/heat/common/wsgi.py

def paste_deploy_app(paste_config_file, app_name, conf):
    """
        paste_config_file = /etc/heat/api-paste.ini
        app_name = heat-api
    """
    setup_paste_factories(conf)
    try:
        return deploy.loadapp("config:%s" % paste_config_file, name=app_name) <===
    finally:
        teardown_paste_factories()

b)
[pipeline:main]
-----------------------
* In /etc/heat/api-paste.ini you can not find the section [pipeline:main], because we are passinf the name of the pipeline to the method "paste.deploy.loadapp" as "name=app_name". Here value of the variable "app_name" is "heat-api", so that will load the pipeline "[pipeline:heat-api]" defined in the https://github.com/openstack/heat/blob/master/etc/heat/api-paste.ini.

* Also note that, same paste config file https://github.com/openstack/heat/blob/master/etc/heat/api-paste.ini is used for "heat-api", "heat-api-cfn" and "heat-api-cloudwatch". So having one 'main' wouldn't work.

def paste_deploy_app(paste_config_file, app_name, conf):
    """
        paste_config_file = /etc/heat/api-paste.ini
        app_name = heat-api
    """
    setup_paste_factories(conf)
    try:
        return deploy.loadapp("config:%s" % paste_config_file, name=app_name) <===
    finally:
        teardown_paste_factories()

OpenStack Heat oslo.config Example

https://github.com/openstack/heat/blob/master/heat/common/config.py

from oslo.config import cfg
def load_paste_app(app_name=None):
    from pprint import pprint
    print "=========", pprint(vars(cfg.CONF)) <===IMP

============

{'_ConfigOpts__cache': {(None, 'config_dir'): N
one,
                        (None, 'config_file'): ['/etc/heat/heat.conf'],
                        (None, 'debug'): True,
                        (None, 'log_config'): None,
                        (None, 'log_date_format'): '%Y-%m-%d %H:%M:%S',
                        (None, 'log_dir'): None,
                        (None, 'log_file'): None,
                        (None, 'log_format'): None,
                        (None, 'paste_deploy'): ,
                        (None, 'publish_errors'): False,
                        (None, 'use_stderr'): True,
                        (None, 'use_syslog'): False,
                        ('paste_deploy', 'api_paste_config'): 'api-paste.ini',
                        ('paste_deploy', 'flavor'): None},
 '_args': ['--config-file=/etc/heat/heat.conf'],
 '_config_opts': [,
                  ],
 '_groups': {'auth_password': ,
             'heat_api': ,
             'heat_api_cfn': ,
             'heat_api_cloudwatch': ,
             'paste_deploy': },
 '_namespace': _Namespace(conf=, config_dir=None, config_file=['/etc/heat/heat.conf'], debug=Tr
ue, files_not_found=[], log_config=None, log_date_format=None, log_dir=None, log_file=None, log_format=None, parser=er object at 0x12561d0>, syslog_log_facility=None, use_syslog=False, verbose=None),
 '_oparser': _CachedArgumentParser(prog='heat-api', usage=None, description=None, version=None, formatter_class=, co
nflict_handler='error', add_help=True),
 '_opts': {'allowed_rpc_exception_modules': {'cli': False,
                                             'opt': },
           'config_dir': {'cli': True,
                          'opt': },
           'config_file': {'cli': True,
                           'opt': },
           'control_exchange': {'cli': False,
                                'opt': },
           'debug': {'cli': True,
                     'opt': },
           'default_log_levels': {'cli': False,
                                  'opt': },
           'deferred_auth_method': {'cli': False,
                                    'opt': },
           'environment_dir': {'cli': False,
                               'opt': },
           'event_purge_batch_size': {'cli': False,
                                      'opt': },
           'fake_rabbit': {'cli': False,
                           'opt': },
           'fatal_deprecations': {'cli': False,
                                  'opt': },
           'heat_metadata_server_url': {'cli': False,
                                        'opt': },
           'heat_stack_user_role': {'cli': False,
                                    'opt': },
           'heat_waitcondition_server_url': {'cli': False,
                                             'opt': },
           'heat_watch_server_url': {'cli': False,
                                     'opt': },
           'host': {'cli': False,
                    'opt': },
           'instance_connection_https_validate_certificates': {'cli': False,
                                                               'opt': },
           'instance_connection_is_secure': {'cli': False,
                                             'opt': },
           'instance_driver': {'cli': False,
                               'opt': },
           'instance_format': {'cli': False,
                               'opt': },
           'instance_user': {'cli': False,
                             'opt': },
           'instance_uuid_format': {'cli': False,
                                    'opt': },
           'log_config': {'cli': True,
                          'opt': },
           'log_date_format': {'cli': True,
                               'opt': },
           'log_dir': {'cli': True,
                       'opt': },
           'log_file': {'cli': True,
                        'opt': },
           'log_format': {'cli': True,
                          'opt': },
           'logging_context_format_string': {'cli': False,
                                             'opt': },
           'logging_debug_format_suffix': {'cli': False,
                                           'opt': },
           'logging_default_format_string': {'cli': False,
                                             'opt': },
           'logging_exception_prefix': {'cli': False,
                                        'opt': },
           'max_events_per_stack': {'cli': False,
                                    'opt': },
           'max_json_body_size': {'cli': False,
                                  'opt': },
           'max_nested_stack_depth': {'cli': False,
                                      'opt': },
           'max_resources_per_stack': {'cli': False,
                                       'opt': },
           'max_stacks_per_tenant': {'cli': False,
                                     'opt': },
           'max_template_size': {'cli': False,
                                 'opt': },
           'periodic_interval': {'cli': False,
                                 'opt': },
           'plugin_dirs': {'cli': False,
                           'opt': },
           'publish_errors': {'cli': False,
                              'opt': },
           'rpc_backend': {'cli': False,
                           'opt': },
           'rpc_cast_timeout': {'cli': False,
                                'opt': },
           'rpc_conn_pool_size': {'cli': False,
                                  'opt': },
           'rpc_response_timeout': {'cli': False,
                                    'opt': },
           'rpc_thread_pool_size': {'cli': False,
                                    'opt': },
           'sql_connection': {'cli': False,
                              'opt': },
           'sql_idle_timeout': {'cli': False,
                                'opt': },
           'syslog_log_facility': {'cli': True,
                                   'opt': },
           'trusts_delegated_roles': {'cli': False,
                                      'opt': },
           'use_stderr': {'cli': False,
                          'opt': },
           'use_syslog': {'cli': True,
                          'opt': },
           'verbose': {'cli': True,
                       'opt': }},
 '_validate_default_values': False,
 'default_config_files': ['/etc/heat/heat.conf'],
 'default_log_levels': ['amqplib=WARN',
                        'qpid.messaging=INFO',
                        'keystone=INFO',
                        'eventlet.wsgi.server=WARN'],
 'prog': 'heat-api',
 'project': 'heat',
 'usage': None,
 'version': None}

          

Python Getting Started With setuptools setup.py and setup.cfg

How to build and install a python module using setuptools

1)
Install setuptools
#sudo apt-get install python-setuptools

2)
select a python project/module which we need to to install using setuptools
https://github.com/openstack/python-keystoneclient

3)
Open python console and try import your project/module and you should get error, since project/module not installed.






4)
* Create a setup.py file in the project/module folder with following lines and run #python setup.py build and install
import setuptools
setuptools.setup()

* Edit setup.py file and add following line and run #python setup.py build and install
import setuptools
setuptools.setup(name="regservice", packages=setuptools.find_packages())

Examples:
http://pythonhosted.org/setuptools/setuptools.html#using-find-packages
https://github.com/django/django/blob/master/setup.py

5)
Run "#python setup.py build"
* Check the folders and files which are created by above command
* Create a requirement.txt file in project/module folder and run "#python setup.py build" again and check the require.txt file in the egg folder.
* Run "#python setup.py install" and open a python console and try to import your project/module
* Try different options of #python setup.py build
* Try different options of #python setup.py install

6)
setup.py commands
--------------------------
#python setup.py --help-commands

a)
#sudo python setup.py build --help

b)
#sudo python setup.py install --help

Options for 'install' command:

  --user                               install in user site-package
                                       '/home/saju/.local/lib/python2.7/site-
                                       packages'

  --record                             filename in which to record list of
                                       installed files


#sudo python setup.py install --user

#sudo python setup.py install --user --record=a.txt

c)
sudo python setup.py sdist

d)
sudo python setup.py nosetests

e)
sudo python setup.py build_sphinx

f)
sudo python setup.py develop -z

g)
sudo python setup.py test

h)
sudo python setup.py install --user saveopts
sudo python setup.py develop -z saveopts

7)
* Modify the setup.py file in the project/module folder with following lines
import setuptools
setuptools.setup(setup_requires=['pbr'], pbr=True)

* What is pbr
https://github.com/openstack-dev/pbr

* Run "#python setup.py build" and examin requirement.txt and egg/require.txt
* Create a setup.cfg file in the project/module folder with following lines
[metadata]
name= blabla
* Run "#python setup.py build" and examin requirement.txt and egg/require.txt
* Run "#python setup.py install" and open python console and try to import your project/module

8)
* Uninstall a package which installed with command "#python setup.py install"
#sudo pip uninstall regservice

9)
* Modify the setup.py file in the project/module folder with following lines
import setuptools
setuptools.setup()

Then run following command to install the module and try to import the module in python console. You can use "#sudo pip uninstall regservice" to unstall the module
#sudo python setup.py install

10)
modify the setup.py in the project folder with following lines
import setuptools
setuptools.setup(name="regservice")


Then run following command to install the module and try to import the module in python console. You can use "#sudo pip uninstall regservice" to unstall the module#sudo python setup.py install

Getting Started with Python WSGI and Paste Deployment

WSGI and Paste Deployment
#####################
http://docs.repoze.org/moonshining/index.html

The Python web development world has adopted on WSGI as an interoperability standard for Python web servers, applications, and middleware components. This tutorial examines that specification in deatil, and shows practial examples of using WSGI to improve development productivity, integrate applications, and solve various deployment problems.

1)
Example Application
=================
http://docs.repoze.org/moonshining/pep333.html#example-application

* Simplest possible WSGI application

##SM:Our Application
def simple_app(environ, start_response):
    """
    ##SM:Simplest Possible Application Object.
    ##SM:Our Application.
    """
    ##SM:HTTP status for the response
    status = '200 OK'
    ##SM:Construct HTTP response headers
    response_headers = [('Content-Type', 'text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

##SM:Server to serve Our Application "simple_app"
##Run server and serve the Application
if __name__ == '__main__':
    ##SM:Import http webserver from paste tool
    from paste import httpserver
    ##SM:Run our application "simple_app" under the Paste webserver
    httpserver.serve(simple_app, host='127.0.0.1', port='8080')

##Flow
Http REQUEST --> WSGI Server creates "environ" and "start_response" -->
calls Application (Example:simple_app) or Application's "__call__" if Application is an instance of a class -->
calls ``start_response`` with status and headers --> returns iterable

2)
Architecture
=============
a)
Architecture of Application
----------------------------
http://docs.repoze.org/moonshining/pep333.html#architecture

Any callable with the following signature:

def __call__(environ, start_response):
    """
    Return an iterable.
    Here "environ" is the WSGI environment dictionary.
    Normally, call 'start_response' with a status and headers before returning.
    """
* OR one like in our simple example "def simple_app(environ, start_response):".

##Flow
Http REQUEST --> WSGI Server creates "environ" and "start_response" -->
calls Application (Example:simple_app) or Application's "__call__" if Application is an instance of a class -->
calls ``start_response`` with status and headers --> returns iterable

b)
Architecture of Server
------------------------------
Must populate the "environ" dictionary, and provide the "start_response" callback.


WSGI Environment ("environ" dictionary)
=======================================
http://docs.repoze.org/moonshining/pep333.html#wsgi-environment

The "environ" dictionary passed by the server is normally a copy of "os.environ" with the standard CGI keys:

* SCRIPT_NAME: the “base” of the URL, representing the root of the application.
* PATH_INFO: the remainder of the URL, to be interpreted by the application.

The environ also contains addtional, WSGI-specific keys, of which the most important are:
* "wsgi.input": represents the body / payload of the request.
* "wsgi.errors": represents a stream to which error logging may be done.
* "wsgi.url_scheme": is typically either “http” or “https”.

The "start_response" Callback
===============================
http://docs.repoze.org/moonshining/pep333.html#the-start-response-callback

The start_response callback takes two arguments, conventionally named “status” and “headers”:
The first argument is a string containing the contents of the HTTP response status line, e.g. '200 OK'.
The second argument is a list of two-tuples representing HTTP response headers, e.g. [('Content-Type', 'text/html'), ('Content-Length', '15')]

3)
Middleware
============
http://docs.repoze.org/moonshining/pep333.html#middleware

A WSGI middleware component is one which plays both the role of of the application and the role of the server: the “upstream” server (Eg:Paste httpserver or Apache webserver) calls it, passing the "environ" and "start_response" arguments, and expects it to return the iterable response body. The middleware component in turn calls “downstream” component (Eg:Our Application), perhaps mutating/change "environ" first, or replacing the "start_response" with another callable. The middleware component may also intercept the returned iterator and transform or replace it, and may add exception handling.

* In computer networking, upstream server refers to a server that provides service to another server. In other words, upstream server is a server that is located higher in the hierarchy of servers.

a)
Example Middleware
---------------------------
http://docs.repoze.org/moonshining/pep333.html#example-middleware

This example middleware component or class filters the output of the “downstream” application (Our Application), converting it all to lowercase:

class Caseless:
    """
        Creates a middleware as class.
        Class as a middleware component.
    """
    def __init__(self, app):
        """
        Save the “downstream” application (Out Application) as an attribute "self.app".
        This middelware should be created before the application starts serving requests.
        """
        self.app = app

    def __call__(self, environ, start_response):
        """
            WSGI Architecture for Application.
            WSGI application interface.
            So this will act as an application.
        """
        ##Call Out Application and convert result to lowercase and return
        for chunk in self.app(environ, start_response):
            yield chunk.lower()

b)
Add middleware to the simplest application "simple_app" created before
--------------------------------------------------------------------------------------------
http://docs.repoze.org/moonshining/pep333.html#exercise-add-middleware-to-the-simplest-application

##Run server and serve the Application "simple_app"
if __name__ == '__main__':
    from paste import httpserver
    from simplest import simple_app
    ##Calls the middleware "Caseless"
    httpserver.serve(Caseless(simple_app), host='127.0.0.1', port='8080')

c)
Pipelines or Chaining Middleware together
-----------------------------------------------------------
http://docs.repoze.org/moonshining/pep333.html#pipelines-chaining-middleware-together

Middleware components can be chained together to form a WSGI “pipeline.”

##Flow
Http REQUEST --> WSGI Server creates "environ" and "start_response" -->
Middleware A's ``__call__`` ---> Middleware B's ``__call__`` --> Middleware C's ``__call__`` -->
Application's ``__call__`` --> calls ``start_response`` with status and headers --> returns iterable

d)
Example of Pipelining or Chaining Middleware
--------------------------------------------------------------
http://docs.repoze.org/moonshining/pep333.html#exercise-chaining-middleware

##Our new Application
def green(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-Type', 'text/plain')]
    start_response(status, response_headers)
    return ['%s\n' % environ.get('GREETING', 'Hello world!')]

##New Middleware
def greetingSetter(app):
    """
        Creates a middleware as closure or nested function or decorator.
    """
    def _curried(environ, start_response):
        environ['GREETING'] = "Hi Saju.."
        return app(environ, start_response)
    return _curried

##Run server and serve the Application "green"
if __name__ == '__main__':
    from paste import httpserver
    from caseless import Caseless
    ##Calls the middlewares "Caseless" and "greetingSetter".
    httpserver.serve(Caseless(greetingSetter(green)),
                        host='127.0.0.1', port='8080')

4)
Making WSGI Configuration Declarative using .ini file: PasteDeploy
==================================================
http://docs.repoze.org/moonshining/tools/paste.html#making-wsgi-configuration-declarative-pastedeploy

In the previous examples, we created the various components (middlewares, our applications) and assembled them to form the WSGI pipeline in the if __name__ == "__main__" section of our applications. PasteDeploy allows us to make drive that process from an INI-style configuration file, rather than from Python code. The configuration file defines the WSGI server (paste.httpserver or Apache) to use, as well as one or more applications (our applications, Example:simple_app, green), as well as the middleware filters and pipelines which compose them with applications.


a)
Example of Configuring the Simplest Application in .ini file
------------------------------------------------------------------------------

* Each section of the configuration file defines a different component like middlware and our Apllications.

##Section to configure Our Applications in .ini file
[app:main]
paste.app_factory = my_module:app_factory
name = Phred
greeting = Wilkommen

* [app:main] --> the app: prefix, used to define a WSGI application endpoint. The name "main" makes this application the default for this file.
* Here the name ":main" is important since it make that section in the .ini file as the default/main application.
* paste.app_factory = my_module:app_factory --> tells PasteDeploy to look up the function app_factory in the module my_module, and call it to get the application.
* name = Phred and greeting = Wilkommen --> provide configuration values to be are passed to the factory method "my_module:app_factory".

b)
Loading the Configuration from .ini file
----------------------------------------------------
"PasteDeploy" provides two APIs for loading a configuration file:

* "paste.deploy.appconfig" loads the configuration for a given application from the file (.ini file) and returns it as a dictionary, merging in any supplied “global” configuration.
* "paste.deploy.loadapp" instantiates and return a given application from the file (.ini file), merging in any supplied “global” configuration.

c)
Example of Configuring the Our Application in .ini file
----------------------------------------------------------------------
http://docs.repoze.org/moonshining/tools/paste.html#exercise-configuring-an-application

##Our Application "OurApplication.py"
class OurApplication:
    def __init__(self, name, greeting):
        self.name = name
        self.greeting = greeting
    def __call__(self, environ, start_response):
        status = '200 OK'
        response_headers = [('Content-Type', 'text/plain')]
        start_response(status, response_headers)
        return ['%s, %s!\n' % (self.greeting, self.name)]

##factory method to get Our Application via .ini file
def app_factory(global_config, name='Johnny', greeting='Howdy'):
    return OurApplication(name, greeting)


* We define both the application (OurApplication) and the factory for the application (app_factory). The factory is necessary to permit passing in values from the configuration file.


##Add a configuration file, configured.ini
[app:main]
paste.app_factory = my_module:app_factory
name = Phred
greeting = Wilkommen

##Code which drives the application from the configuration file configured.ini
if __name__ == '__main__':
    from paste import httpserver
    from paste.deploy import loadapp
    httpserver.serve(loadapp('config:configured.ini', relative_to='.'),
                     host='127.0.0.1', port='8080')

* This file doesn’t import "OurApplication.py" directly, but instead allows "PasteDeploy" to handle that process as part of the configuration. We can therefore tweak/change the configuration (.ini file) to use a different module or factory function, as well as providing configuration values, without changing the software.

Example of Configuring Middleware in .ini file
-------------------------------------------------------------
http://docs.repoze.org/moonshining/tools/paste.html#example-configuring-middleware

* Under "PasteDeploy", middleware components are called “filters”, and can be configured via config file sections using the filter: prefix.

##Section to configure Our Middleware in .ini file
[filter:caseless]
filter_app_factory = my_module:filter_factory


* [filter:caseless] --> Defines the filter/middleware. by default, no filters are included: they must be explicitly configured.
* filter_app_factory = my_module:filter_factory --> Tells "PasteDeploy" to look up the function "filter_factory" in the module my_module, and call it to get the filter/middleware.
* Extra configuration values will be passed to the filter/middeware factory "my_module:filter_factory".

##Example of filter/middelware factory in "my_module.py"
from caseless import Caseless
def filter_factory(app, global_config):
    return Caseless(app)

Example of Configuring the WSGI Pipeline (Chaining Middleware) in .ini file
-----------------------------------------------------------------------------
http://docs.repoze.org/moonshining/tools/paste.html#example-configuring-the-wsgi-pipeline

* PasteDeploy provides another section type, using the "pipeline:" prefix, which it uses to construct a pipline (pipeline section) as a special kind of WSGI application (new application by invoking all middlewares and applications in the order of they specified in the pipeline section on .ini file).

##Section to configure Our Applications in .ini file
[app:greeting]
paste.app_factory = configured:app_factory
name = Bharney
greeting = Bienvenu

##Section to configure filter/middleware in .ini file
[filter:caseless]
paste.filter_app_factory = configured:filter_factory

##Section to specify order of components (middlewares and applications) in which they call.
[pipeline:main]
pipeline =
    caseless
    greeting

* Here the name ":main" is important since it make that section in the .ini file as the default/main application.
* The items in the pipeline, point to other applications (caseless) or filters/middleware (greeting) in the .ini file.
* The Pipeline is configured “bottom up”, with each layer wrapping the one below it,
Example: Caseless(greeting)
httpserver.serve(Caseless(greeting), host='127.0.0.1', port='8080')

Example of Configuring the WSGI Server in .ini file
------------------------------------------------------------------
http://docs.repoze.org/moonshining/tools/paste.html#exercise-configuring-the-wsgi-server

We can use "PasteScript" in conjuction with "PasteDeploy" to replace our hand-rolled run server script.
First, add the following server configuration from the above example to the "configured.ini" file:

##Section to configure Server in .ini file
[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 8080

* [server:main] --> The section with the "server:" prefix, used to define a WSGI server. As with the application, the name "main" marks this server as the default.
* use = egg:Paste#http --> Tells PasteDeploy to look up the http “entry point” in the Paste egg, and call it to as a "factory" to get the server instance (like factory return application in app section).
* host and port --> additional configuration parameters which are passed to the server factory.

5)
paste
======
http://pythonpaste.org/
http://docs.repoze.org/moonshining/tools/paste.html
* Paste is a WSGI Developers’ Toolkit

OpenStack Heat How heat-api service loads URLs

a)
/opt/stack/heat/bin/heat-api
https://github.com/openstack/heat/blob/master/bin/heat-api

from oslo.config import cfg
from heat.common import config
from heat.common import wsgi

app = config.load_paste_app()
server = wsgi.Server()
server.start(app, cfg.CONF.heat_api, default_port=port)
server.wait()

----------

From my understanding, When we run the script https://github.com/openstack/heat/blob/master/bin/heat-api, that will load all urls defined in the https://github.com/openstack/heat/blob/master/heat/api/openstack/v1/__init__.py.

But in the script https://github.com/openstack/heat/blob/master/bin/heat-api, I could not find any line which specifying the name of https://github.com/openstack/heat/blob/master/heat/api/openstack/v1/__init__.py,  then how it loading that ?

* it's all defined in config files
* in /etc/heat/api-paste.ini
* in https://github.com/openstack/heat/blob/master/etc/heat/api-paste.ini
* [app:apiv1app]
* paste.app_factory = heat.common.wsgi:app_factory
* heat.app_factory = heat.api.openstack.v1:API
* See https://github.com/openstack/heat/blob/master/heat/api/openstack/v1/__init__.py
* that app gets added to the heat-api pipeline...
* # heat-api pipeline
* [pipeline:heat-api]
* pipeline = faultwrap ssl versionnegotiation authurl authtoken context apiv1app
* and config.load_paste_app() presumably loads that pipeline

OpenStack Heat devstack tricks with images

a)
Configure DevStack to enable Heat
Adding the following line to your `localrc` file will enable the heat services

ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
## It would also be useful to automatically download and register VM images that Heat can launch.
# 64bit image (~660MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F17-x86_64-cfntools.qcow2"
# 32bit image (~640MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F17-i386-cfntools.qcow2"

b)
devstack/stack.sh
https://github.com/openstack-dev/devstack/blob/master/stack.sh

# Install Images
# ==============
You can Override ``IMAGE_URLS`` with a comma-separated list of UEC images in localrc.

#g-reg: if glance registration service enabled
if is_service_enabled g-reg; then
    #Upload images to glance
    for image_url in ${IMAGE_URLS//,/ }; do
        upload_baremetal_image $image_url $TOKEN <=== Method Call
    done

c)
devstack/lib/baremetal
https://github.com/openstack-dev/devstack/blob/master/lib/baremetal

function upload_baremetal_image() {
    local image_url=$1
    local token=$2

    # Create a directory for the downloaded image tarballs.
    mkdir -p $FILES/images

    # Downloads the image (uec ami+aki style), then extracts it.
    IMAGE_FNAME=`basename "$image_url"`
    if [[ ! -f $FILES/$IMAGE_FNAME || \
        "$(stat -c "%s" $FILES/$IMAGE_FNAME)" = "0" ]]; then
        wget -c $image_url -O $FILES/$IMAGE_FNAME <=Download image
        if [[ $? -ne 0 ]]; then
            echo "Not found: $image_url"
            return
        fi
    fi

* Above function "upload_baremetal_image" will download the images (defined in the localrc) to the folder "devstack/files/", then
upload to glance.

* So if you already downloaded that images, then just copy that to the folder "files" in the "devstack/" folder and run ./stack script.

* ls -lsh ~/openstack/devstack/files/
https://github.com/openstack-dev/devstack/tree/master/files
454M -rw-r--r-- 1 saju saju 454M Mar 26 18:27 F17-i386-cfntools.qcow2
455M -rw-r--r-- 1 saju saju 455M Mar 26 18:28 F17-x86_64-cfntools.qcow2

OpenStack Horizon How to Prevent popup from closing when you click outside popup

1)
There are two files

horizon/static/bootstrap/js/bootstrap.js <=== for dev
https://github.com/openstack/horizon/blob/6954c02f4dc1d3ee98c56cc2a53512b4b625478a/horizon/static/bootstrap/js/bootstrap.js
and
horizon/static/bootstrap/js/bootstrap.min.js <=== for production (compressed)
https://github.com/openstack/horizon/blob/6954c02f4dc1d3ee98c56cc2a53512b4b625478a/horizon/static/bootstrap/js/bootstrap.min.js

bootstrap.js is not included in any html file
bootstrap.min.js is included in horizon/templates/horizon/_scripts.html

2)
http://getbootstrap.com/javascript/

Make following changes in bootstrap.js

Change

  $.fn.modal.defaults = {
      backdrop: true
    , keyboard: true
    , show: true
  }

To

  $.fn.modal.defaults = {
      backdrop: 'static'
    , keyboard: true
    , show: true
  } 

3)
http://jscompress.com/
http://stackoverflow.com/questions/9123112/twitter-bootstrap-make-from-source

* Compress the modified bootstrap.js and create bootstrap.min.js
* From the "makefile", find which compression tool was used to create bootstrap.min.js : https://github.com/twbs/bootstrap/

#sudo apt-get install node-uglify
*Modify bootstrap.js (Step-2) and run following command to create a compressed bootstrap.min.js file

#uglifyjs horizon/static/bootstrap/js/bootstrap.js -o horizon/static/bootstrap/js/bootstrap.min.js
* Run "#git diff" or check bootstrap.min.js and ensure that has Changes.

4)
Commit following files
horizon/static/bootstrap/js/bootstrap.js
horizon/static/bootstrap/js/bootstrap.min.js

=======Ref========

#sudo apt-get install yui-compressor
#yui-compressor horizon/static/bootstrap/js/bootstrap.js -o horizon/static/bootstrap/js/bootstrap.min.js

OpenStack Heat Getting Started

1)
Install heat with Devstack
===================
Heat Installation docs
------------------------------
https://wiki.openstack.org/wiki/Heat
https://wiki.openstack.org/wiki/Heat/GettingStartedUsingDevstack
https://wiki.openstack.org/wiki/Heat/TroubleShooting

Heat Devstack Script
----------------------------
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/heat

a)
Adding the following line to your `localrc` file will enable the heat services

ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
## It would also be useful to automatically download and register VM images that Heat can launch.
# 64bit image (~660MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F17-x86_64-cfntools.qcow2"
# 32bit image (~640MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F17-i386-cfntools.qcow2"

b)
launchpad
https://launchpad.net/heat

2)
Heat Developer docs
================
https://wiki.openstack.org/wiki/Heat/TroubleShooting

* Heat is a service to orchestrate (Arrange or direct the elements of a situation to produce a desired effect) multiple composite cloud applications using the AWS CloudFormation template format, through both an OpenStack-native ReST API and a CloudFormation-compatible Query API.

* Heat provides a template based orchestration for describing a cloud application by executing appropriate OpenStack API calls to generate running cloud applications.

* The software integrates other core components of OpenStack into a one-file template system. The templates allow creation of most OpenStack resource types (such as instances, floating ips, volumes, security groups, users, etc), as well as some more advanced functionality such as instance high availability, instance autoscaling, and nested stacks. By providing very tight integration with other OpenStack core projects, all OpenStack core projects could receive a larger user base.

* Allow deployers to integrate with Heat directly or by adding custom plugins.
a)
Getting Started Guides
http://docs.openstack.org/developer/heat/getting_started/index.html

b)
Example Templates
http://docs.openstack.org/developer/heat/getting_started/on_ubuntu.html#example-templates

c)
Templates
http://docs.openstack.org/developer/heat/templates/index.html

d)
#Template guid
http://docs.openstack.org/developer/heat/template_guide/index.html

#Heat Orchestration Template (HOT)
http://docs.openstack.org/developer/heat/template_guide/hot_guide.html <==IMP
http://docs.openstack.org/developer/heat/template_guide/hot_spec.html <==IMP
http://docs.openstack.org/developer/heat/templates/hot/hello_world.html
https://github.com/openstack/heat-templates/blob/master/cfn/WordPress_Single_Instance.template <==Example

#CloudFormation Template (CFN)
http://docs.openstack.org/developer/heat/template_guide/cfn.html
http://docs.openstack.org/developer/heat/templates/cfn/WordPress_Single_Instance.html
https://github.com/openstack/heat-templates/blob/master/cfn/F17/WordPress_Single_Instance_deb.template <==Example

#Example of HOT template with AutoScaling and Load balancing
https://gist.github.com/sajuptpm/9738831

e)
Glossary
http://docs.openstack.org/developer/heat/glossary.html

f)
AWS CloudFormation
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

g)
Environments
http://docs.openstack.org/developer/heat/template_guide/environment.html#environments
https://wiki.openstack.org/wiki/Heat/Environments <====

h)
Heat/Providers
https://wiki.openstack.org/wiki/Heat/Providers
http://lists.openstack.org/pipermail/openstack-dev/2013-April/007989.html
Example of provider: rackspace.com, openstack.org

i)
Heat/Plugins
Resource Plugin modules
https://wiki.openstack.org/wiki/Heat/Plugins <===IMP

j)
Heat/AutoScaling
https://wiki.openstack.org/wiki/Heat/AutoScaling

k)
Heat/AWSAutoScaling
https://wiki.openstack.org/wiki/Heat/AWSAutoScaling

3)
services and utilities
=================
http://docs.openstack.org/developer/heat/man/index.html

a)
Heat services
-------------------
http://docs.openstack.org/developer/heat/man/index.html
https://wiki.openstack.org/wiki/Heat

a1)
heat:
The heat tool is a CLI which communicates with the heat-api to execute AWS CloudFormation APIs. Of course this is not required—developers could also use the Heat APIs directly.

a2)
heat-api:
http://docs.openstack.org/developer/heat/man/heat-api.html <====
The heat-api component provides an OpenStack-native ReST API that processes API requests by sending them to the heat-engine over RPC.

a3)
heat-api-cfn:
http://docs.openstack.org/developer/heat/man/heat-api-cfn.html
The heat-api-cfn component provides an AWS-style Query API that is compatible with AWS CloudFormation and processes API requests by sending them to the heat-engine over RPC.

a4)
heat-engine:
http://docs.openstack.org/developer/heat/man/heat-engine.html
The heat engine does the main work of orchestrating the launch of templates and providing events back to the API consumer.

a5)
heat-api-cloudwatch:
http://docs.openstack.org/developer/heat/man/heat-api-cloudwatch.html

b)
Heat utilities
-------------------
http://docs.openstack.org/developer/heat/man/index.html
https://wiki.openstack.org/wiki/Heat

b1)
heat-manage:
http://docs.openstack.org/developer/heat/man/heat-manage.html

b2)
heat-db-setup:
http://docs.openstack.org/developer/heat/man/heat-db-setup.html

b3)
heat-keystone-setup:
http://docs.openstack.org/developer/heat/man/heat-keystone-setup.html

4)
API Documentation
===============
http://api.openstack.org/api-ref-orchestration.html

5)
Scaling a Deployment
================
http://docs.openstack.org/developer/heat/scale_deployment.html

6)
Also Read
=========
a)
Metabata Service:
The metadata service is implemented by either the nova-api service or the nova-api-metadata service.
http://docs.openstack.org/admin-guide-cloud/content/section_metadata-service.html

b)
HAProxy:
The Reliable, High Performance TCP/HTTP Load Balancer
http://haproxy.1wt.eu/