Custom Search
Showing posts with label OpenStack Packaging. Show all posts
Showing posts with label OpenStack Packaging. Show all posts

Friday, March 28, 2014

Python Getting Started With setuptools setup.py and setup.cfg

How to build and install a python module using setuptools

1)
Install setuptools

#sudo apt-get install python-setuptools

2)

select a python project/module which we need to to install using setuptools
https://github.com/openstack/python-keystoneclient

3)

Open python console and try import your project/module and you should get error, since project/module not installed.






4)

* Create a setup.py file in the project/module folder with following lines and run #python setup.py build and install
import setuptools
setuptools.setup()


* Edit setup.py file and add following line and run #python setup.py build and install
import setuptools
setuptools.setup(name="regservice", packages=setuptools.find_packages())


Examples:
http://pythonhosted.org/setuptools/setuptools.html#using-find-packages
https://github.com/django/django/blob/master/setup.py

5)
Run "#python setup.py build"
* Check the folders and files which are created by above command
* Create a requirement.txt file in project/module folder and run "#python setup.py build" again and check the require.txt file in the egg folder.
* Run "#python setup.py install" and open a python console and try to import your project/module
* Try different options of #python setup.py build
* Try different options of #python setup.py install

6)
setup.py commands
--------------------------

#python setup.py --help-commands

a)
#sudo python setup.py build --help

b)
#sudo python setup.py install --help

Options for 'install' command:

  --user                               install in user site-package
                                       '/home/saju/.local/lib/python2.7/site-
                                       packages'

  --record                             filename in which to record list of
                                       installed files


#sudo python setup.py install --user

#sudo python setup.py install --user --record=a.txt

c)
sudo python setup.py sdist

d)
sudo python setup.py nosetests

e)
sudo python setup.py build_sphinx

f)
sudo python setup.py develop -z

g)
sudo python setup.py test

h)
sudo python setup.py install --user saveopts
sudo python setup.py develop -z saveopts


7)
* Modify the setup.py file in the project/module folder with following lines
import setuptools
setuptools.setup(setup_requires=['pbr'], pbr=True)


* What is pbr
https://github.com/openstack-dev/pbr

* Run "#python setup.py build" and examin requirement.txt and egg/require.txt
* Create a setup.cfg file in the project/module folder with following lines
[metadata]
name= blabla
* Run "#python setup.py build" and examin requirement.txt and egg/require.txt
* Run "#python setup.py install" and open python console and try to import your project/module

8)
* Uninstall a package which installed with command "#python setup.py install"
#sudo pip uninstall regservice

9)
* Modify the setup.py file in the project/module folder with following lines
import setuptools
setuptools.setup()


Then run following command to install the module and try to import the module in python console. You can use "#sudo pip uninstall regservice" to unstall the module
#sudo python setup.py install

10)
modify the setup.py in the project folder with following lines
import setuptools
setuptools.setup(name="regservice")



Then run following command to install the module and try to import the module in python console. You can use "#sudo pip uninstall regservice" to unstall the module#sudo python setup.py install

Saturday, March 22, 2014

OpenStack How to locate and clone Packaging Git Repository of each Projects

1)
Method1
-------------
#git clone git://git.debian.org/openstack/project_name.git
Replace project_name by any package (like nova, cinder, neutron, horizon, etc.)

#cd /project_name
#git checkout debian/havana
#git checkout debian/icehouse
#cd debian
#ls

Python modules are using either:
git checkout debian/unstable
git checkout debian/experimental

2)
Method2
-------------
http://anonscm.debian.org/gitweb/?a=project_list&s=openstack&btnS=Search

#git clone git://anonscm.debian.org/openstack//project_name.git
Replace project_name by any package (like nova, cinder, neutron, horizon, etc.)

#cd /project_name
#git checkout debian/havana
#git checkout debian/icehouse
#cd debian
#ls

3)
Method3
-------------
Extract from the Ubuntu source packages found in the ubuntu cloud archive.
For example:
http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/h/horizon/horizon_2014.1~b3-0ubuntu2~cloud0.dsc
If you use the "dget" tool, it will download the .dsc, the orig.tar.gz and the .diff.gz....and unpack it.
#dget http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/h/horizon/horizon_2014.1~b3-0ubuntu2~cloud0.dsc
You need dpkg-dev or devscripts installed.So that dget command will download it all and unpack it.So you can see the debian directory.

Goto following link and locate your project's *.dsc file
http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/

Example:
#sudo apt-get install devscripts
#dget http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/h/horizon/horizon_2014.1~b3-0ubuntu2~cloud0.dsc
#tar -xzf horizon_2014.1~b3.orig.tar.gz
#cd debian
#ls

Friday, March 21, 2014

How to use quilt to manage patches in Debian packages

http://raphaelhertzog.com/2012/08/08/how-to-use-quilt-to-manage-patches-in-debian-packages/

0)
#sudo apt-get install quilt
or
Install the packaging-dev package to get a decent set of packages to do Debian packaging. It includes the quilt package.
#sudo apt-get install packaging-dev

1)
Quilt manages a series of patches by keeping track of the changes each of them makes. They are logically organized as a stack, and you can apply, un-apply, update them easily by traveling into the stack (push/pop).

The “stack of patches” is maintained in a dedicated directory (“patches” by default, but in Debian packages we override this value to “debian/patches”). This directory contains the patch files and a “series” file that gives an ordered list of patches to apply.

When quilt is used, it also maintains some internal files in a directory of its own (it’s named “.pc”). This directory is used to know what patches are currently applied (.pc/applied-patches) and to keep backup copies of files modified by the various patches.

2)
Configuring quilt
--------------------
export QUILT_PATCHES=debian/patches

3)
Applying and unapplying patches, navigating in the stack of patches

a)
Apply next patch
#quilt push

b)
Apply all patch
#quilt push -a

c)
To see the name of applied patches
#quilt applied

d)
To see the name of next patch to be applied
#quilt next

e)
To see the name of unapplied pathes
#quilt unapplied

f)
Unapplies the last applied patch
#quilt pop

g)
Unapplies all applied patch
#quilt pop -a

h)
To see the name of last appled patch
#quilt top

i)
To apply up to particular patch
#quilt push name_ofpatch_file.diff

3)
How to Create a new patch
-----------------------------------------
If there’s no quilt series yet, you want to create the “debian/patches” directory first.
If you already have one, you need to decide where to insert the new patch. Quilt will always add the new patch just after the patch which is currently on top. So if you want to add the patch at the end of the series, you need to run “quilt push -a” first.
Then you can use quilt new name-of-my-patch.diff to tell quilt to insert a new empty patch after the current topmost patch. In this operation quilt does almost nothing except updating the series file and recording the fact that the new patch is applied (even if still empty at this point!).
Now to add changes in this patch, you’re supposed to modify files but only after having informed quilt of your intent to modify those files. You do this with quilt add file-to-modify. At this point quilt will make a backup copy of that file so that it can generate the final patch when you’re done with your changes. It’s quite common to forget this step and to be unable to generate the patch afterward. That’s why I recommend you to use quilt edit file-to-modify which is a shorthand for doing quilt add and then opening the file in your favorite text editor.
If you want, you can review your work in progress with quilt diff.
When you’re done with the changes, you should call quilt refresh to generate the patch (or to update it if it was already existing).

a)
if you want to add the patch at the end of the series, you need to run “quilt push -a” first
#quilt push -a

b)
add new patch file
#quilt new name-of-my-new-patch.diff

c)
Specify the files for which you want to create path
#quilt edit file-1-to-modify
#quilt edit file-2-to-modify
#quilt edit file-3-to-modify

d)
Open the files specified above in a text editor and make your changes

e)
How to review the patch before creating it
#quilt diff

f)
Generate the patch
#quilt refresh

4)
Importing an external patch
----------------------------
If someone else already prepared a patch, you can just import it right away with quilt import /tmp/the-patch. If you want to import it under a better name you can use the option “-P better-patch-name”. Like quilt new, it inserts the patch after the topmost patch.

a)
Import a new patch
#quilt import /tmp/the-name-of-patch-file.diff

b)
How to give new name for patch during import
#quilt import /tmp/the-name-of-patch-file.diff -P new-name-for-imported-patch-file.diff


5)
Updating patches for a new upstream version
--------------------------------------------
a)
Apply patches
#quilt push
or
#quilt push -a

b)
manually apply the parts of the patch that were rejected, and then refresh/generate the patch

c)
Generate the patch
#quilt refresh

6)
Other quilt commands
---------------------
a)
To rename the topmost patch
#quilt rename new-name-for-topmost-patch.diff

b)
To rename a particular patch
#quilt rename -P patch-to-rename.diff new-name-for-patch.diff

7)
Editing an existing patch
--------------------------
1)
cd horizon/debian

2)
export QUILT_PATCHES=debian/patches

3)
##Apply app patches
quilt push -a

4)
##Edit files / Make changes
vim ../openstack_dashboard/local/local_settings.py.example

5)
##Create patch
quilt refresh jcloud-settings.patch

6)
##Remove all applied patches
quilt pop -af

7)
Check
git diff

8)
git commit patches/jcloud-settings.patch

9)
quilt push -a
quilt pop -a




Horizon Packaging How to Compile less files into css files

https://wiki.debian.org/UsingQuilt

1)
#sudo apt-get install quilt

2)
#git clone https://github.com/sajuptpm/horizon.git -b packaging
#cd horizon
#git rebase origin/my_dashboard

3)
https://github.com/sajuptpm/horizon/blob/packaging/debian/README.compression
#sudo apt-get install python-lesscpy python-openstack-auth python-compressor

4)
#quilt pop top

5)
#./debian/rules refresh-static-assets
(cd openstack_dashboard/local; ln -sf local_settings.py.example local_settings.py)
python ./manage.py compress --force
/opt/stack/pbr/pbr/version.py:21: UserWarning: Module openstack_dashboard was already imported from /home/saju/rebase_dir/dum2/horizon/openstack_dashboard/__init__.py, but /opt/stack/horizon is being added to sys.path
  import pkg_resources
Non-existent template at: /home/saju/rebase_dir/dum2/horizon/horizon/templates/base.html
Found 'compress' tags in:
/home/saju/rebase_dir/dum2/horizon/horizon/templates/horizon/_scripts.html
/home/saju/rebase_dir/dum2/horizon/openstack_dashboard/templates/_stylesheets.html
/home/saju/rebase_dir/dum2/horizon/horizon/templates/horizon/_conf.html
Compressing... done
Compressed 3 block(s) from 3 template(s).
rm -rf debian/static/openstack-dashboard/*
cp -r static/dashboard/* debian/static/openstack-dashboard/
rm -f openstack_dashboard/local/local_settings.py

6)
$ git status
# On branch packaging
# Your branch and 'origin/packaging' have diverged,
# and have 26 and 13 different commits each, respectively.
#   (use "git pull" to merge the remote branch into yours)
#
# Changes not staged for commit:
#   (use "git add/rm ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
# deleted:    debian/static/openstack-dashboard/css/e1362ce456d5.css
# modified:   debian/static/openstack-dashboard/manifest.json
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
# debian/static/openstack-dashboard/css/2aec13458cd9.css
no changes added to commit (use "git add" and/or "git commit -a")
saju@saju-Inspiron-5521:~/rebase_dir/dum2/horizon$

7)
$ git add debian/static/openstack-dashboard/css/2aec13458cd9.css

8)
$ git commit debian/static/openstack-dashboard/css/2aec13458cd9.css debian/static/openstack-dashboard/css/e1362ce456d5.css debian/static/openstack-dashboard/manifest.json
[packaging 7438705] compile .less files into .css files
 2 files changed, 522 insertions(+), 232 deletions(-)
 rename debian/static/openstack-dashboard/css/{e1362ce456d5.css => 2aec13458cd9.css} (91%)

9)
$ git push origin packaging
Username for 'https://github.com': sajuptpm@gmail.com
Password for 'https://sajuptpm@gmail.com@github.com':
To https://github.com/sajuptpm/horizon.git
 ! [rejected]        packaging -> packaging (non-fast-forward)
error: failed to push some refs to 'https://github.com/sajuptpm/horizon.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Merge the remote changes (e.g. 'git pull')
hint: before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

10)
$ git push -f origin packaging
or
$ git push origin packaging --force
Username for 'https://github.com': sajuptpm@gmail.com
Password for 'https://sajuptpm@gmail.com@github.com':
Counting objects: 136, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (81/81), done.
Writing objects: 100% (135/135), 295.58 KiB | 0 bytes/s, done.
Total 135 (delta 60), reused 96 (delta 42)
To https://github.com/sajuptpm/horizon.git
 + 42f2ee6...7438705 packaging -> packaging (forced update)

Thursday, March 13, 2014

How to Build and Install Python Module using setuptools and Distutils

0)
Install python-setuptools
#sudo apt-get install python-setuptools

1)
http://docs.python.org/2/install/

The bulk of this document is about building and installing modules from standard source distributions.
Building and installing a module distribution using the Distutils is usually one simple command to run from a terminal.
#python setup.py install

2)
Running "python setup.py install" builds and installs all modules in one step.
You can also build everything in one step, and then install everything in a second step, by invoking the setup script twice:
#python setup.py build
#python setup.py install


3)
How building works
-------------------

The "build" command is responsible for putting the files to install into a build directory.
By default, this is "build" under the distribution root.

The default layout for the build tree is as follows:

--- build/ --- lib/
or
--- build/ --- lib./
               temp.
/

4)
How installation works
-----------------------

The work of the install command is relatively simple: all it has to do is copy everything under "build/lib" (or build/lib.plat) to your chosen installation directory.
If you don’t choose an installation directory—i.e., if you just run "python setup.py install" then the install command installs to the standard location for third-party Python modules. This location varies by platform and by how you built/installed Python itself

5)
Alternate installation: the user scheme
----------------------------------------

This scheme is designed to be the most convenient solution for users that don’t have write permission to the global site-packages directory or don’t want to install into it. It is enabled with a simple option:

#python setup.py install --user

Files will be installed into subdirectories of "site.USER_BASE" (Default value is ~/.local for UNIX).
This scheme installs pure Python modules and extension modules in the same location


http://docs.python.org/2/library/site.html#site.USER_BASE

6)
Custom Installation
--------------------

To create a custom installation scheme, you start with one of the alternate schemes and override some of the installation directories used for the various types of files.
Specifying the entire installation scheme every time you install a new module distribution would be very tedious. Thus, you can put these options into your Distutils config file

7)
Distutils Configuration Files
------------------------------

You can use Distutils configuration files to record personal or site preferences for any Distutils options. That is, any option to any command can be stored in one of two or three (depending on your platform) configuration files, which will be consulted before the command-line is parsed. This means that configuration files will override default values, and the command-line will in turn override configuration files. Furthermore, if multiple configuration files apply, values from “earlier” files are overridden by “later” files.

8)
Location and names of config files
-----------------------------------

Type of file : Location and filename
system : prefix/lib/pythonver/distutils/distutils.cfg
personal : $HOME/.pydistutils.cfg
local : setup.cfg <====


9)
Syntax of config files
-----------------------

The Distutils configuration files all have the same syntax. The config files are grouped into sections. There is one section for each 'Distutils command', plus a "global" section for global options that affect every command. Each section consists of one option per line, specified as "option=value".
And if it is used as the "setup.cfg" for a particular module distribution, it affects only that distribution.

10)
Writing the Setup Configuration File
-------------------------------------

http://docs.python.org/2/distutils/configfile.html

a)
Configuration files also let you provide default values for any command option, which the installer can then override either on the command-line or by editing the config file.

 In fact, "setup.cfg" are processed after the contents of the setup script, but before the command-line. This has several useful consequences:

* installers can override some of what you put in setup.py by editing setup.cfg
* you can provide non-standard defaults for options that are not easily set in setup.py
* installers can override anything in setup.cfg using the command-line options to setup.py

b)
The basic syntax of the configuration file is simple:

[command]
option=value
...


where command is one of the Distutils commands (e.g. build_py, install), and option is one of the options that command supports. Any number of options can be supplied for each command, and any number of command sections can be included in the file.

You can find out the list of options supported by a particular command with the universal --help option, e.g.
#python setup.py --help build_ext

11)
Tips for Configuration File
----------------------------

a)
Some configuration file sections
Some "python setup.py" commands
* metadata
* global
* files
* build_sphinx
* nosetests

b)
* Run "python setup.py --help-commands" to see all commands.
* Run "python setup.py cmd --help" to see all options of a command.
Example:
#python setup.py build_sphinx --help
#python setup.py nosetests --help


c)
https://github.com/openstack/nova/blob/master/setup.cfg
https://github.com/openstack/nova/blob/master/setup.py

https://github.com/openstack/horizon/blob/master/setup.cfg
https://github.com/openstack/horizon/blob/master/setup.py

https://github.com/openstack/cinder/blob/master/setup.cfg
https://github.com/openstack/cinder/blob/master/setup.py

https://github.com/openstack/glance/blob/master/setup.cfg
https://github.com/openstack/glance/blob/master/setup.py

https://github.com/openstack/swift/blob/master/setup.cfg
https://github.com/openstack/swift/blob/master/setup.py

12)
Getting Started With setuptools and setup.py
---------------------------------------------

a)
https://pythonhosted.org/an_example_pypi_project/setuptools.html
http://pythonhosted.org/setuptools/setuptools.html

b)
Example setup.py
import setuptools
setuptools.setup(setup_requires=['pbr'], pbr=True)


http://docs.openstack.org/developer/pbr/

pbr is a library for managing setuptools packaging needs in a consistent manner.pbr reads and then filters the setup.cfg data through a setup hook to fill in default values and provide more sensible behaviors, and then feeds the results in as the arguments to a call to setup.py - so the heavy lifting of handling python packaging needs is still being done by setuptools.

13)
manifest template, called MANIFEST.in
-----------------------------------------------------
http://docs.python.org/2/distutils/sourcedist.html

a)
The manifest template is just a list of instructions for how to generate your manifest file, MANIFEST, which is the exact list of files to include in your source distribution. The sdist command processes this template and generates a manifest based on its instructions and what it finds in the filesystem.

b)
MANIFEST.in tells Distutils what files to include in the source distribution but it does not directly affect what files are installed. For that you need to include the appropriate files in the setup.py file, generally either as package data or as additional files.