Build and deploy scientific Python Applications
2 likes764 views
The document discusses best practices for building and deploying scientific Python applications, focusing on essential concepts like application binary interfaces (ABIs), operating system stability, and memory management. It covers tools for packaging and deploying applications on different operating systems, memory management for developers, and examples of versioning frameworks. Additionally, it emphasizes the importance of a well-defined development and deployment environment using tools like virtualenv, Docker, and configuration management systems like Ansible and Chef.
![Virtualenv
Setting up the virtualenv can take a significant amount of time
because it pulls down packages from PyPI. I was surprised that it does
not try to use the locally installed packages. One way to speed up the
rebuild is to use a local caching mirror of PyPI. This can be
accomplished by using devpi.
●
Step 1:
Install devpi package
sudo pip install devpi
●
Step 2: Add the following lines in your /root/.pip/pip.conf file
[global]
index-url = http://localhost:3141/rooint/pypi/+simple/
extra-index-url = https://pypi.python.org/simple/](https://image.slidesharecdn.com/pythondeploy-scipy2014-150123074704-conversion-gate02/85/Build-and-deploy-scientific-Python-Applications-10-320.jpg)
Build and deploy scientific Python Applications
- 1. Build and Deploy Scientific Python Applications SciPy India 2014|IIT Bombay Ramakrishna Reddy Yekulla http://ramkrsna.net/ @ramkrsna
- 2. Prerequisites ● Have written python or any applications ● Basic Linux Commands
- 3. How is this talk going to go about ● Binary Interfaces and Stability ● Python packaging for Application Developer ● Packaging your Scipy app for *nix based OS ● Deployments | Docker
- 4. Application Binary Interfaces ● An application binary interface (ABI) is the interface between two program modules, one of which is often a library or operating system, at the level of machine code. An ABI determines such details as how functions are called and in which binary format information should be passed from one program component to the next, or to the operating system in the case of a system call. ● Adhering to ABIs (which may or may not be officially standardized) is usually the job of the compiler, OS or library writer, but application programmers may have to deal with ABIs directly when writing programs in a mix of programming languages, using foreign function call interfaces between them. ● ABIs differ from application programming interfaces (APIs), which similarly define interfaces between program components, but at the source code level.
- 5. Operating System Stability ● Filesystem Reliability ● Glibc Interfaces ~9000 Stable Interfaces ● Robust Tested Software both by the community and Enterprise Software Industry
- 6. Operating System Stability ● Separation of concerns ● Reliability ● Repeatability. ● Ease of use. ● Built it, Run it. ● Tools of the trade :: Binutils – readelf, objdump, nm, c+ +filt Elfutils – eureadelf, euobjdump, eu nm
- 7. System Packaging Solutions ● GNU/Linux :: apt-get | dpkg, yum | RPM ● Windows :: npackd | chocolatey ● Mac OS X :: homebrew | macports
- 8. What Every Developer Needs to Know about Memory ● Ulrich Drepper
- 9. What Every Python Developer should be aware of.. ● distutils ● buildout ● virtualenv ● pip ● setup.py develop ● **conda
- 10. Virtualenv Setting up the virtualenv can take a significant amount of time because it pulls down packages from PyPI. I was surprised that it does not try to use the locally installed packages. One way to speed up the rebuild is to use a local caching mirror of PyPI. This can be accomplished by using devpi. ● Step 1: Install devpi package sudo pip install devpi ● Step 2: Add the following lines in your /root/.pip/pip.conf file [global] index-url = http://localhost:3141/rooint/pypi/+simple/ extra-index-url = https://pypi.python.org/simple/
- 11. Python Apps on Distributions ● RPM's Based Deployments ● Software Collections ● Context based Security Benefits ● PACKER | Virt-Builder
- 12. General Rule Of Packaging ● MAJOR version when you make incompatible API changes ● MINOR version when you add functionality in a backwards-compatible manner, and ● PATCH version when you make backwards-compatible bug fixes. ● Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format. http://semver.org https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/ http://nvie.com/posts/pin-your-packages/
- 13. Fedora ● In Fedora we have multiple python runtimes, one for each supported major release. At this point that's one for python2.x and one for python3.x ● Each runtime corresponds to a binary of the form ● /usr/bin/python$MAJOR.$MINOR ● One of these python runtimes is the "system runtime" which is what we run when invoking ● All python runtimes have a virtual provide for ● python(abi) = $MAJOR-$MINOR . ● /usr/bin/python . ● On Fedora 20 this is ● /usr/bin/python2.7
- 14. Package Versioning ● MAJOR version when you make incompatible API changes ● MINOR version when you add functionality in a backwards-compatible manner, and ● PATCH version when you make backwards-compatible bug fixes. ● Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format. http://semver.org https://access.redhat.com/documentation/en-US/Red_Hat_Software_Co llections/ http://nvie.com/posts/pin-your-packages/
- 15. Typical Dev Setup ● Vagrant, virtualenv, virtualenvwrapper ● Zero configuration Python/Unix web based teaching env :: https://github.com/ptone/jiffylab ● Another Approach for Dev Environment :: http://collaboratool.berkeley.edu
- 16. Typical Deploy Setup ● BCFG2, Python-based but XML-infested. ● CFEngine ● Ansible, Python-based, gaining a lot of traction lately ● Chef, Puppet ( Ruby Based) ● Salt, Python ● Juju, Ubuntu ● Docker