Writing a Python C extension
0 likes182 views
This document discusses different approaches for creating Python extensions and bindings to C/C++ libraries. It summarizes the author's experience using ctypes to create a minimal binding called PyMiniRacer to the V8 JavaScript engine. The author argues that combining ctypes, which allows shipping a single Python-independent binary, with pre-built wheel distributions can provide an optimal solution for packaging and distributing Python extensions.
![How to put this together?
$ cat setup.py
from distutils.core import setup, Extension
extension = Extension('hello', ['hello.c'])
setup(name=‘hello',
version='1.0',
ext_modules=[extension])
$ python setup.py build
running build
running build_ext
building 'hello' extension
clang […] -c hello.c -o hello.o
creating build/lib.macosx-10.6-intel-2.7
clang -bundle […] hello.o -o hello.so](https://image.slidesharecdn.com/2018-05-16writingapythoncextension-180517092149/85/Writing-a-Python-C-extension-14-320.jpg)
Writing a Python C extension
- 1. Writing a C Python extension in 2018 Paris.py 2018
- 2. Jean-Baptiste Aviat CTO & Co-founder of sqreen.io Former Hacker at Apple (Red Team) PyMiniRacer author [email protected] @jbaviat Who Am I?
- 3. Someday… we needed to use V8 from Python. What we ship: • is public • is widely used • need to have frictionless install $ pip install sqreen —> it just works™!
- 5. PyMiniRacer: cool JS binding github.com/sqreen/PyMiniRacer
- 6. The problem • V8 is C++ • How do you run C++ in Python? …We need some kind of binding between these 2 worlds.
- 7. Many popular packages rely on binary cryptography numpy pymongo psycopg simplejson lxml sqlalchemy…
- 8. People do it! Let’s do it too.
- 9. What are our goals? • minimize maintenance • make setup easy • make testing easy • have great performance • have low memory fingerprint • dev time is a constraint We want to: And (obviously)…
- 10. built-in pythonic Python version independant open to other languages high throughput capable CPython ✔ ✔ ✔ ctypes ✔ ✔ ✔ cffi ✔ ✔ ✔ ✔ Cython ✔ ✔ SWIG ✔ ✔
- 11. ctypes Built into Python Binary is Python independant: • can be used on any Python version • can be used in other languages! No tight integration to Python • not high throughput capable • less Pythonic Complex syntax (C types wrapped in Python…) Not for C++ 👍 👎
- 12. $ python >>> path = "./hello.so" >>> import ctypes >>> lib = ctypes.cdll.LoadLibrary(path) >>> lib.hello_world() Hello world! C file Python interface binary object
- 13. Overview V8 (C++ interface)C interface to V8Python interface 3rd party binaries import ctypes class PyMiniRacer(object): … #include <v8.h> int miniracer_init(); … V8 library (libv8.a) V8 headers (v8.h) linkingctypes C/C++ code Python library
- 14. How to put this together? $ cat setup.py from distutils.core import setup, Extension extension = Extension('hello', ['hello.c']) setup(name=‘hello', version='1.0', ext_modules=[extension]) $ python setup.py build running build running build_ext building 'hello' extension clang […] -c hello.c -o hello.o creating build/lib.macosx-10.6-intel-2.7 clang -bundle […] hello.o -o hello.so
- 17. Ship it
- 18. Ever had trouble installing packages?
- 20. (at Sqreen, we are dog friendly) Come get your dog sticker!
- 21. Python packaging history sdist (source distribution) eggs wheels —> manylinux wheels (built distribution) 2004 2012 2016 Python 2.4 Python 3.3 Python 3.6 ❤
- 22. manylinux wheels Python standard: PEP513 / PEP571 Compatible on most (real world) Linux Only in pip >= 8.1 Need build for all Python versions Binaries need to be built on 🚨🚨CentOS 5🚨🚨 👍 👎
- 23. CentOS 5?? PEP571 → bumped requirement to CentOS 6!
- 24. Wheels vs compilation iso builds (crash can be reproduced) many packages need maintenance one build per user only one package harder to install…
- 25. Best of 2 worlds We can have: ctypes: use a Python independant binary wheels: ship compiled binaries By combining these… 🥂🍾 ctypes + wheels = ship only one binary