Python meetup: coroutines, event loops, and non-blocking I/O
6 likes2,677 views
This document discusses asynchronous programming concepts like non-blocking I/O, event loops, coroutines, and Python libraries that support them like Twisted, gevent, and asyncio. Coffee metaphors are used to explain blocking vs non-blocking I/O. Coroutines and generators allow functions to pause and resume while yielding to the event loop. Libraries like Twisted focus on networking protocols while gevent aims to make synchronous code asynchronous via monkey patching. asyncio is part of the Python standard library and uses yield from to support nested coroutines.
![magic
import asyncio
import asyncio_redis
!
@asyncio.coroutine
def my_subscriber(channel):
# Create connection
connection = yield from asyncio_redis.Connection.create(host='localhost',
port=6379)
# Create subscriber.
subscriber = yield from connection.start_subscribe()
# Subscribe to channel.
yield from subscriber.subscribe([channel])
# Wait for incoming events.
while True:
reply = yield from subscriber.next_published()
print('Received: ', repr(reply.value), 'on channel', reply.channel)
!
loop = asyncio.get_event_loop()
asyncio.async(my_subscriber('channel-1'))
asyncio.async(my_subscriber('channel-2'))
loop.run_forever()
source: asyncio-redis](https://image.slidesharecdn.com/coroutines-140620052056-phpapp02/85/Python-meetup-coroutines-event-loops-and-non-blocking-I-O-2-320.jpg)
![yield to the event loop
import asyncio
import asyncio_redis
!
@asyncio.coroutine
def my_subscriber(channels):
# [snip]
# Wait for incoming events.
while True:
reply = yield from subscriber.next_published()
print('Received: ', repr(reply.value), 'on channel', reply.channel)
!
loop = asyncio.get_event_loop()
asyncio.async(my_subscriber('channel-1'))
asyncio.async(my_subscriber('channel-2'))
loop.run_forever()](https://image.slidesharecdn.com/coroutines-140620052056-phpapp02/85/Python-meetup-coroutines-event-loops-and-non-blocking-I-O-12-320.jpg)
![gevent
def a_generator():
yield from nested_generator()
def nested_generator():
yield from deeper_nested_generator()
def deeper_nested_generator():
event_loop.register(self, for_io_op='recv',
on_socket=a_socket)
yield
return a_socket.recv()
from gevent import monkey; monkey.patch_all()
!
def a_function():
nested_function()
def nested_function():
deeper_nested_function()
def deeper_nested_function():
return a_socket.recv() # monkey-patched!
!
import gevent
jobs = [gevent.spawn(a_function) for _ in range(5)]
gevent.wait(jobs)
how?!](https://image.slidesharecdn.com/coroutines-140620052056-phpapp02/85/Python-meetup-coroutines-event-loops-and-non-blocking-I-O-21-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (11)
Ad
Similar to Python meetup: coroutines, event loops, and non-blocking I/O (20)
Ad
More from Buzzcapture (20)
Recently uploaded (20)
Python meetup: coroutines, event loops, and non-blocking I/O
- 1. Tikitu de Jager • @tTikitu • [email protected] going async coroutines event loops non-blocking I/O PUN • Utrecht • 20-6-2014
- 2. magic import asyncio import asyncio_redis ! @asyncio.coroutine def my_subscriber(channel): # Create connection connection = yield from asyncio_redis.Connection.create(host='localhost', port=6379) # Create subscriber. subscriber = yield from connection.start_subscribe() # Subscribe to channel. yield from subscriber.subscribe([channel]) # Wait for incoming events. while True: reply = yield from subscriber.next_published() print('Received: ', repr(reply.value), 'on channel', reply.channel) ! loop = asyncio.get_event_loop() asyncio.async(my_subscriber('channel-1')) asyncio.async(my_subscriber('channel-2')) loop.run_forever() source: asyncio-redis
- 3. non-blocking I/O queueing for coffee starbucks just waiting around
- 4. coffee as metaphor for I/O ❖ blocking I/O is queueing for coffee ❖ guy in front wants 17 litres of kopi luwak ❖ all I want is an espresso ❖ non-blocking I/O is the starbucks model ❖ give your order, then go wait somewhere ❖ they call you back when it’s ready
- 5. “non-blocking”? starbucks queueing is not useful if your application is ❖ CPU-bound ❖ I/O-bound by pushing bits it is useful if you spend most of your time waiting doing stuff still takes time
- 6. waiting most I/O is not pushing bits: ❖ server waits for connections ❖ call a service: wait for response ❖ wait for socket buffer to fill if you’re just waiting: yield the CPU … but then who will give it back to you? …
- 7. event loop did anything happen? how about now? callback hell
- 8. you know this ❖ GUI programming: “when this button is clicked…” ❖ (old-fashioned) javascript onclick &c ❖ event loop checks for events and runs callbacks ❖ (select module makes polling for events easy)
- 9. callbacks for non-blocking I/O? a_socket.recv(bufsize=16) event_loop.when_socket_has_ready_buffer(a_s, 16, callback_f) “callback hell”
- 11. coroutines and generators a coroutine is a routine (function) that can pause and resume its execution def a_coroutine(): do_some_stuff() yield do_some_more_stuff() def a_coroutine(): do_some_stuff() yield to some_other_coroutine # invented syntax do_some_more_stuff() A generator is a coroutine that can only yield to its caller
- 12. yield to the event loop import asyncio import asyncio_redis ! @asyncio.coroutine def my_subscriber(channels): # [snip] # Wait for incoming events. while True: reply = yield from subscriber.next_published() print('Received: ', repr(reply.value), 'on channel', reply.channel) ! loop = asyncio.get_event_loop() asyncio.async(my_subscriber('channel-1')) asyncio.async(my_subscriber('channel-2')) loop.run_forever()
- 13. roll-your-own event loop def loop(): while True: for coroutine in waiting_list: if io_is_ready_for(coroutine): running_list.push(coroutine) coroutine = running_list.pop() coroutine.next() (p.s. don’t do this)
- 15. twisted ❖ networking protocols ❖ callbacks (methods on a “protocol” class) ❖ e.g. connectionMade(self), dataReceived(self, data) ❖ “you don't port an application to Twisted: You write a Twisted application in most cases.” —Jesse Noller ❖ “deferred”: abstraction now usually called Future or Promise (proxy for a value that will be computed later)
- 16. asyncio ❖ similar high-level protocols but also ❖ intends to provide a base layer for other libraries ❖ yield from ❖ python3 stdlib ❖ (how I got interested in this whole business)
- 17. yield from event_loop.please_run_for_me(a_generator()) ! def a_generator(): for val in nested_generator(): yield val ! def nested_generator(): for val in deeper_nested_generator(): yield val ! def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv() … but what if we have to support generator send()? def a_function(): nested_function() def nested_function(): deeper_nested_function() def deeper_nested_function(): return a_socket.recv()
- 18. send() the wrong way gen = a_generator() gen.next() gen.send(1) gen.send(2) gen.next() ! def a_generator(): gen = nested_generator() to_yield = gen.next() while True: to_send = yield to_yield if to_send is None: to_yield = gen.next() else: to_yield = gen.send(to_send) D anger! Untested probably incorrect code!
- 19. next() send() throw() close() _i = iter(EXPR) try: _y = next(_i) except StopIteration as _e: _r = _e.value else: while 1: try: _s = yield _y except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e except BaseException as _e: _x = sys.exc_info() try: _m = _i.throw except AttributeError: raise _e else: try: _y = _m(*_x) except StopIteration as _e: _r = _e.value break else: try: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value break RESULT = _r RESULT = yield from EXPR def a_generator(): yield from nested_generator() def nested_generator(): yield from deeper_nested_generator() def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv()
- 20. asyncio ❖ yield from ❖ its own low-level I/O operations (socket recv &c.) ❖ futures, promises, timers, … ❖ networking protocols
- 21. gevent def a_generator(): yield from nested_generator() def nested_generator(): yield from deeper_nested_generator() def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv() from gevent import monkey; monkey.patch_all() ! def a_function(): nested_function() def nested_function(): deeper_nested_function() def deeper_nested_function(): return a_socket.recv() # monkey-patched! ! import gevent jobs = [gevent.spawn(a_function) for _ in range(5)] gevent.wait(jobs) how?!
- 22. greenlets ❖ full coroutines ❖ monkey-patched primitives can “yield to” the event loop directly ❖ C extension for CPython
- 23. node.js really?
- 25. module summary twisted ❖ venerable ❖ focus on networking protocols ❖ perceived as large and complex; porting not easy gevent ❖ near “drop-in” in synchronous code ❖ python 3 support is … coming? asyncio ❖ python 3 stdlib ❖ (“Tulip” is a python 2 port: “yield From(a_generator)”) ❖ aims to be both low-level and protocol-level library
- 26. tech summary greenlets ❖ full coroutines ❖ CPython hack; some pypy support yield from ❖ python 3 ❖ nested generators ❖ goldilocks solution? callbacks ❖ low-tech, no special support needed ❖ promises, futures, etc: there is space for useful abstraction ❖ node.js
- 27. buzzwords ❖ non-blocking I/O: yield the CPU instead of waiting ❖ an event loop gives it back to you when what you’re waiting for happens ❖ coroutines and generators let you write synchronous- style functions and still yield to the event loop mid-way
- 28. that’s all folks and yes we’re hiring
- 29. thank you’s and references ❖ y’all for your attention ❖ @saghul for getting me started on this whole business ❖ Peter Portante for a 2011 PyCon talk on coroutines ❖ Reinout & Maurits for PyGrunn summaries ❖ PEPs: ❖ 3156 (asyncio) ❖ 380 (yield from)