Asynchronous Python with Twisted
Download as PPTX, PDF5 likes1,865 views
The document provides an overview of asynchronous programming, highlighting its advantages such as efficient CPU utilization and reduced memory footprint per connection. It introduces Twisted, an asynchronous framework for Python, detailing its deferred objects, callbacks, and error handling mechanisms. Additionally, it covers the Twisted web server features and provides examples of how to create resources and handle requests concurrently.
Asynchronous Python with Twisted
- 2. Who Am I? Developer/Evangelist at LaunchKey Organizer of Las Vegas Python User Group Founder/Co-Organizer of Las Vegas PHP UG Co-Organizer of Las Vegas Developer UG National Junior Basketball Coach
- 3. What is Asynchronous Programming? • Non-Linear • Event Driven • Functional • Depends on non-blocking code
- 4. Why Asynchronous? One thread for all connections with the following benefits: Shared application level objects and data reduces memory footprint per connection Limited threads allows for better CPU utilization of each core thus fewer cores overall. Both of the above enable large applications to run on smaller virtual machines in the cloud.
- 5. What’s the Catch? Functional non-blocking code development has a learning curve. Developing applications with objects that persist beyond a single request can be more difficult than standard WSGI applications. Requires asynchronous libraries which can be scarce in Python.
- 6. Twisted Established Asynchronous Python Framework 1.0 Release was in March 2004 Utilizes asynchronous I/O Server runs in Python 2.6+ and 3.2+
- 7. Deferred = Asynchronous Analogous to promises/futures Returned by asynchronous and concurrent calls Use callbacks to process result of fulfillment Process callbacks in the order they are registered with each being passed the result of the predecessor Can be quickly created for static data with defer.sucess and defer.failure functions
- 8. Callback Example d = defer.Deferred() d.addCallback(lambda x: x + “ is an “) d.addCallback(lambda x: x + “ example!“) d.callback(“This”) >>> This is an example!
- 9. Deferred Error Handling Deferred objects catch all callback Exceptions Exceptions are wrapped in a Failure object which can also be returned by a callback or errback If no errbacks remain, exception is swallowed and processing is stopped Remaining callbacks before next errback are skipped If errback returns non-Failure, callback processing resumes
- 10. Callback Non-Failure Flow Deferred callback(result) • callback - success • callback – success • errback - skipped • errback – skipped
- 11. Errback Non-Failure Flow Deferred callback(result) • callback - success • errback - failure • callback – success • errback – skipped
- 12. Errback Failure Flow Deferred callback(result) • callback - success • callback – failure • errback - failure • errback – success
- 13. Threads = Concurrent Threads run in thread pool managed by reactor callInThread executes function in its own thread with supplied data deferToThread works like callInThread but returns a Deferred object and processes the callbacks when complete Use threads for blocking code
- 14. Reactor Manages the main event loop Waits for and dispatches events or messages May have predefined services to provide access to network communications, threading, and event dispatching. Available services are operating system dependent
- 16. Application Starts and stops the reactor Manages services Handles logging Can run as daemon
- 17. Service Registers with the reactor Processed in the main event loop Manages connections Communicates via Protocols
- 18. Predefined Services File Descriptor Process Threads Time (scheduling) SSL TCP UDP UNIX (UNIX sockets) Socket (3rd Party Sockets)
- 19. Protocol Utilized for interaction with services Services with stateful connections utilize protocol factories to generate a protocol instance per connection. Provided with method to send data to connection in the form of transports or producers
- 20. Protocol Factory Builds protocol instances Injected into every protocol instance Meant to perform inter-instance interaction Can be used as testable registry for registry pattern implementations
- 22. Available Protocols Mongo LDAP Dict Finger FTP GPS HTB HTTP Ident Loopback Memcache Mice PCP Portforward Postfix Shoutcast SIP SOCKSv4 Stateful Telnet TLS Wire MongoDB* LDAP* * External packages maintained by Twisted Matrix Labs
- 23. Twisted Web Web server built on Twisted HTTP Protocol Can be stand alone web server Can embedded as part of an application Supports all of the features of most modern web servers
- 24. Web Server Features Virtual hosts Proxy/Reverse proxy Scripts (hello.py) Static files CGI WSGI URL rewrites MIME processors Sessions
- 25. Simple Web Application Site object is root Resource Resource object represents URL segment Tree hierarchy of resources Can directly return string or write response via request object for deferred processing
- 27. Example Hierarchy Code root = RootResource() foo = FooResource() foo.putChild(“bar”, BarResource()) foo.putChild(“bat”, BatResource()) root.putChild(“foo”, foo) root.putChild(“fiz/buzz”, FizBuzzResource()) reactor.listenTCP(8080, Site(root)) reactor.start()
- 28. Concurrent Example from twisted.web import resource class DemoResource(resource.Resource): def render_GET(self, request): return "<html>foo</html>"
- 29. Deferred Example def render_GET(self, request): def process_resp(html, request): request.write(html) request.finish() deferred = self.async_client.request() deferred.addCallback(process_resp, request) return NOT_DONE_YET
- 30. Demo Time
- 31. Example on GitHub: https://github.com/aenglander/demo-python-twisted- websocket @adam_englander #launchkey on freenode.net #vegastech on freenode.net [email protected]