More Related Content
What's hot (20)
Viewers also liked (20)
Similar to Background processes and tasks in an async world (20)
Recently uploaded (20)
Background processes and tasks in an async world
- 2. background jobs in an async world Cody Stoltman @particlebanana
- 3. sponsors
- 4. who are you? Cody Stoltman @particlebanana co-founder of Treeline Sails.js core team maintainer of Waterline ORM
- 5. what’s a background job?
- 6. browser /foo/bar does some stuff normal request/response lifecycle
- 7. browser /foo/bar does some stuff slow function long running process lifecycle
- 8. browser /foo/bar does some stuff slow function
- 9. this isn’t ruby node is async we have callbacks
- 10. what does blocking mean?
- 11. browser 1 traditionally browser 2 server do something
- 12. browser 1 with node.js browser 2 server do something browser 3
- 13. browser /foo/bar does some stuff slow fuction
- 14. how can we handle that function? • callbacks • just ignore the callback • over engineer something related to process spawning
- 16. why this isn’t ideal we aren’t blocking so the server can keep responding but we have to wait until the process is done before responding to the client’s request
- 17. how can we handle that function? • callbacks • just ignore the callback • over engineer something related to process spawning
- 18. browser /foo/bar does some stuff slow function
- 20. why this isn’t ideal the client get’s data back but we have no clue as to the success/failure of the task. we could add logs but we have no way to give the client feedback or to re-try.
- 21. how can we handle that function? • callbacks • just ignore the callback • over engineer something related to process spawning
- 22. let’s learn from our synchronous, threaded brothers and sisters whats the answer then?
- 23. how can we handle that function? • remote queues
- 24. remote queue queue worker slow function server does some stuff job browser job
- 25. why this is better We can observe our queue We can scale out horizontally and add capacity as needed Our queue can be preserved so no jobs are abandoned
- 26. scalable remote queue server server server server worker worker worker worker load balancer queue
- 27. back to that event loop thing
- 29. call stack libuv api callback queue event loop .on() handler handler ()
- 30. lets see what blocking actually means
- 31. call stack libuv api callback queue event loop .on() handler handler () delayFn () handler handler handler handler ()
- 32. so synchronous code blocks the event loop, don’t do that what happens if we don’t block it?
- 33. call stack libuv api callback queue event loop .on() handler handler() sendEmail() queue cbhandler handler cb()handler() sendEmail() queue cb
- 34. remote queues are good because they let you do more, faster so more work in the event loop means less concurrency
- 35. when should you move a function into a background job?
- 36. if for any reason it will block the event loop
- 37. it will prevent the response from returning in a timely manner
- 39. most queues are built on redis
- 41. you could also build on a traditional message queue
- 42. summary remote queues in node same as anywhere else actually better because you can do more on a single cpu simultaneously (example: sending emails)
- 43. scheduled tasks
- 44. you need a database backed scheduler
- 45. someone should write one agenda (mongo only) github.com/rschmukler/agenda until then
- 46. demo
- 47. thanks! Phillip Roberts (@latentflip) github.com/latentflip/loupe Evan Tahler (@evantahler) blog.evantahler.com