Deferred Processing in Ruby - Philly rb - August 2011

1. Just Do It Later...Procrastination in Ruby Philly.Rb – August 9, 2011

2. Hi I'm Rob Twitter: iotr

3. Github: https://github.com/robdimarco/

4. Email: rob @ 416 software . com

5. More: http://www.innovationontherun.com

6. This Presentation: https://github.com/robdimarco/deferred_processing_talk

7. Out of Stream Processing

8. Always Looking to Speed Up User Experience

9. One Way to Get Faster..Do Less

10. Targets for Deferral Sending email

11. Image resizing

12. HTTP/FTP downloads

13. Updating indexes

14. Batch import / exports

15. Analytics

17. Thread.fork {do_stuff}

18. Using Threads Easy... But Process cannot exit before thread

19. Work kept in same process

20. Need to control thread count Use when Runtime makes sense within current process

21. Shared access to data is desired

22. Want communication with other threads

23. pid = fork {do_stuff} Process.detach pid # Maybe??

24. Using Forks Easy...But Need to control processes

25. Need to control output Use if Separate resources desired

26. Runtime outside of main process but on same machine

27. Let's Take It Up A Notch Queueing!!!

28. “Queues” Queue Request Response Push Pop

29. Delayed::Job

30. Delayed::Job Brought to you by Shopify

31. Your code creates a Delayed::Job persisted object

32. Worker will go through and work off jobs

33. Installation Gemfile gem 'delayed_job' script/rails generate delayed_job Creates migration

34. Adds script/delayed_job rake db:migrate

35. Using delay Any method can be converted into a delayed job by calling delay

36. User.find(1).follow(User.find(2)

37. becomes

38. User.find(1).delay.follow!(User.find(2))

39. Using enqueue Can create an object that defines a perform method

40. Enqueue with

41. Delayed::Job.enqueue(MyJob.new)

42. Delayed::Job Configuration On delay/enqueue :priority – a number

43. :run_at – a time As defaults sleep_delay

44. max_attempts

45. max_run_time

46. destroy_failed_jobs

47. Persistence Saved to a data store

48. By default, uses ActiveRecord, other frameworks available

49. Stores job off using YAML !! Model objects may have been changed between enqueuing and processing !!

50. Processing The Jobs Start the script/delayed_job Can pass in # of workers and priority

51. Uses daemons library Workers lock jobs by editing the object in the database

52. Errors are stored alongside the job

53. Operating Delayed::Job Need same code that you used to enqueue to process

54. One solitary queue with priorities, cannot create individual queues

55. Have had problem with workers processing long jobs not dying

56. Big queues can lead to database contention

57. What's To Like Simple installation

58. Minimally invasive to object model Any object can be queued Numeric priorities

59. Maybe Not Right For You If... Want multiple queues

60. Want “out of the box” GUI

61. Big queues

62. Lots of workers

63. Want to process outside your code base

64. Resque

65. Resque Brought to you by GitHub

66. Uses Redis to persist queues

67. Rake task for starting a worker

68. Sinatra app for monitoring queues, jobs, and workers.

69. Installation Install Redis

70. Gemfile gem 'resque'

71. require 'resque' Configuration config/initializers/resque.rb

72. config/resque.yml

73. Enqueue Class that has class methods perform – Method to do the work

74. queue – The name of the queue to use (symbol/string) Enqueue with Resque.enqueue(class, *args)

75. More on Methods Arguments to enqueue are serialized to JSON Pro tip...only send in objects that can be encoded/decoded to JSON queue can be class attribute or class method

76. perform takes in arguments sent to enqueue

77. Resque Admin Sinatra application

78. Standalone or mounted in your application

79. Good snapshot of what is happening

80. About Redis Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

81. In-memory data set Can persist to disk Easy m aster-slave replication

82. Resque Uses Redis For... Storing list of queues in a set

83. Creating a list of jobs for each queue

84. Tracking workers

85. Maintaining stats

86. Enqueuing a User Follow Add queue and perform methods to the User class

87. Enqueue the job

88. Resque.enqueue User, 2, 1

89. Process echo “require 'resque/tasks'” > lib/tasks/resque.rake QUEUE=* rake environment resque:work Forks off child to do actual work Makes it easier to stop/start

90. Looks to see if you are using Ruby EE

91. What's To Like Fast on/off queue

92. Large queues

93. Multiple queues

94. Nice admin tool

95. Workers resistant to job quirks

96. Maybe Not Right For You If... Want priorities

97. Do not want Redis

98. Do not want queues in RAM

99. Do not want JSON serialization

100. Want to manually query the job queue

101. Amazon SQS

102. What is SQS Web-based Message Queue Language agnostic On Amazon's Cloud High Availability

103. Scalable Queue level security

104. Using SQS Requires AWS Account Currently 100,000 messages / month free

105. $.01 / 10,000 messages after that All interactions over SOAP queries

106. Using RightAws gem to abstract this away

107. How to Queue Up Jobs sqs = RightAws::SqsGen2.new( ENV['ACCESS_KEY'], ENV['SECRET_KEY']) q = sqs.queue('my_queue_name') q.push('my data')

108. Processing sqs = RightAws::SqsGen2.new( ENV['ACCESS_KEY'], ENV['SECRET_KEY']) q = sqs.queue('my_queue_name') loop;process(q.pop);end

109. What's To Like Easy to process out of your application code E.g. processing needs to happen on Windows Easy to process out of your data center

110. Easy to add/remove queues

111. No additional services to manage

112. Maybe Not Right For You If... You do not want to figure out the serialization / de-serialization

113. You do not want to handle worker process management

114. You want to minimize processing time / job.

115. AMQP Messaging - The Big Gun

116. AMQP Advanced Message Queue Protocol

117. Protocol defining how messaging clients and brokers will interact

118. Open, c ross-vendor specification

119. ruby-amqp Requires EventMachine

120. Connects to an AMQP-compliant server Rabbit MQ

121. Apache Qpid Will need to specify host, channel, queue, and message

122. Look Into AMQP-Based System If Needs extend beyond just simple queue Publish/subscribe

123. Transactional messaging High volume / fast processing

124. Cross-systems

125. Other Options Beanstalkd Standalone server Starling From twitter, speaks memcached SimpleWorker SaaS job processing http://www.simpleworker.com/ Cloud Crowd Map-reduce framework

126. Key Links Source Code https://github.com/robdimarco/deferred_processing_talk Delayed::Job https://github.com/collectiveidea/delayed_job Resque https://github.com/defunkt/resque

127. https://github.com/blog/542-introducing-resque

128. More Links SQS http://aws.amazon.com/sqs/

129. https://github.com/rightscale/right_aws AMQP http://rdoc.info/github/ruby-amqp/amqp/master/file/docs/GettingStarted.textile

130. http://www.amqp.org/ More http://ruby-toolbox.com/categories/queueing.html

131. http://kr.github.com/beanstalkd/

Deferred Processing in Ruby - Philly rb - August 2011

Recommended

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Deferred Processing in Ruby - Philly rb - August 2011 (20)

Recently uploaded (20)

Deferred Processing in Ruby - Philly rb - August 2011