Why Task Queues - ComoRichWeb

Why Task Queues in
Python, Ruby and More!
~ a talk by Bryan Helmig

task queue noun ˈtask ˈkyü
a system for parallel execution of discrete
tasks in a non-blocking fashion.
celery, resque, or home-grown...

broker noun broh-ker
the middle man holding the tasks
(messages) themselves.
rabbitmq, gearman, redis, etc...

producer noun pruh-doo-ser
the code that places the tasks to be
executed later in the broker.
your application code!

consumer noun kuhn-soo-mer
also called the worker, takes tasks from
the broker and perform them.
usually a daemon under supervision.

imagine this...
your app: posting a new message triggers
“new message” emails to all your friends.

def new_message(request):
user = get_user_or_404(request)
message = request.POST.get('message', None)

if not message:
raise Http404

user.save_new_message(message)

for friend in user.friends.all():
friend.send_email(message)

return redirect(reverse('dashboard'))

the problem:
that works good for like 0-6 friends... but
what if you have 100,000? or more? will
your user ever get a response?

bad idea #1: ignore
make your users wait through your long
request/response cycle.
your users are important, right?

bad idea #2: ajax
return the page fast with JS code that calls
another script in the browser’s background.
duplicate calls & http cycles are not cool.

bad idea #3: cronjob
make a email_friends table with user_id &
message column. cron every 5/10 minutes.
backlogs will destroy you.

good idea: queues
the task to potentially email thousands of
users is put into a queue to be dealt with
later, leaving you to return the response.

@task
def alert_friends(user_id, message):
user = User.objects.get(id=user_id)

for friend in user.friends.all():

def new_message(request):
user = get_user_or_404(request)
message = request.POST.get('message', None)
if not message:
raise Http404
user.save_new_message(message)

alert_friends.delay(user.id, message)

return redirect(reverse('dashboard'))

RULE #1:
adding a task to a queue should be
faster than performing the task itself.

RULE #2:
you should consume tasks faster than you
produce them. if not, add more workers.

queue libraries:
1. celery (python) earlier...
2. resque (ruby) up next...
3. build your own grand finalé!

class MessageSend
def self.perform(user_id, message)
user = User.find(user_id)
user.friends.each do |friend|
end
end
end

class MessageController < ActionController::Base
def new_message
render(file: 'public/404.html',
status: 404) unless params[:message]
current_user.save_new_message(params[:message])

Resque.enqueue(MessageSend,
current_user.id, params[:message])

redirect_to dashboard_url
end
end

let’s build our own!
and let’s do it with redis, in php and
in less than 50 lines of code!
(not including the task code)

quick redis aside:
so, redis is a key-value store. values can
be strings, blobs, lists or hashes. we’ll use
lists and RPUSH and LPOP.

step #1: task runner
i’m envisioning this as a Task class with
various methods named after each task.
each method accepts an array $params.

class Tasks {
public function email_friends($params) {
$user_id = $params->user_id;
$message = $params->message;

# get $friends from a db query...
$friends = array('paul@example.com',
'john@example.com');

foreach ($friends as $friend) {
echo "Fake email ".$friend.
" with ".$message."n";
}
}
}

step #2: worker
i think an infinite loop with a blocking
redis BLPOP will work. then it needs to run
the right method on the Task object.

include_once 'redis.php'; # sets up $redis, $queue
include_once 'tasks.php';

$tasks = new Tasks();

while (true) {
# BLPOP will block until it gets an item..
$data = $redis->blpop($queue, 0);
$obj = json_decode($data[1]); # grab json...

$method = $obj->task_name;
$params = $obj->params;

# calls the task: Task->$task_name($params)...
call_user_func(array($tasks, $method), $params);
}

step #3: run it!
we will need to run the worker:

➜ php worker.php

step #4: producer
let’s make a handy add_task function for
easy placement of tasks into the queue.

include_once 'redis.php'; # sets up $redis & $queue

function add_task($task_name, $params) {
global $redis, $queue;
$data = Array('task_name' => $task_name,
'params' => $params);
$json = json_encode($data)
$redis->rpush($queue, $json);
}

# an example of our task api...
add_task('email_friends',
array('user_id' => 1234,
'message' => 'I just bought a car!')
);

step #5: watch it
a command line that runs a worker is
standard. use something like supervisord
or god to run it & monitor it.

things missing:
our php example does not: store return
results, handle errors, route tasks,
degrade gracefully, log activity, etc...

Why Task Queues - ComoRichWeb

Recommended

More Related Content

What's hot (20)

Similar to Why Task Queues - ComoRichWeb (20)

Recently uploaded (20)

Why Task Queues - ComoRichWeb