Gearman work queue in php
28 likes8,578 views
This document discusses using Gearman for work queue processing in PHP. It begins with an introduction to Gearman and how it allows work to be distributed across languages and machines. Several examples are provided of using Gearman in PHP for both client and worker applications. CodeIgniter integration is also demonstrated. Overall the document provides an overview of Gearman and examples of implementing work queues in PHP using this system.
![14
<?php
$link = mysqli_connect("host", "user", "passw", "db");
$query = 'SELECT * FROM jobs where status = "on"';
$result = $link->query($query);
while($row = mysqli_fecth_array($result)) {
// …....
// do work
// …....
$sql = 'update jobs set status = "off" where id = {$row["id"]}';
$link->query($sql);
}](https://image.slidesharecdn.com/2013-131005010337-phpapp01/85/Gearman-work-queue-in-php-14-320.jpg)
![19
每 10 秒執行一次
<?php
$link = mysqli_connect("myhost","myuser","mypassw","mybd");
$query = 'SELECT * FROM jobs where status = "on"';
$done = false;
while(!$done) {
$result = $link->query($query);
if ($result->num_rows > 0) {
// do work
$sql = 'update jobs set status = "off" where id = {$row["id"]}';
$link->query($sql);
} else {
sleep(10);
}
}](https://image.slidesharecdn.com/2013-131005010337-phpapp01/85/Gearman-work-queue-in-php-19-320.jpg)
![73
Setup Gearmand Server and Port
config/gearman.php
$config['gearman_server'] = array('localhost');
$config['gearman_port'] = array('4730');](https://image.slidesharecdn.com/2013-131005010337-phpapp01/85/Gearman-work-queue-in-php-73-320.jpg)
![93
Worker Setting
Worker[1-4]:Server[1-2]
…...](https://image.slidesharecdn.com/2013-131005010337-phpapp01/85/Gearman-work-queue-in-php-93-320.jpg)
Gearman work queue in php
- 1. Gearman work queue in php 用 Gearman 打造 PHP 排程 2013.10.05 @appleboy
- 2. 2 Who am I Bo-Yi Wu @appleboy http://blog.wu-boy.com https://github.com/appleboy 任職於瑞昱半導體 RealTek(IC Design House)
- 3. 3 今日來聊聊排程系統
- 4. 4 排程使用時機 ?
- 6. 6 哪些應用需要排程 ?
- 7. 7 使用者註冊系統
- 9. 9 缺點 Response time 過長 使用者不需要等伺服器寄信 ( 來自知名插畫家彎彎 )
- 11. 11 Email Queue? Github Social Coding
- 12. 12 Application Server Database Database Database Work Queue
- 14. 14 <?php $link = mysqli_connect("host", "user", "passw", "db"); $query = 'SELECT * FROM jobs where status = "on"'; $result = $link->query($query); while($row = mysqli_fecth_array($result)) { // ….... // do work // ….... $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); }
- 15. 15 每分鐘執行一次 */1 * * * * root cron.sh
- 18. 18 換個寫法來試試看
- 19. 19 每 10 秒執行一次 <?php $link = mysqli_connect("myhost","myuser","mypassw","mybd"); $query = 'SELECT * FROM jobs where status = "on"'; $done = false; while(!$done) { $result = $link->query($query); if ($result->num_rows > 0) { // do work $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); } else { sleep(10); } }
- 20. 20 缺點 ● 真的 10 秒執行一次 ? ● Race condition ● 更浪費 Database 資源
- 22. 22 #!/bin/sh (sleep 5 && /path/script.sh) & (sleep 15 && /path/script.sh) & (sleep 25 && /path/script.sh) & (sleep 35 && /path/script.sh) & (sleep 45 && /path/script.sh) & (sleep 55 && /path/script.sh) & 每 10 秒執行一次
- 23. 23 如何分配處理這些問題 ?
- 24. 24 Using Gearman
- 25. 25 What is Gearman? It allows you to do work in parallel, to load balance processing, and to call functions between languages.
- 28. 28 How Does Gearman Work?
- 29. 29 Gearman 優點
- 30. 30 Work Queues ● 淺顯易懂 Got job do job repeat→ → Application Gearmand Worker Got job Do job
- 31. 31 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可
- 33. 33 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時， Worker 不需要啟動
- 35. 35 Work Queues ● 淺顯易懂 Get job do job repeat→ → ● 跨語言 任何平台只要能存入字串即可 ● 跨平台 當 Client 寫入工作時， Worker 不需要啟動 ● 即時性 可以為同步或非同步
- 45. 45 不論要幾台 gearmand 跟數個 worker 都可以
- 47. 47 多個 Application 不管 丟出多少個 Jobs
- 53. 53 Installation ● Job Server (C) ● Client library (PHP,Python ...) ● Worker library (PHP,Python ...)
- 55. 55 Ubuntu/Debian aptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0
- 56. 56 CentOS/Fedora yum -y install libgearman gearmand libgearman-devel libdrizzle
- 58. 58 Gearmand enabled using MySQL(Mariadb) Postgresql SQL Lite Memcached
- 60. 60 Client Library pecl install channel://pecl.php.net/gearman-1.1.2 echo 'extension=gearman.so' > /path/gearman.ini
- 61. 61 Quick Test php -i | grep 'gearman'
- 62. 62 Gearman in PHP Client
- 63. 63 PHP Client # Create a client object $gm = new GearmanClient(); # Add default server, default is localhost:4730 $gm->addServer(); # Do Job $result = $gm->doNormal("reverse", "Hello!"); echo "Success: $resultn";
- 64. 64 Do background Job $job = $gm->doBackground("reverse", "apple");
- 65. 65 Gearman in PHP Worker
- 66. 66 PHP Worker # Define worker job function function reverse_fn($job) { //.... } # fetch message from client function reverse_fn($job) { $data = $job->workload(); }
- 67. 67 PHP Worker # Create our worker object. $gm= new GearmanWorker(); # Add default server (localhost). $gm->addServer(); # Register function "reverse" $gm->addFunction("reverse", "reverse_fn"); # Waiting for job... while($gm->work());
- 68. 68 Multiple Jobs $gm->addFunction("reverse", "reverse_fn"); $gm->addFunction("resize", "resize_fn"); $gm->addFunction("queue", "queue_fn"); …...
- 69. 69 Anonymous function $gm->addFunction("reverse", function($job){ $data = $job->workload(); });
- 72. 72 Installation via Spark php tools/spark install -v1.0.1 codeigniter-gearman
- 73. 73 Setup Gearmand Server and Port config/gearman.php $config['gearman_server'] = array('localhost'); $config['gearman_port'] = array('4730');
- 75. 75 Implement Image Resize Job create app controller
- 76. 76 class App extends CI_Controller { public function resize_sync() public function resize_async() }
- 77. 77 public function resize_sync() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); do { $this->lib_gearman->do_job_foreground('resize', $data); switch($client->returnCode()) { //.... } } while ($client->returnCode() != GREARMAN_SUCCESS); }
- 78. 78 class App extends CI_Controller { public function resize_sync() public function resize_async() }
- 79. 79 public function resize_async() { $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); $this->lib_gearman->do_job_background('resize', $data)); }
- 80. 80 CodeIgniter Worker create cli controller
- 81. 81 class Cli extends CI_Controller { public static function image_resize() public static function email_queue() public function worker() }
- 82. 82 public static function image_resize($job) { $data = json_decode($job->workload(), true); $ci =& get_instance(); $ci->image_lib->initialize($config); $ci->image_lib->resize(); $ci->image_lib->clear(); }
- 83. 83 Define Worker
- 84. 84 public function worker() { $worker = $this->lib_gearman->gearman_worker(); $this->lib_gearman ->add_worker_function('resize', 'Cli::image_resize'); $this->lib_gearman ->add_worker_function('queue', 'Cli::email_queue'); while ($this->lib_gearman->work()); }
- 85. 85 Start Worker $ php index.php cli worker
- 86. 86 Running workers via script #!/bin/sh script="index.php cli worker" folder="/script_path/" max_worker="5" php_bin="/usr/bin/php" count=0 current_count=$(ps aux | grep "index.php cli worker" | grep -v "grep" | wc -l) if test $current_count -lt $max_worker then cd $folder $php_bin $folder$script & else echo "The worker count has been limited." fi https://gist.github.com/appleboy/6806323
- 87. 87 About Job Data Don't pass 2GB object. Gearman is not data center.
- 88. 88 Keep your message data small Amazone SQS message siez: 256K http://goo.gl/0MBq8W
- 89. 89 Scalability
- 90. 90 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client
- 91. 91 you can scale out your clients and workers as needed
- 92. 92 Client Setting Client1, Client2: Gearman1 Client3, Client4: Gearman2 …...
- 94. 94 Add new Job Server...
- 95. 95 Reconfigure all setting (client and worker program)
- 96. 96 Reconfigure all setting (client and worker program) even Deploy Tool
- 97. 97 如何控制易怒的情緒?
- 98. 98 How to improve your architecture?
- 99. 99 Client Client Job Server Client ClientWorker Worker Worker Worker Job Server Client Proxy (Haproxy ...)
- 101. 101 Any Questions?
- 102. 102 Thanks 全體工作人員和聽眾