How to use Horizon and Redis together for distributed systems in Laravel

Ever wonder how prominent web applications like Facebook etc keep getting better and better, bigger and bigger with thousands of code lines? Well, it’s all thanks to different mechanisms and technologies that keep them operating at large scale. One of these concepts is a distributed system that helps to manage background jobs. And the cool thing is, we’ve got these awesome distributed systems that can handle a ton of tasks at once, especially in modern, scalable apps.

Laravel’s ecosystem has some amazing tools like Horizon and Redis to set up and manage the distributed job processing systems.

So, let’s dive in and see how we can build and run these distributed systems using Laravel 11, Horizon, and Redis.

1- Discover the world of distributed systems

A distributed system is like a team of robots working together to get a job done (nodes). It’s not like a one-size-fits-all monolithic system. With a distributed system, you can add more nodes to handle more work, and it’ll still work as they should. Even if some nodes fail, the system can still keep working.

In Laravel, we use queues and workers to make things work, so tasks can be processed one after the other without getting in each other’s way. Redis, a super fast in-memory data structure store, and Horizon, a monitoring tool for Laravel queues, are key components in this architecture.

2- Redis

Redis plays a pivotal role in Laravel’s queue management along with other supported services, serving as the underlying infrastructure that enables efficient and reliable operation of queues within the Laravel framework.

Redis serves as a queue backend in Laravel, adding tasks (jobs) on queues that workers process asynchronously in different ways.

Enable and Install Predis in Laravel:

Predis is a powerful Redis client written entirely in PHP that doesn’t need any additional extensions:

Run the below code to install predis in your Laravel application:

or with docker:

Next, update your .env file as following:

Configuration

You can easily configure your application’s Redis settings through the config/database.php configuration file. In this file, you’ll find a redis array with the Redis servers being used by your application.

We’re going to use the Predis package client, so let’s make sure the REDIS_CLIENT environment variable’s value is set to predis in the database.php config file:

3- Laravel Horizon: Helping You Keep an Eye on Your Queues and Manage Them with Ease

PS. Laravel Horizon doesn’t currently come with official support for Windows. This is mostly because it relies on certain extensions, like ext-pcntl and ext-posix, which are not available on Windows. These extensions help with managing processes, signals, and executing jobs on an asynchronous basis and they’re required for Horizon to do its thing!

Don’t worry if Horizon can’t run natively on Windows, there’s still a way to use it! All you have to do is set up a Linux-based environment on your Windows machine, by using a virtual machine or in a more convenient way, using the lovely Docker.

Laravel Horizon is an awesome tool that helps you keep an eye on and manage your queues in Laravel applications. It’s specially made for Redis-powered queues, and it’s got a super user-friendly dashboard.

With Horizon, you can check jobs in real-time, manage your workers, And then there’s measuring and monitoring the rate at which jobs are processed in a system over time, handle failed jobs, and make your queued tasks perform even better. With Horizon, developers can define worker processes, scaling queues, and having deep insights into job execution, retries, and failures.

Installing and running Horizon monitor

Install Horizon via Composer:

Publish Horizon assets using the horizon:install Artisan command:

Run Horizon:

Access the dashboard at http://our-app/horizon

Why do we need Horizon?

With Laravel Horizon you can view in real-time, job statistics, pending jobs, and failed jobs on queue all in one simple dashboard. Additionally, you can define and scale workers across queues, and configure job priorities as needed to ensure critical tasks are processed first in segments.

Not just that, Laravel Horizon helps us retry or delete failed jobs directly on site from the dashboard.

This is how Horizon dashboard looks like:

4- Building a Distributed Job Processing System in Laravel

Let’s walk through the steps to build a distributed job management system using Laravel, Redis, and Horizon.

One of the cool things about this concept is that it uses queues and workers to spread tasks across processes. This makes it super handy for handling a ton of work at once, since you can run jobs asynchronously and hand them off to workers.

To make things even easier, I’ve put together a step-by-step guide with a simple example:

4.1- Create a Job class:

Queues allow you to defer resources-consuming tasks (e.g., sending emails, parsing data) to workers for background processing.

Use Laravel’s Artisan command to generate a job class:

Define the logic to be executed in the generated app/Jobs/ProcessData.php, and update the handle method:

Illuminate\Contracts\Queue\ShouldQueue interface is used to let Laravel know that you’d like the created job is to be put on the queue and run in the background.

The Queueable trait that our created job is using, helps to serialized and unserialize model instance and load its relationships if the model passed into the constructor when the job is processing.

4.2- Dispatch the Job

You can dispatch jobs from controllers, events, routes or anywhere in your application. For example:

4.3- Run the Worker

Workers will listen to the queue and process the jobs. To start a worker using Artisan, just run this artisan command:

Or in containerized laravel in docker:

In a nutshell

Distributed systems using Laravel, Horizon, and Redis are a great way to handle background jobs. They’re really scalable and efficient! Redis is perfect for high-performance queuing, and Horizon is awesome for real-time monitoring. With these tools, developers can build systems that can handle even the most complex workloads. And with the right configuration, optimization, and monitoring, Laravel’s tools make it easier to set up a strong distributed system.