Recently, for a client I had to setup an automation server. He had a CRM and he needed to run automations created by his customers.
So that meant, alot of emails, SMS and time delays and pipeline changes. His customers database was huge so plain old style cronjobs won’t cut any more for his workload. Thats when I started searching a solution and found queues.
His backend was in laravel so I just setup a EC2 instance with a redis queue. Main api server use to hit redis whenever a trigger happens. For his scenario triggers were simple like pipeline stage change or lead creation. So when a trigger happens we simply added a entry in the queue. After that we setup another server which was the main automation server. That server use to pick jobs from the queue and execute them. Simple.
Good thing about the queue is, I am able to control the time delays as well. So lets say user wants to send an email instantly, then he wants to wait 2 hours before the next email goes. I don’t need to keep track of the time and manually trigger the email. Queue does it for me. I just requeue the job to run after 2 hours with the required parameters. So my main job is further divided into smaller jobs.
MyAutomationJob::dispatch($params)->delay(now()->addHours(2));
I even setup Laravel Horizon on the automation to keep track of jobs. If you don’t know it yet check it out. Its pretty cool, you can check the status of jobs, failed jobs and even re-run the failed jobs.
This is just a bird-eye view, of how you can also setup up a robust automation server with setting up a queue.
That’s a wrap. You can hit me up on my email if you have any question about the setup.