Leveraging the Drupal Queue to Create Awesome

How we've done third-party integrations before Working with the YMCAs we've used PHP scripts running off a Cron job that would send an API call and retrieve classes from third-parties such as Daxko and CCC. Then we would import these classes from our call into actual nodes on the site. While this worked adequately to get classes imported, there is always room for more awesome.

// See if we can find an existing class that matches the Course ID
$existingClass = load_existing_class( $program );

if ( $existingClass && $existingClass->nid ) {
    print '[UPDATING]' . PHP_EOL;
    $nodeTmp = $existingClass;
}
else {
    if($pgm_closed) {
        print '[SKIPPING] - Closed Program' . PHP_EOL;
        logError($logfp, 'Skipped, Program is Closed' ,$program);
        continue;
    }
    print '[INSERTING]' . PHP_EOL;
    $nodeTmp = new stdClass();
    $nodeTmp->type = 'ccc_program';
    node_object_prepare( $nodeTmp );
}

$nodeTmp->uid = $uidAuthor;
$nodeTmp->language = LANGUAGE_NONE;
$nodeTmp->title = (string)$program->DESC_1;
$nodeTmp->datereminder_enabled = 3;
if(!empty($program->EXT_DESC)) {
    $nodeTmp->body[$nodeTmp->language][0]['value'] = (string)$program->EXT_DESC;
    $nodeTmp->body[$nodeTmp->language][0]['format'] = 'filtered_html';
}

The problem with these PHP scripts is they are prone to "choke" and die a terrible, painful, unproductive death where nothing gets imported. With an upcoming project for Van Keppel we decided it was time to start using Drupal's built in Queue System to run these functions, which will help prevent the "chokes" and also to preserve performance on the front end.

Drupal's Queue system

Drupal's Queue system is pretty awesome. You can create a "queue" object that will populate with data that will be stored and can be accessed or processed by other functions. We decided to use the Job Scheduler module to make our API call, and then use Drupal's built in Cron to actually process the information retrieved by the API call.

The idea is that the API call, which could potentially pull in information on 10,000+ products, should be separate from the processing function. This way if the API call is successful, then we can slowly iterate over the response and build nodes. Which is better than witnessing a single corrupt character (or bad information retrieved) causing the entire script to crash harder than an ACME anvil dropped out of a plane at 35,000 feet.

Setting up our .install file

We start by having our module's install file set up our job through the Job Scheduler module:

/**
* Implements hook_install().
*/
function example_module_install() {
    // Create a default scheduled job.
    $job = array(
        'type' => 'example_job',
        'crontab' => '* */12 * * *', // Every day, every 12 hours for example
        'periodic' => TRUE,
    );
    JobScheduler::get('example_job_schedule')->set($job);

    variable_set('queue_class_example_job_schedule', 'EquipmentPullQueue');
}
function example_module_uninstall() {
    variable_del('queue_class_example_job_schedule');
}  

Where the type is a string identifier we give our specific job and below that where we call JobScheduler::get('example_job_queue')->set($job); we are calling the queue that we will be creating in the .module file and setting the job to use that queue. The variable set we are doing is to identify our queue function we will be calling that is then followed by the name of our Queue that we are extending from Drupal's Memory Queue.

Always remember if you are setting variables to always have an uninstall hook set to remove your variables! (Otherwise a kitten dies when you try to disable then re-enable your module).

Setting up our module

Creating the Cron Job Scheduler Hook

Here we are going to begin by creating our job scheduler hook that will request our queue when the job is triggered:

function example_module_cron_job_scheduler_info() {
    $info = array();
    $info['example_job_schedule'] = array(
        'worker callback' => 'example_job_schedule_queue',
    );
    return $info;
}  

The key in the info array we are setting up is the same name as the queue we are calling in our install file. The worker callback is the function that will be called when this job triggers. So example_job_schedule_queue is our function that gets our response and puts all the information in the queue.

Filling up our queue with data

In this function we do a couple things:

We create our queue by calling $queue = DrupalQueue::get('example_job_schedule'); which creates a fresh object out of our queue we created. Make our API call, encoded as JSON, and send any parameters we need to. JSON decode our response and foreach over each item in the array

foreach ($decoded_response->data as $item) {
    $queue->createItem($item);
}

Last step; have our cron process the items!

Our last step is creating a cron queue that will process the items in the queue that our job just populated:

/**
* Implements hook_cron_queue_info().
*/
function example_module_cron_queue_info() {
    $queues['request_all_equipment'] = array(
        'worker callback' => 'example_cron_run_queue', // This is the callback function for each queue item.
        'time' => 180, // This is the max run time per cron run in seconds.
    );
    return $queues;
}  

This sets up a cron queue. This function will fire whenever the Drupal cron is triggered. The key here is that we are calling on the same queue that we created with our job scheduler function. Except this time the worker callback is now calling a different function. This function will get each queue item passed to it, one at a time. We are also setting time to prevent Cron from spending more than 180 seconds (3 minutes) on an individual item.

Our individual queue function is where we are processing individual items and creating entites and nodes on the site, which is pretty awesome! And when an item has been successfully passed through this function it empties out of the queue. Bonus!

To the future!

This method of integrating with third-parties will be far more effective than running PHP scripts. By using the Drupal Queue we are preventing scripts from "choking" and also relieving pressure off the front-end by processing information from our API calls in a queue that simply runs off Cron. Moving forward, using the Drupal Queue is a proactive way of processing information and using that to programmatically create dynamic content on your Drupal site! Hope this information is helpful!