Call us: +1-415-738-4000

Quartz Scheduler Where (Locality API)

Introduction

Terracotta Quartz Scheduler Where is an Enterprise feature that allows jobs and triggers to run on specified Terracotta clients instead of randomly chosen ones. Quartz Scheduler Where provides a locality API that has a more readable fluent interface for creating and scheduling jobs and triggers. This locality API, together with configuration, can be used to route jobs to nodes based on defined criteria:

  • Specific resources constraints such as free memory.
  • Specific system characteristics such as type of operating system.
  • A member of a specified group of nodes.

This document shows you how to configure and use the locality API. You should already be familiar with using Quartz Scheduler (see the installation guide and the Quartz Scheduler documentation).

Configuring Quartz Scheduler Where

To configure Quartz Scheduler Where, follow these steps:

  1. Edit quartz.properties to cluster with Terracotta. See Clustering Quartz Scheduler for more information.
  2. If you intend to use node groups, configure an implementation of org.quartz.spi.InstanceIdGenerator to generate instance IDs to be used in the locality configuration. See Understanding Generated Node IDs for more information about generating instance IDs.
  3. Configure the node and trigger groups in quartzLocality.properties. For example:

    # Set up node groups that can be referenced from application code.
    # The values shown are instance IDs:
    org.quartz.locality.nodeGroup.slowJobs = node0, node3
    org.quartz.locality.nodeGroup.fastJobs = node1, node2
    org.quartz.locality.nodeGroup.allNodes = node0, node1, node2, node3
    
    # Set up trigger groups whose triggers fire only on nodes
    # in the specified node groups. For example, a trigger in the
    # trigger group slowTriggers will fire only on node0 and node3:
    org.quartz.locality.nodeGroup.slowJobs.triggerGroups = slowTriggers
    org.quartz.locality.nodeGroup.fastJobs.triggerGroups = fastTriggers
    
  4. Ensure that quartzLocality.properties is on the classpath, the same as quartz.properties.

    See Quartz Scheduler Where Code Sample for an example of how to use Quartz Scheduler Where.

Understanding Generated Node IDs

Terracotta clients each run an instance of a clustered Quartz Scheduler scheduler. Every instance of this clustered scheduler must use the same scheduler name, specified in quartz.properties. For example:

# Name the clustered scheduler.
org.quartz.scheduler.instanceName = myScheduler

myScheduler's data is shared across the cluster by each of its instances. However, every instance of myScheduler must also be identified uniquely, and this unique ID is specified in quartz.properties by the property org.quartz.scheduler.instanceId. This property should have one of the following values:

  • <string> – A string value that identifies the scheduler instance running on the Terracotta client that loaded the containing quartz.properties. Each scheduler instance must have a unique ID value.
  • AUTO – Delegates the generation of unique instance IDs to the class specified by the property org.quartz.scheduler.instanceIdGenerator.class.

For example, you can set org.quartz.scheduler.instanceId to equal "node1" on one node, "node2" on another node, and so on.

If you set org.quartz.scheduler.instanceId equal to "AUTO", then you should specify a generator class in quartz.properties using the property org.quartz.scheduler.instanceIdGenerator.class. This property can have one of the values listed in the following table.

Value Notes
org.quartz.simpl
.HostnameInstanceIdGenerator
Returns the hostname as the instance ID
org.quartz.simpl
.SystemPropertyInstanceIdGenerator
Returns the value of the org.quartz.scheduler.instanceId system property. Available with Quartz 2.0 or higher.
org.quartz.simpl
.SimpleInstanceIdGenerator
Returns an instance ID composed of the local hostname with the current timestamp appended. Ensures a unique name. If you do not specify a generator class, this generator class is used by default. However, this class is not suitable for use with Quartz Scheduler Where because the IDs it generates are not predictable.
Custom Specify your own implementation of the interface org.quartz.spi.InstanceIdGenerator.

Using SystemPropertyInstanceIdGenerator

org.quartz.simpl.SystemPropertyInstanceIdGenerator is useful in environments that use initialization scripts or configuration files. For example, you could add the instanceId property to an application server's startup script in the form -Dorg.quartz.scheduler.instanceId=node1, where "node1" is the instance ID assigned to the local Quartz Scheduler scheduler. Or it could also be added to a configuration resource such as an XML file that is used to set up your environment.

The instanceId property values configured for each scheduler instance can be used in quartzLocality.properties node groups. For example, if you configured instance IDs node1, node2, and node3, you can use these IDs in node groups:

org.quartz.locality.nodeGroup.group1 = node1, node2
org.quartz.locality.nodeGroup.allNodes = node1, node2, node3

Available Constraints

Quartz Scheduler Where offers the following constraints:

  • CPU – Provides methods for constraints based on minimum number of cores, available threads, and maximum amount of CPU load.
  • Resident keys – Use a node with a specified Enterprise Ehcache distributed cache that has the best match for the specified keys.
  • Memory – Minimum amount of memory available.
  • Node group – A node in the specified node group, as defined in quartzLocality.properties.
  • OS – A node running the specified operating system.

See the code samples provided below for how to use these constraints.

Quartz Scheduler Where Code Sample

A cluster has Terracotta clients running Quartz Scheduler running on the following hosts: node0, node1, node2, node3. These hostnames are used as the instance IDs for the Quartz Scheduler scheduler instances because the following quartz.properties properties are set as shown:

org.quartz.scheduler.instanceId = AUTO

#This sets the hostnames as instance IDs:
org.quartz.scheduler.instanceIdGenerator.class = 
    org.quartz.simpl.HostnameInstanceIdGenerator

quartzLocality.properties has the following configuration:

org.quartz.locality.nodeGroup.slowJobs = node0, node3
org.quartz.locality.nodeGroup.fastJobs = node1, node2
org.quartz.locality.nodeGroup.allNodes = node0, node1, node2, node3

org.quartz.locality.nodeGroup.slowJobs.triggerGroups = slowTriggers
org.quartz.locality.nodeGroup.fastJobs.triggerGroups = fastTriggers

The following code snippet uses Quartz Scheduler Where to create locality-aware jobs and triggers.

// Note the static imports of builder classes that define a Domain Specific Language (DSL).
import static org.quartz.JobBuilder.newJob;
import static org.quartz.TriggerBuilder.newTrigger;
import static org.quartz.locality.LocalityTriggerBuilder.localTrigger;
import static org.quartz.locality.NodeSpecBuilder.node;
import static org.quartz.locality.constraint.NodeGroupConstraint.partOfNodeGroup;

import org.quartz.JobDetail;
import org.quartz.locality.LocalityTrigger;
// Other required imports...

// Using the Quartz Scheduler fluent interface, or the DSL.

/***** Node Group + OS Constraint
Create a locality-aware job that can be run on any node 
from nodeGroup "group1" that runs a Linux OS:
*****/

LocalityJobDetail jobDetail1 =
        localJob(
            newJob(myJob1.class)
                .withIdentity("myJob1")
                .storeDurably(true)
                .build())
            .where(
                node()
                    .is(partOfNodeGroup("group1"))
                    .is(OsConstraint.LINUX))
            .build();

// Create a trigger for myJob1:
Trigger trigger1 = newTrigger()
            .forJob("myJob1")
            .withIdentity("myTrigger1")
            .withSchedule(simpleSchedule()
                   .withIntervalInSeconds(10)
                   .withRepeatCount(2))
            .build();

// Create a second job:
JobDetail jobDetail2 = newJob(myJob2.class)
                .withIdentity("myJob2")
                .storeDurably(true)
                .build();

/***** Memory Constraint
Create a locality-aware trigger for myJob2 that will fire on any
node that has a certain amount of free memory available:
*****/
LocalityTrigger trigger2 =
        localTrigger(newTrigger()
            .forJob("myJob2")
            .withIdentity("myTrigger2"))
            .where(
                node()
                    // fire on any node in allNodes
        //  with at least 100MB in free memory.
                    .is(partOfNodeGroup("allNodes"))  
                    .has(atLeastAvailable(100, MemoryConstraint.Unit.MB)))  
            .build();

/***** A Locality-Aware Trigger For an Existing Job
 The following trigger will fire myJob1 on any node in the allNodes group 
 that's running Linux:
*****/

LocalityTrigger trigger3 =
        localTrigger(newTrigger()
            .forJob("myJob1")
            .withIdentity("myTrigger3"))
            .where(
                node()
                    .is(partOfNodeGroup("allNodes")))
            .build();


/***** Locality Constraint Based on Cache Keys

The following job detail sets up a job (cacheJob) that will be fired on the node 
where myCache has, locally, the most keys specified in the collection myKeys.

After the best match is found, missing elements will be faulted in. 
If these types of jobs are fired frequently and a large amount of data must often be
faulted in, performance could degrade. To maintain performance, ensure that most of 
the targeted data is already cached.

*****/

// myCache is already configured, populated, and distributed.
Cache myCache = cacheManager.getEhcache("myCache"); 

// A Collection is needed to hold the keys for the elements to be targeted by cacheJob.
// The following assumes String keys.

Set<String> myKeys = new HashSet<String>();

... // Populate myKeys with the keys for the target elements in myCache.

// Create the job that will do work on the target elements.

LocalityJobDetail cacheJobDetail =
        localJob(
            newJob(cacheJob.class)
                .withIdentity("cacheJob")
                .storeDurably(true)
                .build())
                .where(
                    node()
                         .has(elements(myCache, myKeys)))
                .build();

Notice that trigger3, the third trigger defined, overrode the partOfNodeGroup constraint of myJob1. Where triggers and jobs have conflicting constraints, the triggers take priority. However, since trigger3 did not provide an OS constraint, it did not override the OS constraint in myJob1. If any of the constraints in effect — trigger or job — are not met, the trigger will go into an error state and the job will not be fired.

CPU-Based Constraints

The CPU constraint allows you to run jobs on machines with adequate processing power:

...

import static org.quartz.locality.constraint.CpuConstraint.loadAtMost;

...

// Create a locality-aware trigger for someJob.
LocalityTrigger trigger =
        localTrigger(newTrigger()
            .forJob("someJob")
            .withIdentity("someTrigger"))
            .where(
                node()
                    // fire on any node in allNodes
                    // with at most the specified load:
                    .is(partOfNodeGroup("allNodes"))  
                    .has(loadAtMost(.80)))  
            .build();

The load constraint refers to the CPU load (a standard *NIX load measurement) averaged over the last minute. A load average below 1.00 indicates that the CPU is likely to execute the job immediately. The smaller the load, the freer the CPU, though setting a threshold that is too low could make it difficult for a match to be found.

Other CPU constraints include CpuContraint.coresAtLeast(int amount), which specifies a node with a minimum number of CPU cores, and CpuConstraint.threadsAvailableAtLeast(int amount), which specifies a node with a minimum number of available threads.

NOTE: Unmet Constraints Cause Errors
If a trigger cannot fire because it has constraints that cannot be met by any node, that trigger will go into an error state. Applications using Quartz Scheduler Where with constraints should be tested under conditions that simulate those constraints in the cluster.

This example showed how memory and node-group constraints are used to route locality-aware triggers and jobs. trigger2, for example, is set to fire myJob2 on a node in a specific group ("allNodes") with a specified minimum amount of free memory. A constraint based on operating system (Linux, Microsoft Windows, Apple OSX, and Oracle Solaris) is also available.

Failure Scenarios

If a trigger cannot fire on the specified node or targeted node group, the associated job will not execute. Once the misfireThreshold timeout value is reached, the trigger misfires and any misfire instructions are executed.

Locality With the Standard Quartz Scheduler API

It is also possible to add locality to jobs and triggers created with the standard Quartz Scheduler API by assigning the triggers to a trigger group specified in quartzLocality.properties.