Oozie Development Guide

Oozie defines workflows, which are a fundamental organization of multiple Hadoop Jobs in a certain order and a whole that runs along a predetermined path. By starting a workflow, multiple Hadoop Jobs in the workflow will be executed until they are completed. This is the lifecycle of a workflow.

Oozie introduced the concept of a Coordinator, which can run each Job of the workflow as an Action, which is equivalent to an execution node in the workflow. This way, multiple workflow Jobs can be organized into a Coordinator Job, and the triggering time and frequency can be set, as well as the dataset, concurrency, etc. A Coordinator Job contains the semantics of setting the execution cycle and frequency outside the Job. It is similar to having a coordinator outside the workflow to manage the running of these workflow Jobs.

If Hue is installed on the cluster, the workflow can also be configured through page operation, for the specific operation steps, click here to view. The following introduces the method of configuring the workflow through the background:

1. Run the Coordinator Job

Let’s first look at a simple example included in the official distribution package oozie/examples/src/main/apps/cron. It can implement timed scheduling of a workflow Job. A blank workflow Job is given in this example, also to demonstrate that the Coordinator system can be used for scheduling.

There are 3 configuration files in this example. After modification, they are as follows:

• job.properties configuration

nameNode=hdfs://uhadoop-XXXXXX-master1:8020
jobTracker=uhadoop-XXXXXX-master1:23140
queueName=default
examplesRoot=examples
 
oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/cron
start=2016-12-01T19:00Z
end=2016-12-31T01:00Z
workflowAppUri=${nameNode}/user/${user.name}/${examplesRoot}/apps/cron

The configuration of the Hadoop cluster was modified, as well as the scheduling start and end time range.

• wordflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="one-op-wf">
    <start to="action1"/>
    <action name="action1">
        <fs/>
    <ok to="end"/>
    <error to="end"/>
    </action>
    <end name="end"/>
</workflow-app>

This is an empty Job, no modifications were made.

• coordinator.xml configuration

<coordinator-app name="cron-coord" frequency="${coord:minutes(2)}" start="${start}" end="${end}" timezone="UTC" xmlns="uri:oozie:coordinator:0.2">
     <action>
          <workflow>
               <app-path>${workflowAppUri}</app-path>
               <configuration>
                    <property>
                         <name>jobTracker</name>
                         <value>${jobTracker}</value>
                    </property>
                    <property>
                         <name>nameNode</name>
                         <value>${nameNode}</value>
                    </property>
                    <property>
                         <name>queueName</name>
                         <value>${queueName}</value>
                    </property>
               </configuration>
          </workflow>
     </action>
</coordinator-app>

The scheduled frequency was changed to 2 minutes. Then, upload these 3 files to HDFS.

Starting a Coordinator Job is similar to starting an Oozie workflow Job. Simply execute the following command:

bin/oozie job -oozie http://uhadoop-XXXXXX-master2:11000/oozie -config /home/hadoop/oozie/examples/src/main/apps/cron/job.properties -run

Running the above command will return the ID of this Job on the console. We can also view it through the Oozie Web console.

2. Coordinator Action

A Coordinator Job will create and execute Coordinator Actions. Usually, a Coordinator Action is a workflow Job, and this workflow Job will generate a dataset instance and process this dataset. Once a Coordinator Action is created, it will wait for the completion of all input events that meet the execution conditions and then execute, or it will time out.

Each Coordinator Job has a driving event, which determines the initialization of the Coordinator Actions it contains. For synchronous Coordinator Jobs, the trigger execution frequency is a driving event. Similarly, the basic unit that makes up a Coordinator Job is a Coordinator Action. Unlike the Oozie workflow Job, which only has OK and Error execution results, a Coordinator Action status set, as shown below:

WAITING

READY

SUBMITTED

TIMEDOUT

RUNNING

KILLED

SUCCEEDED

FAILED

3. Coordinator Application

When certain conditions are met, the Coordinator Application will trigger the Oozie workflow. Among them, the triggering conditions can be a time frequency, whether a dataset instance is available, or possibly other external events. A Coordinator Job is a running instance of a Coordinator application, which runs on the Coordinator engine provided by Oozie, and this instance starts at a specified time and runs until it ends. A Coordinator Job has the following statuses:

PREP

RUNNING

RUNNINGWITHERROR

PREPSUSPENDED

SUSPENDED

SUSPENDEDWITHERROR

PREPPAUSED

PAUSED

PAUSEDWITHERROR

SUCCEEDED

DONEWITHERROR

KILLED

FAILED

The status of the Coordinator Job is much more complex than that of a basic Oozie workflow Job. Because the basic execution unit of the Coordinator Job may be a basic Oozie Job, and some scheduling information is added, it is necessary to increase additional states to describe it.

4. Definition of Coordinator Application

The syntax format of a synchronous Coordinator Appliction definition is as follows:

<coordinator-app name="[NAME]" frequency="[FREQUENCY]" start="[DATETIME]" end="[DATETIME]" timezone="[TIMEZONE]" xmlns="uri:oozie:coordinator:0.1">
     <controls>
          <timeout>[TIME_PERIOD]</timeout>
          <concurrency>[CONCURRENCY]</concurrency>
          <execution>[EXECUTION_STRATEGY]</execution>
     </controls>
     <datasets>
          <include>[SHARED_DATASETS]</include>
          ...
          <!-- Synchronous datasets -->
          <dataset name="[NAME]" frequency="[FREQUENCY]" initial-instance="[DATETIME]" timezone="[TIMEZONE]">
               <uri-template>[URI_TEMPLATE]</uri-template>
          </dataset>
          ...
     </datasets>
     <input-events>
          <data-in name="[NAME]" dataset="[DATASET]">
               <instance>[INSTANCE]</instance>
               ...
          </data-in>
          ...
          <data-in name="[NAME]" dataset="[DATASET]">
               <start-instance>[INSTANCE]</start-instance>
               <end-instance>[INSTANCE]</end-instance>
          </data-in>
          ...
     </input-events>
     <output-events>
          <data-out name="[NAME]" dataset="[DATASET]">
               <instance>[INSTANCE]</instance>
          </data-out>
          ...
     </output-events>
     <action>
          <workflow>
               <app-path>[WF-APPLICATION-PATH]</app-path>
               <configuration>
                    <property>
                         <name>[PROPERTY-NAME]</name>
                         <value>[PROPERTY-VALUE]</value>
                    </property>
                    ...
               </configuration>
          </workflow>
     </action>
</coordinator-app>

Based on the above definition syntax format, we respectively explain the meaning of the corresponding elements, as follows:

• control elements

• dataset elements

In the Coordinator Job, there is a concept of Dataset, which can provide computing data for actual computation, mainly referring to data directories or files on HDFS, and can configure the frequency of dataset generation (Frequency), URI template, time and other information. The syntax format of the dataset is as follows:

<dataset name="[NAME]" frequency="[FREQUENCY]" initial-instance="[DATETIME]" timezone="[TIMEZONE]">
     <uri-template>[URI TEMPLATE]</uri-template>
     <done-flag>[FILE NAME]</done-flag>
</dataset>

An example is as follows:

<dataset name="stats_hive_table" frequency="${coord:days(1)}" initial-instance="2016-12-25T00:00Z" timezone="America/Los_Angeles">
     <uri-template>
          hdfs://m1:9000/hive/warehouse/user_events/${YEAR}${MONTH}/${DAY}/data
     </uri-template>
     <done-flag>donefile.flag</done-flag>
</dataset>

The above example will generate a user event table every day, which can be queried and analyzed by Hive. The location of this dataset is specified here, and subsequent calculations will use this part of the data. The uri-template specifies a matching template, and paths that meet this template will be used as the basis for computation. There is another way to define a dataset collection, to combine multiple datasets into a group to define, the syntax format is as follows:

<datasets>
     <include>[SHARED_DATASETS]</include>
     ...
     <dataset name="[NAME]" frequency="[FREQUENCY]" initial-instance="[DATETIME]" timezone="[TIMEZONE]">
          <uri-template>[URI TEMPLATE]</uri-template>
     </dataset>
     ...
</datasets>

• input-events and output-events elements

The input event of a Coordinator Application specifies the input conditions that must be met to execute a Coordinator Action. In the current version of Oozie, only dataset instances are supported.

A Coordinator Action may generate one or more dataset instances. In the current version of Oozie, the output event only supports the output of dataset instances.