Creating a New Processing Task

Voyager Search has implemented a new processing framework written entirely in Python. This guide describes how to create and add your own custom processing task using the Python framework. For the purpose of this guide, we will use the built-in Zip Files task as the example.

NOTE: Before you begin, ensure you have permissions to manage Voyager and have write access to the Voyager installation folders.

Creating a New Processing Task

Follow these steps to create a new processing task.

Step 1: Create the required task files

There are two files required to successfully create a processing task:

A JSON file defining the task UI and parameter information

Create a file named zip_files_sample.info.json file in the info folder located in the install folder for Voyager Search (e.g. C:\voyager\server_1.9.8\app\py\processing\info
The JSON file must include the task name and have the extension, ".info.json"

A Python script containing the source code for processing

Create a Python script named zip_files_sample.py in the tasks folder located in the install location for Voyager Search (e.g. C:\voyager\server_1.9.8\app\py\processing\tasks)
The Python script name must be identical to the task name(e.g. zip_files.py)

Step 2: Defining the Task and Parameter Information

Each processing task has one or more input parameters. For example, the built-in Zip Files task has two parameters:

The input search results represents the contents/items in the cart. It is not visible in the UI, but it is required for all tasks and must be named input_items.
A check box that controls how the files are zipped

The task information is defined using JSON notation and saved in the info.json file. Edit the new file and copy and paste the following information. You may also copy the text from the existing zip_files.info.json file.

{
"name": "zip_files_sample",
"runner": "python",
"categories": ["download"],
"params": [
{
"type": "VoyagerResults",
"name": "input_items",
"required": true
},
{
"type": "CheckBox",
"name": "flatten_results"
}
],
"display": {
"en": {
"display": "Zip Files Sample",
"description": "Compress input files into a single zip file",
"params":{
"flatten_results":
{
"display": "Flatten Results",
"description": "Place all items in the root folder"
}
} } } }

The task name must be identical to the script name, and the runner indicates the programming language used for the execution code. In this example, the categories is "download" since the output of this task is a zip file that can be saved. Currently, there is only one category - "download". If your task does not create a new output such as Delete Files, the category can be blank or omitted.

As mentioned above, every task requires a parameter for the input search results and additional parameters can be added. Parameters have properties such as name, required, value, etc. For a complete listing of parameter types and JSON examples, see the Parameter Types section at the bottom of this guide.

After saving this file, check if the task is visible in Voyager Search by going to Process… from the Tools menu. It may be necessary to click the refresh task list button in the upper right to first see the new task.

If the task named Zip Files Sample is not visible, check the syntax of the JSON entered in the info file. It is a common mistake to enter extra commas or forget commas when constructing JSON documents. Here is an online tool that can help validate JSON: json-validator

Step 3: Adding the Processing Code

The next step is to add the code necessary to do the processing. Let’s to this by opening the existing script named zip_files.py shipped with Voyager Search and copy the entire contents. Paste the contents into the new Python script named zip_files_sample.py and save.

For every task script, the necessary libraries must be imported and every script must have an execute function. The input to the execute function is a request in the form of a dictionary. This dictionary contains the task information and parameter information. The parameter information will only include the first 25 search results. Therefore, if the number of input results is greater than 25, it is required to query the index for results. Please take a close look at the source code for the Zip Files task to see how this is done. Also, it is highly recommended to read through the source code for other Voyager processing tasks.

The status module contains a Writer class with functions to report status as the task is executing.
The task_utils module contains helper functions for many different operations such as retrieving the parameter values.
The status and task_utils modules are located in the utils folder within the voyager_tasks folder. It’s strongly encouraged to read through these scripts and familiarize with the many different functions.

Any time a task is executed, a temporary working folder is required for storing results and log files. This folder is created the in the Voyager data/Process folder (i.e. C:\Voyager\data\process\2014-10-16\T0001MDJV2UH_zip_files)

The code at the bottom of the script is used to update the state of the task if necessary, and generates a report in the working folder of what has passed and failed.

Step 4: Testing the Task

At this point, the task is ready for testing. To test the task:

Open Voyager (no need to restart)
Search for a small number of files (i.e. text files, pdfs, etc.)
Click Process in the Tools menu
Choose the new task named Zip Files Sample
Enter the task details
Run the task

If the task succeeds, a zip should will be available for download.

If it fails, check the error messages provided and verify the syntax of your code.

Often, it will be necessary to debug your script to identify any issues; see Debugging the Task Script section below. Also, go to the job folder for a better understanding of what is created (i.e. C:\Voyager\data\process\2014-10-16\T0001MDJV2UH_zip_files)

Parameter Types

Following is a list of supported parameter types, examples on how to define them and an illustration of each type as it appears on the UI, if available.

VoyagerResults (Required by all tasks):

{  "type": "VoyagerResults",
  "name": "input_items",
  "required": true
}

String:

{
  "type": "String",
  "name": "input_user_name",
  "required": false,
  "value": "voyager"
}

Integer:

{
  "type": "Integer",
  "name": "compression_quality",
  "value": 75,
  "min": 1,
  "max": 100

StringChoice:

CatalogPath (A dataset such as a shapefile or Geodatabase table):

FolderLocation:

CheckBox (True or False):

Projection (list of projections):

Geometry (map can be initialized to the maximum extent of the results):

By setting the property, "extentParam", you can have a map view without the option to draw an irregular shape polygon:

MapView (a map view without drawing options):

Debugging the Task Script

A processing task is a single process launched by a Python script named VoyagerTaskRunner.py located in the tasks folder (i.e. C:\Voyager\tasks). From the command line, you can execute the job above as follows:

To debug a processing task you will require a .json file that represents a processing job. Here is an example job for zip files. Save this json in a new file (e.g. “TestZipFiles.json”).

If you already use a Python IDE, then it is recommended to use the debugging tools available for that IDE. The other option is to import and use the built-in module named pdb.

Using this module, you can add the following line of code any where you want to break the execution to start debugging:

Now, run the command above. The execution will stop where you set the trace. Type the letter s, and hit the <enter> key to begin execution. Continue to hit <enter>. This will execute each line of code individually. If you need to see a value of a variable, type the letter p and the name of the variable. Repeat these steps to continue debugging.

For more information about the pdb module, see this article.