Adding Pipeline Steps in HQ

Owned by Cooper Labenske
Last updated: Apr 05, 2022
2 min read

There are four places you can add Pipeline steps, shown in the following diagram:

 

  • Post-Scan - steps here are executed immediately after Scanning is complete

  • Pre-Extraction - steps here are executed just prior to Extraction

  • Post-Extraction - steps here are executed immediately after Extraction is complete

  • Pre-Index -steps here are executed just prior to the information being written to the Index

See Creating Pipelines for details on how to create and save Pipelines.

Pipeline Steps

You can add one or more of the following steps to different points in the Pipeline sequence:

  • S3 Thumbnail Upload
    Uploads thumbnails to Amazon S3Calculate MD5 ChecksumCalculates an MD5 checksum of the source file content

  • Index Debug Information
    Indexes raw debug information about the indexing job

  • Index Children
    Indexes child entries

  • Index Links
    Indexes information about linked entities

  • Copy Field
    This transformer copies the contents of an existing index field (source) into another field (target). The source field is not modified.

  • Rename Field
    Renames one field (Source Field) to another (Destination Field). The source field is then removed from the index. 

  • Remove Field
    Removes a field from the Index

  • Transform Field Value
    Transforms a field value to a different value

  • Convert from 2.x to 1.x Index
    Converts documents from Voyager 2.x to 1.x index schema

  • Remove Temp Files
    Removes temporary files downloaded or generated during indexing

  • Check Blob Store Files
    Checks for meta files in Microsoft Azure blob storage

  • Set Geo Field
    Sets the geo field for the entry from spatial information

  • Voyager Server Thumbnail Upload
    Uploads thumbnails generated by the agent to a Voyager Server instance

  • Propagate Fields
    Propagates fields to children

  • Add Format Tags
    Adds format tags based on a record's mime type

  • Set Value for Field
    Sets a field to a specific value. Useful for grouping.

  • Append Value to Field
    Appends a value to a field If the field is empty, behaves like Set Value for Field

  • Extract Entities with NLP
    Uses Natural Language Processing to extract categorized entities from the text content

  • Create Thumbnail with Base Map
    Create Thumbnail using the current Base Map

  • GeoTag Standard
    Geotags the document content with the standard Gazetteer

  • Geotag Custom
    Geotags the document content with a custom Gazetteer

  • Set Extent from PRJ File
    Transforms spatial extent based on projection information found in a component prj file

  • Add meta XML tags
    Adds meta XML tags