Blog from September, 2023

Adding ESRI Deep Learning Models to Voyager
Cooper Labenske posted on Sept 18, 2023

Required Software:

  1. ArcGIS Pro (latest version) and the license must include Image Analyst Extension

  2. Deep Learning Library...https://github.com/esri/deep-learning-frameworks

  3. Voyager (Vose)

  4. Voyager Server must be configured to use the Python Environment from ArcGIS Pro. This JSON should exist in the python.dex file. This file should exist in the Voyager Server’s data\config folder.

    {"environments": [  
  {
    "name": "Python39",
    "path": "D:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\python.exe",
    "type": "ARCGIS",
    "version": "3.10",
    "disabled": false,
    "bits": "64bit",
    "scopes": [
      "WORKER",
      "PIPELINE",
      "TASK",
      "SERVICE",
      "TRANSFORMER"
    ]
  }
]}

Strongly Recommended:

Complete this course to learn some of the ArcGIS Image Analyst Geoprocessing tools required in the Voyager processing task: https://www.esri.com/training/catalog/6410c0524d750615175c0b58/classifying-objects-using-deep-learning-in-arcgis-pro/

Running Processing Tasks in Voyager

To run processing tasks, search items are added to the cart, then from the Cart dropdown, users can review the items, select a task to run on those items

Creating Processing Tasks in Voyager

A task is composed of two files placed in well-known folders in the Voyager server app\py directory:

  1. tree_detection.info.json

  2. tree_detection.py

    The info.json file contains the task name, description, parameters and the parameter descriptions. The python script contains the code to retrieve the items from the cart and perform the execution.

** The best approach to creating a new task is to make a copy of an existing task, rename as necessary and begin to make the necessary code changes.

Creating ESRI Deep Learning Tasks in Voyager

Currently, there are 4 fully functional Deep Learning tasks in Voyager:

damage_assessment, oil_spil_detection, solar_panel_detection, wind_turbine_detection

The damage assessment task performs a building classification based on the training course above. The other 3 tasks perform feature detection. These models can be downloaded from the Esri’s Living Atlas:
https://livingatlas.arcgis.com/en/browse/?q=dlpk%20detection#d=2&q=dlpk+detection

It’s encouraged to download and use the model guide in ArcGIS Pro to learn the tools and parameters. Those same tools and parameters are used in the processing task in Voyager. For example, this is the Esri GP tool for tree detection. When this is run in ArcGIS, the Python code can be copied and pasted into the task:

Tree Detection Geoprocessing Tool Python Snippet:

arcpy.ia.DetectObjectsUsingDeepLearning(imagery,
            output_features,
            r"D:\EsriTraining\ClassifyingObjectsDeepLearning\TreeDetection\TreeDetection.dlpk",
            "padding 100;threshold 0.1;nms_overlap 0.1;batch_size 4;exclude_pad_detections True;test_time_augmentation False",
            "NO_NMS", "Confidence", "Class", 0, "PROCESS_AS_MOSAICKED_IMAGE")

Videos showing how to create a task to do Tree Detection in Voyager:

  1. Deep Learning Models - Video 1.mp4 - This video demonstrates where to download the deep learning model, view the guide, setup a working folder, run the Detect Objects Using Deep Learning tool and copy the tool as a Python snippet.

  2. Deep Learning Models - Video 2.mp4 - This video demonstrates how to setup up a repository in HQ. This repository is required for the tree features that will be detected for an AOI when the task runs. This feature class will get indexed after the task completes.

  3. Deep Learning Models - Video 3.mp4 - This video demonstrates how to create a pipeline step required to be set on the repository. This pipeline step will render the items on a basemap in the index and set a name field.

  4. Deep Learning Models - Video 4.mp4 - This video demonstrates creating a saved search for the repository and how to get the Saved Search ID needed for the Python code.

  5. Deep Learning Models - Video 5.mp4 - This video demonstrates configuring the info.json file for Tree Detection. This tree_detection.info.json file must exist in the app\py\processing\info directory. The JSON can be copied from below and configured as necessary for other detection models. The name of the task must be exact as the name of the python task script. Every task has the Voyager Results parameter. The only other parameter for Tree Detection is a processing extent.

  6. Deep Learning Models - Video 6.mp4 - This video demonstrates the Python code required for tree detection. The Python snippet shown above is required, as well as, the code to update and insert new features to the trees feature class. Code exists to also index the repository after new features are detected and a Saved Search is opened. The code below can be used and edited as necessary for other detection models. It’s important to get and set the repository ID, pipeline ID and Saved Search ID so that indexing can happen when new features are detected. Every task has and Execute() function.

  7. Deep Learning Models - Video 7.mp4 - This video demonstrates refreshing the list of tasks and run the new task. For this task, users do not need to add any items to the cart. They can simply go to the cart and select this task. This video ends by showing the task completing and the saved search opening with items.

Tree Detection Task Code:

{
    "name": "tree_detection",
    "runner": "python",
    "categories": ["ESRI - Deep Learning Machine Models (Samples Only. Not for Official Use)"],
    "params": [
      {
        "type": "VoyagerResults",
        "name": "input_items",
        "required": true
      },
      {
        "type": "Geometry",
        "name": "processing_extent",
        "extentParam": true,
        "initWithResultsExtent": true
      }
    ],
    "display":
    {
      "en":
      {
        "display": "Tree Detection - Sample Only. Not for Official Use",
        "description": "Deep learning model to detect trees from high resolution imagery. Sample only.",
        "helpURL": "https://www.arcgis.com/home/item.html?id=4af356858b1044908d9204f8b79ced99",
        "params":
        {
          "processing_extent":
          {
              "display": "Processing Extent",
              "description": "The geographic extent of the area to be processed (in the WGS84 coordinate system). Use a small AOI"
          }
        }
      }
    }
  }
  
# -*- coding: utf-8 -*-
# (C) Copyright 2014 Voyager Search
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import matplotlib
import time
import webbrowser
import requests

matplotlib.use('TkAgg', force=True)

from utils import status
from utils import task_utils

# Get SSL trust setting.
verify_ssl = task_utils.get_ssl_mode()

status_writer = status.Writer()
import arcpy


def execute(request):
    """Runs Deep Learning model for tree detection.
    :param request: json as a dict.
    """
    parameters = request['params']
    geocode_url = "https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/reverseGeocode?token=AAPK0ff1c762638447808364d9569cb73dc7H0nOcHAr44aooTybfWW_oSHphAGTlMwd6MsCpMiVwoIxZI7rDUYFsbfL5VoqSQ5H&{0}&f=json"

    # Get the extent for which to use to calculate statistics.
    extent = ''
    try:
        try:
            ext = task_utils.get_parameter_value(parameters, 'processing_extent', 'wkt')
            if ext:
                sr = task_utils.get_spatial_reference("4326")
                extent = task_utils.from_wkt(ext, sr).extent
        except KeyError:
            ext = task_utils.get_parameter_value(parameters, 'processing_extent', 'feature')
            if ext:
                extent = arcpy.Describe(ext).extent
    except KeyError:
        pass

    # Create the temporary workspace.
    task_folder = os.path.join(request['folder'], 'temp')
    if not os.path.exists(task_folder):
        os.makedirs(task_folder)
        arcpy.CreateFileGDB_management(task_folder, "temp.gdb")

    # RUN THE MODEL AND INDEX THE ITEMS.
    description = 'Information can be found here: https://www.arcgis.com/home/item.html?id=4af356858b1044908d9204f8b79ced99'
    imagery_url = 'https://naip.imagery1.arcgis.com/arcgis/rest/services/NAIP/ImageServer'
    arcpy.env.extent = extent
    arcpy.env.cellSize = 0.3
    output_features = os.path.join(task_folder, "temp.gdb/Trees") # Temporary Feature clas
    status_writer.send_status("Detecting Trees for AOI...may take several minutes...")
    imagery = "https://naip.imagery1.arcgis.com/arcgis/rest/services/NAIP/ImageServer"
    arcpy.ia.DetectObjectsUsingDeepLearning(imagery,
            output_features,
            r"D:\EsriTraining\ClassifyingObjectsDeepLearning\TreeDetection\TreeDetection.dlpk",
            "padding 100;threshold 0.1;nms_overlap 0.1;batch_size 4;exclude_pad_detections True;test_time_augmentation False",
            "NO_NMS", "Confidence", "Class", 0, "PROCESS_AS_MOSAICKED_IMAGE")

    try:
        # Update the Trees feature class
        status_writer.send_status("Updating tree features...")
        tree_features = r"D:\EsriTraining\ClassifyingObjectsDeepLearning\TreeDetection\TreeDetection.gdb\Trees"
        fields = arcpy.ListFields(tree_features)
        field_names = [f.name for f in fields]
        if 'Address' not in field_names:
            arcpy.AddField_management(tree_features, 'Address', "TEXT", field_length=500)
        if 'imagery_url' not in field_names:
            arcpy.AddField_management(tree_features, 'imagery_url', "TEXT", field_length=500)
        if 'description' not in field_names:
            arcpy.AddField_management(tree_features, 'description', "TEXT", field_length=500)
        with arcpy.da.InsertCursor(tree_features, ["shape@json", "Address", "imagery_url", "description", "Confidence"]) as cursor:
            with arcpy.da.SearchCursor(output_features, ['shape@json', 'shape@xy', 'Confidence'], spatial_reference=arcpy.SpatialReference(4326)) as rows:
                for row in rows:
                    loc = "location={0},{1}".format(row[1][0], row[1][1])
                    geocode_url = geocode_url.format(loc)
                    res = requests.get(geocode_url)
                    address = res.json()['address']['LongLabel']
                    confidence = row[2]
                    cursor.insertRow([row[0], address, imagery_url, description, confidence])
    except:
        pass
    # Must get the repository ID, Pipeline ID and Saved Search ID
    status_writer.send_status("Indexing trees...")
    res = requests.put("https://ogc-dp23.voyagersearch.com/hq/api/repos/r18a8a1d7ce1/index?delta=true&pipeline=p18a8a1d810a&clear=false", auth=("admin", "Voyager909"))
    saved_search = "https://ogc-dp23.voyagersearch.com/navigo/search?disp=default&fq=location:r18a8a1d7ce1&sort=score%20desc&filter=true&basemap=ESRI%20Imagery%20Map&id=S18A8EEC53CA"
    status_writer.send_status('Updating index...')
    r = requests.get("https://ogc-dp23.voyagersearch.com/hq/api/repos/r18a8a1d7ce1/state", auth=("admin", "Voyager909"))
    while not r.json()['state'] == 'IDLE':
        time.sleep(5)
        r = requests.get("https://ogc-dp23.voyagersearch.com/hq/api/repos/r18a8a1d7ce1/state", auth=("admin", "Voyager909"))
        continue
    webbrowser.open_new_tab(saved_search)

Creating Voyager Processing Task Stubs
Cooper Labenske posted on Sept 18, 2023
Voyager Processing Stubs.mp4

Resources:

https://livingatlas.arcgis.com/en/browse/?q=dlpk%20detection#d=2&q=dlpk+detection

EXAMPLE.info.json

{
    "name": "EXAMPLE",
    "runner": "python",
    "categories": ["ESRI - Deep Learning Machine Models (Coming Soon)"],
    "params": [
      {
        "type": "Geometry",
        "name": "processing_extent",
        "extentParam": true,
        "initWithResultsExtent": true
      },
      {
        "type": "VoyagerResults",
        "name": "input_items",
        "required": true
      },
      {
        "type": "CatalogPath",
        "name": "EXAMPLE_model",
        "required": true
      }
    ],
    "display":
    {
      "en":
      {
        "display": "EXAMPLE - Sample Only. Not for Official Use",
        "description": "EXAMPLE. Sample only.",
        "helpURL": "EXAMPLE URL",
        "params":
        {
          "processing_extent":
          {
              "display": "Processing Extent",
              "description": "The geographic extent of the area to be processed (in the WGS84 coordinate system). If not specified, the full extent of inputs is used."
          },
          "EXAMPLE_model": {
            "display": "EXAMPLE Model File (.dplk)",
            "description": "The Model Definition parameter value can be an Esri model definition JSON file (.emd), or a deep learning model package (.dlpk). The .dlpk file must be stored locally."
          }
        }
      }
    }
  }