Page Comparison

...

• Links To Follow
  Add XPath Selectors for links to follow if Crawl All Links is false

• Concurrent Requests
  The Number of requests to be made concurrently (default is 10) - setting to 0 will disable any request throttling

• Depth Limit
  Depth of links to crawl from the start URLs, (default is 5) - set to 0 for unlimited depth

• Files To Index
  Specify the file extensions that will be indexed, if present

• Index Files As Linked Data
  If set to true, any files matching Files To Index will be added as links to the indexed page (viewable in the relationships tab of a record’s Detail View in Navigo) - if set to false, these files will be added as individual documents

• Allowed Domains
  Only links within the specified domains will be followed and indexed.

• Crawler Settings
  You can find information about additional settings here

• Field Mapping
  Specify fields and their corresponding XPath selectors - any content found with the selector will be added to the specified field