RoktCrawl: Rokt's storefront crawler

If RoktCrawl appears in your server logs, this page explains what it is, what it does on your site, and how to reach us or ask it to stop.

Who we are

Rokt is an e-commerce technology company. We work with online retailers and other businesses to present relevant offers to their customers at the moment of a purchase. RoktCrawl is operated by Rokt and visits the public storefronts of businesses that work with us.

What the crawler does

RoktCrawl learns how a storefront's web addresses are structured: which URLs are search results, which are product pages and which are category pages, and how a search term or product identifier is written into the address. Rokt's systems use that knowledge to recognise the kind of page a shopper is on from its address alone, without inspecting the page itself.

RoktCrawl runs two storefront crawl modes:

How to recognise it

Every request the crawler makes carries this token at the end of an otherwise standard browser user-agent string:

RoktCrawl/0.1 (+https://docs.crawl.rokt.com)

A full user-agent string from the crawler looks like this (the Chrome version follows the browser build in use):

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 RoktCrawl/0.1 (+https://docs.crawl.rokt.com)

The token links to this site, where the crawler's policy is at docs.crawl.rokt.com/policy.html. The rest of the string is the real browser the crawler drives; nothing about the platform or browser is disguised.

Every request is also signed (Web Bot Auth, RFC 9421), so it can be verified against the crawler's published key rather than its user-agent string alone; the key directory and the fixed addresses the crawler sends from are in the policy's verification and source addresses sections.

What it does not do

robots.txt

The crawler reads robots.txt before anything else and obeys it strictly. A URL that robots.txt disallows is not fetched: if the site's search form produces a disallowed address, the address is noted but the page is never loaded. The storefront crawler matches rules addressed to RoktCrawl first and falls back to *; the product-catalogue crawler matches RoktCrawl-Catalog first, then RoktCrawl, then *. Rules are read afresh on every visit.

Allowing RoktCrawl (recommended for partners): Many storefronts disallow /search, query strings or product paths under User-agent: *. Adding an explicit RoktCrawl block overrides those general blocks for both storefront URL classification and product-catalogue enrichment:

User-agent: RoktCrawl
Allow: /

What each path rule enables:

Controlling product-catalogue rate or access separately (RoktCrawl-Catalog and Crawl-delay): Because product-catalogue requests carry RoktCrawl/0.1 (+https://docs.crawl.rokt.com) RoktCrawl-Catalog/0.1, a site operator can govern both crawls with a single User-agent: RoktCrawl block, or set a separate Crawl-delay (in seconds, honoured on top of the 1-second catalog minimum spacing) or access rule for the daily product-catalogue crawl under User-agent: RoktCrawl-Catalog:

User-agent: RoktCrawl
Allow: /

User-agent: RoktCrawl-Catalog
Allow: /
Crawl-delay: 2

Opting out: To keep the crawler off your site entirely, add:

User-agent: RoktCrawl
Disallow: /

Request rate

Storefront discovery page loads to one site are spaced at least 2.5 seconds apart, plus up to 1.5 seconds of random extra delay, and the crawler visits one page of a site at a time. Within a page load, the browser fetches the page's own images, styles and scripts as any browser would. Product-catalogue fetches make a single HTTP request per product page when structured product markup is present in the HTML, spaced at least 1 second apart (plus up to 0.5 seconds of random jitter), or at the Crawl-delay set under User-agent: RoktCrawl-Catalog.

Egress addresses

The crawler leaves Rokt's network only through the fixed addresses below, so a request claiming to be RoktCrawl can be verified against them. The full list, with the policy that governs it, is on the crawler policy page and as plain text at ips.txt.

RegionAddresses
US West (Oregon)16.145.117.88/32 32.186.91.66/32 184.33.228.4/32
US East (N. Virginia)3.212.239.12/32 34.195.230.14/32 100.56.59.13/32
Europe (Ireland)54.73.34.189/32 3.248.25.71/32 34.242.239.9/32
Asia Pacific (Sydney)13.54.60.54/32 32.236.51.124/32 13.211.111.197/32

Contact and opting out

The quickest way to stop the crawler is the robots.txt rule above. To ask us to exclude your site, to report a problem with the crawler's behaviour on your site, or to verify that a request came from us, contact us at:

crawler@rokt.com

Please include your site's hostname and, if you have it, the time and source address of the requests you are asking about.

The crawler policy states these commitments formally.