RoktCrawl crawler policy

Effective 29 September 2026.

This policy states how Rokt's storefront crawler, RoktCrawl, identifies itself, how its requests can be verified, what it accesses, how it limits its requests, what it retains, and how a site operator can opt out.

1. Scope

Rokt is an e-commerce technology company. We work with online retailers and other businesses to present relevant offers to their customers at the moment of purchase. RoktCrawl is operated by Rokt and visits the public storefronts of businesses that work with us, whether as a partner running our placement or as an advertiser whose campaigns we serve. We crawl a site because understanding how its pages are structured — its search results, product pages and category pages — is what lets our systems classify pages and better serve offers, without you having to send us that information yourself. RoktCrawl is not a general web crawler and does not index the web.

2. Identification

Every request carries the token RoktCrawl/0.1 (+https://docs.crawl.rokt.com) appended to a standard browser user-agent string; product-catalogue requests also append the suffix token RoktCrawl-Catalog/0.1 after it. The token links to this site, and this page, https://docs.crawl.rokt.com/policy.html, is the crawler's policy. The browser and platform named in the rest of the string are the ones actually in use; the crawler does not disguise itself as another client.

3. Verification (Web Bot Auth)

Every request the crawler sends is signed with an HTTP Message Signature (RFC 9421, Ed25519) in the form the Web Bot Auth drafts describe, so a site or the bot-management service in front of it can verify cryptographically that a request came from RoktCrawl rather than from something copying its user-agent string. The Signature-Agent header names the origin that publishes the key; the public key directory at that origin is served without credentials, with the content type application/http-message-signatures-directory+json, and is itself signed with the same key. The keyid on each signature is the JWK thumbprint of the key in the directory.

Signature-Agent
https://crawl.rokt.com
Key directory
https://crawl.rokt.com/.well-known/http-message-signatures-directory

4. Robots exclusion

The crawler requests robots.txt before any other page and applies its rules strictly. The storefront crawler honours rules addressed to RoktCrawl first and falls back to *; the product-catalogue crawler honours rules addressed to RoktCrawl-Catalog first, then RoktCrawl, then *. A disallowed address is not requested, even when the site's own search form produced it. If robots.txt cannot be read because the site withholds or blocks it, the crawler treats the site as disallowing everything and requests nothing further. Rules are read afresh on every visit.

Partner storefronts whose User-agent: * rules disallow /search, query strings or product paths can allow RoktCrawl explicitly:

User-agent: RoktCrawl
Allow: /

Allowing product paths (Allow: /, Allow: /products/, Allow: /p/, Allow: /dp/, Allow: /pd/) enables both storefront URL-pattern discovery and product-catalogue metadata extraction (title, brand, category, price, availability and image) for offer relevancy. Allowing search and category paths (Allow: /search, Allow: /s, Allow: /collections/, Allow: /c/) enables search and category URL pattern classification. To throttle or scope the daily product-catalogue crawl separately from the weekly storefront sample, a site operator can add a User-agent: RoktCrawl-Catalog block or set Crawl-delay (in seconds, honoured on top of the 1-second catalog minimum spacing):

User-agent: RoktCrawl
Allow: /

User-agent: RoktCrawl-Catalog
Allow: /
Crawl-delay: 2

5. What is accessed

A storefront URL-pattern visit consists of: the site's robots.txt, its sitemap files, its homepage, and a browser session that submits a short list of generic product words to the site's own search box and browses a bounded sample (up to 60 pages) of the resulting search, product and category pages, including their sorting, filtering and paging controls. A product-catalogue crawl fetches individual product pages seen on partner traffic to read public structured product metadata (schema.org/Product JSON-LD and OpenGraph fields) after checking the host's robots.txt.

The crawler does not sign in, does not create accounts, does not add items to a cart, does not enter checkout, and does not submit any form other than the search box. Addresses for cart, bag, checkout, order, account and login pages are excluded from browsing. It does not attempt to bypass rate limiting, bot detection or any other access control: a page that answers with a block or a challenge is recorded as blocked and is not retried.

6. Request rate

Storefront discovery requests to a single site are spaced at least 2.5 seconds apart, with up to 1.5 seconds of additional random delay, and the crawler loads one page of a site at a time. Within a page load, the browser retrieves the page's own assets as an ordinary browser would. Each visit covers a bounded number of pages, and scheduled crawls run approximately weekly per region. Product-catalogue fetches make a single HTTP request per product page when structured product markup is present in the HTML, spaced at least 1 second apart (with up to 0.5 seconds of random jitter), or at the Crawl-delay set under User-agent: RoktCrawl-Catalog. Requests time out after 30 seconds and follow at most ten redirects.

7. What is retained

The crawler retains, for each page it loads: the address, the response headers, a copy of the page body, a screenshot, the addresses of requests the page made, and the crawler's own log of the visit. This material is stored in Rokt's own systems and used to derive and verify the address patterns described above and to describe the storefront's technical characteristics (platform, search capabilities, structured data). Retention period: to be set.

The crawler has no account on any site and sees only what an anonymous visitor sees. It does not seek out personal data, and when it later reads search terms out of addresses, values that appear to be a person's details are withheld rather than recorded.

8. Source addresses

The crawler operates only from the fixed network addresses below, so that a site operator can verify that a request identifying itself as RoktCrawl originated from Rokt. A request carrying the RoktCrawl token from any other address did not come from Rokt. The same list is published as plain text, one address per line in CIDR form, at https://docs.crawl.rokt.com/ips.txt.

RegionAddresses
US West (Oregon)16.145.117.88/32 32.186.91.66/32 184.33.228.4/32
US East (N. Virginia)3.212.239.12/32 34.195.230.14/32 100.56.59.13/32
Europe (Ireland)54.73.34.189/32 3.248.25.71/32 34.242.239.9/32
Asia Pacific (Sydney)13.54.60.54/32 32.236.51.124/32 13.211.111.197/32

9. Opting out

A site operator may exclude the crawler at any time with a robots.txt rule:

User-agent: RoktCrawl
Disallow: /

The exclusion takes effect at the next visit. An operator may also ask Rokt to exclude a site directly using the contact below; such requests are honoured without requiring a robots rule.

10. Contact

Questions, exclusion requests and reports about the crawler's behaviour go to crawler@rokt.com. Include the site's hostname and, where available, the time and source address of the requests concerned.

11. Changes to this policy

If the crawler's identification, behaviour or source addresses change, this page is updated before the change takes effect.