RoktCrawl: Rokt's storefront crawler
If RoktCrawl appears in your server logs, this page explains what it is,
what it does on your site, and how to reach us or ask it to stop.
Who we are
Rokt is an e-commerce technology company. We work with online retailers and other businesses to present relevant offers to their customers at the moment of a purchase. RoktCrawl is operated by Rokt and visits the public storefronts of businesses that work with us.
What the crawler does
RoktCrawl learns how a storefront's web addresses are structured: which URLs are search results, which are product pages and which are category pages, and how a search term or product identifier is written into the address. Rokt's systems use that knowledge to recognise the kind of page a shopper is on from its address alone, without inspecting the page itself.
RoktCrawl runs two storefront crawl modes:
- Storefront URL-pattern discovery (weekly sample): Fetches the site's
robots.txt, its sitemap and its homepage, then opens a real browser session that submits a handful of generic search terms to the site's own search box and browses a bounded sample (up to 60 pages per weekly visit) of search, category and product pages. - Product-catalogue enrichment (active product pages): Fetches individual product
pages seen on partner traffic to read public structured product metadata (
schema.org/ProductJSON-LD and OpenGraph fields such as title, brand, category, price, availability and image) to improve offer and campaign relevancy. Product-catalogue requests append the suffix tokenRoktCrawl-Catalog/0.1afterRoktCrawl/0.1 (+https://docs.crawl.rokt.com).
How to recognise it
Every request the crawler makes carries this token at the end of an otherwise standard browser user-agent string:
RoktCrawl/0.1 (+https://docs.crawl.rokt.com)
A full user-agent string from the crawler looks like this (the Chrome version follows the browser build in use):
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 RoktCrawl/0.1 (+https://docs.crawl.rokt.com)
The token links to this site, where the crawler's policy is at docs.crawl.rokt.com/policy.html. The rest of the string is the real browser the crawler drives; nothing about the platform or browser is disguised.
Every request is also signed (Web Bot Auth, RFC 9421), so it can be verified against the crawler's published key rather than its user-agent string alone; the key directory and the fixed addresses the crawler sends from are in the policy's verification and source addresses sections.
What it does not do
- It never logs in or creates an account. Pages behind a sign-in are never requested.
- It never enters the cart or checkout. Links to cart, bag, checkout, account, order and login pages are excluded from browsing.
- It submits no form other than the search box, and types nothing into it but a short list of generic product words.
- It does not evade access controls. A page that answers with a block or a challenge is
recorded as blocked and not retried; a site whose
robots.txtis withheld is treated as disallowing everything. - It collects no personal data. It has no account on your site and sees only what an anonymous visitor sees. Search terms it extracts from addresses later are screened so that anything that looks like a person's details is withheld.
robots.txt
The crawler reads robots.txt before anything else and obeys it strictly. A URL that
robots.txt disallows is not fetched: if the site's search form produces a disallowed address, the
address is noted but the page is never loaded. The storefront crawler matches rules addressed to
RoktCrawl first and falls back to *; the product-catalogue crawler matches
RoktCrawl-Catalog first, then RoktCrawl, then *. Rules are read afresh on
every visit.
Allowing RoktCrawl (recommended for partners): Many storefronts disallow
/search, query strings or product paths under User-agent: *. Adding an explicit
RoktCrawl block overrides those general blocks for both storefront URL classification and
product-catalogue enrichment:
User-agent: RoktCrawl
Allow: /
What each path rule enables:
- Product-catalogue enrichment (
Allow: /or your product paths such asAllow: /products/,Allow: /p/,Allow: /dp/,Allow: /pd/): lets Rokt read structured product metadata (title, brand, category, price, availability and image) from product pages to improve offer relevancy. - Storefront search and category classification (
Allow: /search,Allow: /s,Allow: /collections/,Allow: /c/): lets Rokt learn search, category and product URL patterns and query parameters whenUser-agent: *disallows/searchor query strings.
Controlling product-catalogue rate or access separately (RoktCrawl-Catalog and
Crawl-delay): Because product-catalogue requests carry
RoktCrawl/0.1 (+https://docs.crawl.rokt.com) RoktCrawl-Catalog/0.1, a site operator can
govern both crawls with a single User-agent: RoktCrawl block, or set a separate
Crawl-delay (in seconds, honoured on top of the 1-second catalog minimum spacing) or access rule
for the daily product-catalogue crawl under User-agent: RoktCrawl-Catalog:
User-agent: RoktCrawl
Allow: /
User-agent: RoktCrawl-Catalog
Allow: /
Crawl-delay: 2
Opting out: To keep the crawler off your site entirely, add:
User-agent: RoktCrawl
Disallow: /
Request rate
Storefront discovery page loads to one site are spaced at least 2.5 seconds apart, plus up to
1.5 seconds of random extra delay, and the crawler visits one page of a site at a time. Within a page
load, the browser fetches the page's own images, styles and scripts as any browser would.
Product-catalogue fetches make a single HTTP request per product page when structured product markup
is present in the HTML, spaced at least 1 second apart (plus up to 0.5 seconds of random jitter), or
at the Crawl-delay set under User-agent: RoktCrawl-Catalog.
Egress addresses
The crawler leaves Rokt's network only through the fixed addresses below, so a request claiming to be RoktCrawl can be verified against them. The full list, with the policy that governs it, is on the crawler policy page and as plain text at ips.txt.
| Region | Addresses |
|---|---|
| US West (Oregon) | 16.145.117.88/32 32.186.91.66/32 184.33.228.4/32 |
| US East (N. Virginia) | 3.212.239.12/32 34.195.230.14/32 100.56.59.13/32 |
| Europe (Ireland) | 54.73.34.189/32 3.248.25.71/32 34.242.239.9/32 |
| Asia Pacific (Sydney) | 13.54.60.54/32 32.236.51.124/32 13.211.111.197/32 |
Contact and opting out
The quickest way to stop the crawler is the robots.txt rule above. To ask us to exclude
your site, to report a problem with the crawler's behaviour on your site, or to verify that a request
came from us, contact us at:
Please include your site's hostname and, if you have it, the time and source address of the requests you are asking about.
The crawler policy states these commitments formally.