The BumbleCrawl crawler

For site owners who found BumbleCrawl in their server logs: what it is, why it came by, and how to keep it out.

What it is

BumbleCrawl is a desktop SEO crawler. People install it on their own computer to audit websites: it reads pages and the links between them and checks them for technical issues such as broken links, redirects and missing titles.

By default it identifies itself with this user agent:

Mozilla/5.0 (compatible; BumbleCrawl/0.1; +https://bumblecrawl.com/bot)

Why it visited your site

A BumbleCrawl user started a crawl of your site, or of a site that links to yours. There is no central BumbleCrawl crawler and no index we build: every request comes from the computer of the person running the app, on sites they choose, and their IP address is the one in your logs. We have no list of who crawls which site.

Most users crawl their own sites or their clients' sites. Links to other sites are only checked for their status code by default, not crawled further.

How it behaves

  • It reads robots.txt and, by default, follows the rules for bumblecrawl, or for * when there are none.
  • By default it makes at most 5 requests at once. Users can lower that or set a rate limit.
  • It starts from the address it was given and follows links, and reads XML sitemaps.
  • With JavaScript rendering switched on, pages are loaded in a headless Chromium browser, which also requests the scripts, styles and images a page needs.

Because the app runs on users' own computers, they can change its settings: for example send a different user agent, or ignore robots.txt when auditing a site they are responsible for. Requests that do not carry the user agent above may still come from BumbleCrawl, and we cannot stop or trace them.

Blocking it

Keep BumbleCrawl off your site entirely:

User-agent: bumblecrawl
Disallow: /

Or keep it out of one directory with a narrower Disallow rule. robots.txt is read when a crawl starts, so a change applies from the next crawl on. To block requests regardless of robots.txt, match BumbleCrawl in the user agent at your server or firewall.

Contact

If BumbleCrawl causes trouble on your site, write to info@bumblecrawl.com. Include your domain and, if you have them, the times of the requests. As the requests come from our users' own computers, blocking the IP addresses in your logs is usually the fastest fix.