Run a crawl

Start it

On the project page, choose Start crawl (or Crawl again once a crawl has finished). A dialog says what the crawl does and how much of your page budget is left, and Crawl now starts it. The crawl runs on our servers: you can close the tab and come back later. A project runs one crawl at a time; Cancel on the progress card stops it.

The stages

The progress card steps through six stages, each with its own percentage:

Finding pages

Reads robots.txt, the sitemap and the start page, so the crawl knows where to begin.

Crawling pages

Fetches every page in scope and records its links. The counter shows pages fetched out of pages found so far; it can grow while the crawl finds new pages.

Connecting the graph

Resolves every link to the page it lands on, following redirects, and separates navigation links from content links.

Ranking pages

Computes internal PageRank, click depth and the layout of the link graph.

Reading content

Splits each page into sections and reads their text, so pages can be compared by topic.

Finding link opportunities

Matches passages with the pages they should link to. These become the Suggestions.

How long a crawl takes depends mostly on the site: how many pages it has and how fast it answers. A site that asks for a delay between requests in robots.txt takes longer.

How the crawler behaves

The crawler is LinkTasteBot. It starts from the project's start address and the sitemap and follows links from there, staying inside the project's scope.

• It follows robots.txt: the rules for LinkTasteBot, or for * when there are none, and a Crawl-delay of up to 5 seconds.
• It makes at most 6 requests at once and starts at most 5 per second.
• It reads the HTML your server sends and does not run JavaScript. Links that only appear after scripts run are not seen; the overview warns when a site looks like that.

Your server or firewall should allow its user agent. Details and the exact string are on the LinkTasteBot page.

Where a crawl stops

A crawl ends when it has fetched every page it can reach in scope, or when it reaches the page limit of your plan. Pages found beyond the limit are not fetched; the crawl is marked Unfinished and says how many it found, so you know how much of the site it covered. A crawl also stops after 72 hours, which only a very slow site reaches; the pages fetched until then are analysed.

Every fetched page also counts against your plan's page budget. If the budget runs out mid-crawl, the crawl stops fetching and analyses the pages it has, and is marked Unfinished too. Once your plan has room again, after an upgrade or when the budget renews, choose Finish crawl in its results or on the project page: it fetches the pages it left out and analyses the whole crawl again. See Plans & limits.

Crawl again

Each crawl is a snapshot of the site at the time it ran. After you have changed your links, crawl again to see the effect. A project keeps its three newest finished crawls; when a fourth finishes, the oldest is deleted. The crawl history on the project page lists them, and the crawl switcher in the results header moves between them.

A new crawl is compared with the one before: the crawl card lists the new suggestions, new, changed and gone pages, and the approved links that need a look. Under Schedule on the project page, the project crawls itself every week or every month. See Your links & the snippet.