LinkTasteBot
What it is
LinkTasteBot is the crawler of LinkTaste, an internal linking tool. It reads a website's pages and the links between them so the site's owner or their SEO can see how link value flows and where links are missing.
It identifies itself with this user agent:
Mozilla/5.0 (compatible; LinkTasteBot/1.0; +https://linktaste.com/bot)
Why it visited your site
A LinkTaste user added your site as a project and started a crawl. LinkTasteBot does not crawl the web on its own: every visit is one crawl a user started, or a repeat crawl on a weekly or monthly schedule that user set for the project. A crawl stays on the host it was started for and ends at the page limit of that user's plan.
How it behaves
- It reads robots.txt once at the start of each crawl and follows the rules for LinkTasteBot, or for * when there are none.
- It honours Crawl-delay, up to 5 seconds between requests.
- Without a Crawl-delay it makes at most 6 requests at once and starts at most 5 per second.
- It starts from the address it was given and the sitemap, then follows links.
- It only sends GET requests. It never submits forms and does not run JavaScript.
- It only reads HTML pages and sitemaps.
Slowing it down
Ask for more time between requests:
User-agent: LinkTasteBot Crawl-delay: 2
Blocking it
Keep it off your site entirely:
User-agent: LinkTasteBot Disallow: /
Or keep it out of one directory with a narrower Disallow rule. robots.txt is read when a crawl starts, so a change applies from the next crawl on.
Contact
If LinkTasteBot causes trouble on your site, write to info@linktaste.com with your domain and, if you have them, the times of the requests.