What is SmutRecsBot?

SmutRecsBot is the crawler smutrecs.com uses to read a small number of public pages about specific romance books, such as an author's content-warning page or a review. We use those pages to rate spice, tropes and content warnings, and we link back to them.

• It identifies itself as SmutRecsBot/0.1 (+https://smutrecs.com/bot).
• It reads and obeys robots.txt, including after every redirect.
• It makes at most one request per second to a site, and fetches only pages a search returned for a specific book.
• It doesn't log in, fill in forms, run scripts, or try to get past bot checks.
• We never copy book text. From each page we keep up to 8 word-for-word extracts of at most 300 characters, privately, to back up our ratings. They're never shown on smutrecs.com. Book pages link to the source and summarize it in our own words.

Some pages reach us through Anthropic's web fetch tool instead, for example pages that need scripts to show their text. That tool sends Anthropic's own user agent, so we check your robots.txt rules for SmutRecsBot ourselves and discard anything from a page they block.

To block it, add User-agent: SmutRecsBot and Disallow: / to your robots.txt. That also keeps your pages out of what we keep from Anthropic's tool.