Crawl URLs contained in robots.txt

baptisteC's Avatar


23 May, 2019 09:34 AM

I've seen that Arachni has a very good coverage and it checks for a robots.txt file. It could be interesting if it crawled also URLs contained in the robots.txt file.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:


Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac