tag:support.arachni-scanner.com,2012-07-01:/discussions/suggestions/14112-crawl-urls-contained-in-robotstxtArachni: Discussion 2019-05-23T09:34:57Ztag:support.arachni-scanner.com,2012-07-01:Comment/472907522019-05-23T09:34:56Z2019-05-23T09:34:56ZCrawl URLs contained in robots.txt<div><p>I've seen that Arachni has a very good coverage and it checks for a robots.txt file. It could be interesting if it crawled also URLs contained in the robots.txt file.</p></div>baptisteC