how do I limit the crawler in this instance?
Hi, I discovered your software the other day and got it up and installed in a Kali VM earlier.
I had a question about how to curtail the crawler in this particular incident I ran across doing a learning-the-ropes scan of my personal Redmine website.
Redmine has an activity page where you can search for logged project events by date, and the crawler hit it and started supplying dates into the URL moving backward through time. It started at the current date and I finally stopped it in the 1780's or so, because I had a feeling it would roll all the way back to year 0. I saw the exclusion boxes in the webUI config, but I wasn't sure what I should put in them, or indeed if that's where I should be looking. I know nothing of Regex's (something I'll need to learn later for sure) but I'm guessing a relevant regex is what I'm supposed to place in that box?
Comments are currently closed for this discussion. You can start a new one.
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
Support Staff 1 Posted by Tasos Laskos on 01 May, 2013 12:33 AM
Hi Jon,
You're looking for the redundant option, it lets you specify a pattern (regexp in this case, as you rightly pointed out) and a counter, that tells the crawler to only go through paths that match the pattern only a certain amount of times.
So, if the URL is something like:
http://redmineserver/redmine/projects/myproject/activity?from=2013-03-31You can pass the following as an auto-redundant filter:
activity:1(i.e. only crawl activity paths one time).
Feel free to re-open this discussion if you need further assistance.
Cheers
Tasos Laskos closed this discussion on 01 May, 2013 12:33 AM.
Tasos Laskos re-opened this discussion on 01 May, 2013 12:43 AM
Support Staff 2 Posted by Tasos Laskos on 01 May, 2013 12:43 AM
I forgot to mention that there is a simpler way to do this sort of thing and since you're not familiar with regular expressions it may suit you better.
You can use the auto-redundant option, here's an explanation of what it does: https://github.com/Arachni/arachni/wiki/Command-line-user-interface.... You just pass it the counter.
(It's also available in the web interface, just pasted the CLI version as a reference.)
Tasos Laskos closed this discussion on 01 May, 2013 12:43 AM.