Arachni crawler not scanning pages requiring HTTP authentication

averdonsmith-arachni's Avatar


22 Nov, 2013 03:55 PM

I'm currently having some difficulty with the Autologin plugin for the web version of Arachni. I believe that I have supplied the correct details and I've checked this with the command line version of Arachni which logged that login was successful, and for both the web and command line version I'm getting the appropriate cookie session for the user. However the problem that I'm getting is that the crawler is not identifying any pages past this login screen so the pages that require HTTP authentication are not being scanned.

Can you tell me please if there are any other plugins/settings that have to be configured to enable the crawler to identify these HTTP restricted pages please?

  1. Support Staff 1 Posted by Tasos Laskos on 22 Nov, 2013 04:05 PM

    Tasos Laskos's Avatar

    Since a session is being established properly the problem probably lies elsewhere. Are there any paths to the restricted pages from the seed URL?

    If not, you can supply them yourself with the extend-paths option -- appears as a text area in the WebUI Profiles in the Spider section.

  2. 2 Posted by averdonsmith-ar... on 22 Nov, 2013 04:33 PM

    averdonsmith-arachni's Avatar

    OK, thanks for this. I'll give this a go and see what I get.

  3. 3 Posted by averdonsmith-ar... on 10 Dec, 2013 11:07 AM

    averdonsmith-arachni's Avatar


    Sorry for the update on this but I'm still getting this same issue, infact it appears to be very similar to the problem detailed on this post: I've just updated to the latest version of Arachni but site its not crawling the HTTP authenticated pages. The only difference with the application that I'm testing is that there is a redirect after the form is submitted, which is going to '/'. Would this redirect make much of a difference and is there a work around for this?

  4. Support Staff 4 Posted by Tasos Laskos on 10 Dec, 2013 06:47 PM

    Tasos Laskos's Avatar

    Just to make it clear, we're talking about 2 types of authentication here right? One form based and one using HTTP authentication?

  5. 5 Posted by averdonsmith-ar... on 11 Dec, 2013 08:18 AM

    averdonsmith-arachni's Avatar

    Sorry, I'm talking about form based authentication as in a user being required to enter a valid username and password. Using the command line Arachni appears to be getting the correct authentication as it receives the correct session cookie but after this it doesn't appear to make any attempt on scanning pages within the authenticated side of the site.

  6. Support Staff 6 Posted by Tasos Laskos on 11 Dec, 2013 08:23 AM

    Tasos Laskos's Avatar

    Thanks for clearing that up, however you probably forgot to answer my earlier question:
    Are there any actual paths from the initial page you are providing to the pages within the authenticated side for the crawler to follow? Any forms or links or static assets? (Btw, JS generated links can't be followed yet.)

    Also, did you try using the extended-paths option as I previously suggested to feed the crawler at least one path that's inside the restricted side of the site?


  7. 7 Posted by averdonsmith-ar... on 11 Dec, 2013 08:33 AM

    averdonsmith-arachni's Avatar

    Yes, from the initial page after login there are a number of HTML links for Aracnhi to be able to select. Once the parameters have been submitted there is a redirect to '/' - would this make a significant difference.

    I did try the extended-paths option as you previously mentioned but that didn't make any difference and was not included in the site map (list of pages scanned).

    Just so that you know the commands that I'm using are:

    ./arachni --plugin=autologin:url=,params='Username=&Password=',check='.' -e /logout

  8. 8 Posted by averdonsmith-ar... on 11 Dec, 2013 08:51 AM

    averdonsmith-arachni's Avatar

    Not sure if you got my last post as it hasn't appeared on here yet. There are some paths from the initial landing page that are HTML based and there is also a data table that contains links but these are JS generated so based on what you said these won't get scanned.

    It did try the extended paths option and that did scan that page but it didn't lead to any other pages being found.

  9. Support Staff 9 Posted by Tasos Laskos on 11 Dec, 2013 11:27 PM

    Tasos Laskos's Avatar

    Sorry you got caught by the spam filter for some reason, I suggest you register for an account to prevent this from happening again.

    The check you've provided for the plugin will match anything, you won't be able to tell if the login was successful or not.

    Also, regarding the redirect, it shouldn't be an issue, Arachni will follow it and update its cookies no-matter where in the login process they are set.

    Unfortunately, I don't think I can be of further help until I examine the website, would that be possible?


  10. 10 Posted by averdonsmith-ar... on 12 Dec, 2013 08:53 AM

    averdonsmith-arachni's Avatar

    I'm sorry this website is an internal project that contains sensitive information that cannot be opened up. I appreciate the time you've spent on this and I'll continue to investigate this further.

    I think the main obstacle is that there is javascript being used and quite a specific work flow pattern in getting from page to page. I don't know if Arachni supports this but I could see it being useful if Arachni could 'follow' a scan conducted manually by the user and then use this as a basis for its own automated scan?

  11. Support Staff 11 Posted by Tasos Laskos on 12 Dec, 2013 08:56 AM

    Tasos Laskos's Avatar

    Sure, you can use the proxy plugin to train the scanner via your browser.

  12. 12 Posted by averdonsmith-ar... on 12 Dec, 2013 08:58 AM

    averdonsmith-arachni's Avatar

    Ah right ok, I'll give this ago then. Thanks for this.

  13. Tasos Laskos closed this discussion on 12 Apr, 2014 12:21 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac