Home → Problems →

Arachni crawler not scanning pages requiring HTTP authentication

• Edit

averdonsmith-arachni

22 Nov, 2013 03:55 PM

I'm currently having some difficulty with the Autologin plugin for the web version of Arachni. I believe that I have supplied the correct details and I've checked this with the command line version of Arachni which logged that login was successful, and for both the web and command line version I'm getting the appropriate cookie session for the user. However the problem that I'm getting is that the crawler is not identifying any pages past this login screen so the pages that require HTTP authentication are not being scanned.

Can you tell me please if there are any other plugins/settings that have to be configured to enable the crawler to identify these HTTP restricted pages please?

1. Support Staff 1 Posted by Tasos Laskos on 22 Nov, 2013 04:05 PM

   

   Since a session is being established properly the problem probably lies elsewhere. Are there any paths to the restricted pages from the seed URL?

   If not, you can supply them yourself with the extend-paths option -- appears as a text area in the WebUI Profiles in the Spider section.

   • Edit

2. 2 Posted by averdonsmith-ar... on 22 Nov, 2013 04:33 PM

   

   OK, thanks for this. I'll give this a go and see what I get.

   • Edit

3. 3 Posted by averdonsmith-ar... on 10 Dec, 2013 11:07 AM

   

   Hi,

   Sorry for the update on this but I'm still getting this same issue, infact it appears to be very similar to the problem detailed on this post: http://support.arachni-scanner.com/discussions/problems/36-autologi.... I've just updated to the latest version of Arachni but site its not crawling the HTTP authenticated pages. The only difference with the application that I'm testing is that there is a redirect after the form is submitted, which is going to '/'. Would this redirect make much of a difference and is there a work around for this?

   • Edit

4. Support Staff 4 Posted by Tasos Laskos on 10 Dec, 2013 06:47 PM

   

   Just to make it clear, we're talking about 2 types of authentication here right? One form based and one using HTTP authentication?

   • Edit

5. 5 Posted by averdonsmith-ar... on 11 Dec, 2013 08:18 AM

   

   Sorry, I'm talking about form based authentication as in a user being required to enter a valid username and password. Using the command line Arachni appears to be getting the correct authentication as it receives the correct session cookie but after this it doesn't appear to make any attempt on scanning pages within the authenticated side of the site.

   • Edit

6. Support Staff 6 Posted by Tasos Laskos on 11 Dec, 2013 08:23 AM

   

   Thanks for clearing that up, however you probably forgot to answer my earlier question:
   Are there any actual paths from the initial page you are providing to the pages within the authenticated side for the crawler to follow? Any forms or links or static assets? (Btw, JS generated links can't be followed yet.)

   Also, did you try using the extended-paths option as I previously suggested to feed the crawler at least one path that's inside the restricted side of the site?

   Cheers

   • Edit

7. 7 Posted by averdonsmith-ar... on 11 Dec, 2013 08:33 AM

   

   Yes, from the initial page after login there are a number of HTML links for Aracnhi to be able to select. Once the parameters have been submitted there is a redirect to '/' - would this make a significant difference.

   I did try the extended-paths option as you previously mentioned but that didn't make any difference and was not included in the site map (list of pages scanned).

   Just so that you know the commands that I'm using are:

   ./arachni --plugin=autologin:url=,params='Username=&Password=',check='.' -e /logout

   • Edit

8. 8 Posted by averdonsmith-ar... on 11 Dec, 2013 08:51 AM

   

   Not sure if you got my last post as it hasn't appeared on here yet. There are some paths from the initial landing page that are HTML based and there is also a data table that contains links but these are JS generated so based on what you said these won't get scanned.

   It did try the extended paths option and that did scan that page but it didn't lead to any other pages being found.

   • Edit

9. Support Staff 9 Posted by Tasos Laskos on 11 Dec, 2013 11:27 PM

   

   Sorry you got caught by the spam filter for some reason, I suggest you register for an account to prevent this from happening again.

   The check you've provided for the plugin will match anything, you won't be able to tell if the login was successful or not.

   Also, regarding the redirect, it shouldn't be an issue, Arachni will follow it and update its cookies no-matter where in the login process they are set.

   Unfortunately, I don't think I can be of further help until I examine the website, would that be possible?

   Cheers

   • Edit

10. 10 Posted by averdonsmith-ar... on 12 Dec, 2013 08:53 AM

    

    I'm sorry this website is an internal project that contains sensitive information that cannot be opened up. I appreciate the time you've spent on this and I'll continue to investigate this further.

    I think the main obstacle is that there is javascript being used and quite a specific work flow pattern in getting from page to page. I don't know if Arachni supports this but I could see it being useful if Arachni could 'follow' a scan conducted manually by the user and then use this as a basis for its own automated scan?

    • Edit

11. Support Staff 11 Posted by Tasos Laskos on 12 Dec, 2013 08:56 AM

    

    Sure, you can use the proxy plugin to train the scanner via your browser.

    • Edit

12. 12 Posted by averdonsmith-ar... on 12 Dec, 2013 08:58 AM

    

    Ah right ok, I'll give this ago then. Thanks for this.

    • Edit

13. Tasos Laskos closed this discussion on 12 Apr, 2014 12:21 AM.