tag:support.arachni-scanner.com,2012-07-01:/discussions/problems/187-arachni-crawler-not-scanning-pages-requiring-http-authenticationArachni: Discussion 2018-10-19T07:41:33Ztag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-11-22T16:05:38Z2013-11-22T16:05:38ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Since a session is being established properly the problem
probably lies elsewhere. Are there any paths to the restricted
pages from the seed URL?</p>
<p>If not, you can supply them yourself with the
<code>extend-paths</code> option -- appears as a text area in the
WebUI Profiles in the <em>Spider</em> section.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-11-22T16:33:44Z2013-12-12T08:54:23ZArachni crawler not scanning pages requiring HTTP authentication<div><p>OK, thanks for this. I'll give this a go and see what I get.</p></div>averdonsmith-arachnitag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-10T11:07:47Z2013-12-12T08:54:24ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Hi,</p>
<p>Sorry for the update on this but I'm still getting this same
issue, infact it appears to be very similar to the problem detailed
on this post: <a href=
"http://support.arachni-scanner.com/discussions/problems/36-autologin-does-not-crawl">
http://support.arachni-scanner.com/discussions/problems/36-autologi...</a>.
I've just updated to the latest version of Arachni but site its not
crawling the HTTP authenticated pages. The only difference with the
application that I'm testing is that there is a redirect after the
form is submitted, which is going to '/'. Would this redirect make
much of a difference and is there a work around for this?</p></div>averdonsmith-arachnitag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-10T18:47:49Z2013-12-10T18:47:49ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Just to make it clear, we're talking about 2 types of
authentication here right? One form based and one using HTTP
authentication?</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-11T08:18:01Z2013-12-12T08:54:24ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Sorry, I'm talking about form based authentication as in a user
being required to enter a valid username and password. Using the
command line Arachni appears to be getting the correct
authentication as it receives the correct session cookie but after
this it doesn't appear to make any attempt on scanning pages within
the authenticated side of the site.</p></div>averdonsmith-arachnitag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-11T08:23:57Z2013-12-11T08:23:57ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Thanks for clearing that up, however you probably forgot to
answer my earlier question:<br>
Are there any actual paths from the initial page you are providing
to the pages within the authenticated side for the crawler to
follow? Any forms or links or static assets? (Btw, JS generated
links can't be followed yet.)</p>
<p>Also, did you try using the <code>extended-paths</code> option
as I previously suggested to feed the crawler at least one path
that's inside the restricted side of the site?</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-11T08:33:31Z2013-12-12T08:54:24ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Yes, from the initial page after login there are a number of
HTML links for Aracnhi to be able to select. Once the parameters
have been submitted there is a redirect to '/' - would this make a
significant difference.</p>
<p>I did try the extended-paths option as you previously mentioned
but that didn't make any difference and was not included in the
site map (list of pages scanned).</p>
<p>Just so that you know the commands that I'm using are:</p>
<p>./arachni
--plugin=autologin:url=,params='Username=&Password=',check='.'
-e /logout</p></div>averdonsmith-arachnitag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-11T08:51:33Z2013-12-12T08:54:25ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Not sure if you got my last post as it hasn't appeared on here
yet. There are some paths from the initial landing page that are
HTML based and there is also a data table that contains links but
these are JS generated so based on what you said these won't get
scanned.</p>
<p>It did try the extended paths option and that did scan that page
but it didn't lead to any other pages being found.</p></div>averdonsmith-arachnitag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-11T23:27:53Z2013-12-11T23:27:53ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Sorry you got caught by the spam filter for some reason, I
suggest you register for an account to prevent this from happening
again.</p>
<p>The check you've provided for the plugin will match anything,
you won't be able to tell if the login was successful or not.</p>
<p>Also, regarding the redirect, it shouldn't be an issue, Arachni
will follow it and update its cookies no-matter where in the login
process they are set.</p>
<p>Unfortunately, I don't think I can be of further help until I
examine the website, would that be possible?</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-12T08:53:39Z2013-12-12T08:53:40ZArachni crawler not scanning pages requiring HTTP authentication<div><p>I'm sorry this website is an internal project that contains
sensitive information that cannot be opened up. I appreciate the
time you've spent on this and I'll continue to investigate this
further.</p>
<p>I think the main obstacle is that there is javascript being used
and quite a specific work flow pattern in getting from page to
page. I don't know if Arachni supports this but I could see it
being useful if Arachni could 'follow' a scan conducted manually by
the user and then use this as a basis for its own automated
scan?</p></div>averdonsmith-arachnitag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-12T08:56:20Z2013-12-12T08:56:20ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Sure, you can use the <code>proxy</code> plugin to train the
scanner via your browser.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/301418672013-12-12T08:58:12Z2013-12-12T08:58:12ZArachni crawler not scanning pages requiring HTTP authentication<div><p>Ah right ok, I'll give this ago then. Thanks for this.</p></div>averdonsmith-arachni