Crawling with 302 to alternate domains.
We are having issues crawling sites which perform 302's to alternate domains.
The user flow is as follows:
- User starts on: abc.example.com - server responds with 302 to:
login.alternatedomain.com - user logs in - user is redirected with
auth cookie back to abc.example.com/dashboard
The "follow redirects" setting only appears to honor redirects on the initial domain. Excluding every third-party domain in the included scripts is a pain.
Is there a way around this? or is it a configuration issue on our end?
Comments are currently closed for this discussion. You can start a new one.
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
Support Staff 1 Posted by Tasos Laskos on Apr 16, 2014 @ 04:01 PM
That is permitted for log-in purposes only, the
autologinandproxyplugins for example are allowed to visit out-of-scope resources.The only way to get around this is to properly authenticate so that you won't be redirected, either by using one of the plugins or by setting the cookies manually via the cookie-jar or cookie-string options.
2 Posted by trey.keifer on Apr 16, 2014 @ 04:20 PM
You may want to consider adding an allowed_domains scope to the crawler.
Properly authenticating is difficult at this point without strong JS support and we are seeing more and more of this type of activity with enterprise customers who are bringing together apps across many different parts of their cloud infrastructure.
Support Staff 3 Posted by Tasos Laskos on Apr 16, 2014 @ 04:25 PM
Those are different problems and v0.5 will take care of the JS form issue.
As for the allowed domains, you actually want to scan the SSO page? Because other than that I can't think of another reason it'd be beneficial in this case.
4 Posted by trey.keifer on Apr 16, 2014 @ 04:36 PM
Understood. We are looking forward to the 0.5 release...
We would prefer the ability to scan/spider multiple domains. The reason being - most large companies spread out parts of their web applications to multiple different hosts/sub-applications to form one overall solution. You may have multiple domains handling SSO, reporting interfaces, API's, etc.. as a part of an overall "web application" offering. As technologists, we view all of those as separate apps - but companies just view it as one "solution" and want everything tested together.
Support Staff 5 Posted by Tasos Laskos on Apr 16, 2014 @ 04:57 PM
Well, yeah, but that doesn't make it a good reason.
Arachni's responsibility is to scan a web application[1] not a infrastructure/network of web applications. Your responsibility is to make this look transparent to your clients. Your clients' responsibility is to take steps to ensure they're webapps are secure, so they hire you.
Also, if you have foreknowledge of the different domains to scan (which you must if you want to whitelist them) why not scan them in parallel in the first place?
Still, the one thing I'm not willing to change in Arachni is the 1 site per scan, 1 scan per process architecture.
[1] Ok there may be different webapps under the same domain but you got to draw the line somewhere and so far asI can tell this is the best compromise.
6 Posted by trey.keifer on Apr 16, 2014 @ 05:26 PM
Fair enough. You obviously understand the impact of architectural changes on the overall codebase better than we do.
I can tell you from using Arachni on hundreds of sites that the idea of a web application being tied to a single domain is not that straightforward nowaday. With SSO and URL tokens a "web app" can span many different properties.
Regardless, I think the solution is "not possible" so you can close this out. If you would like to discuss further, we've got each others contact info and I'm happy to share our experiences thus far.
Thanks Tasos
Tasos Laskos closed this discussion on Apr 16, 2014 @ 05:34 PM.