Links with onclick event not followed
Hi,
I'm scanning a webapp with Arachni but it doesn't seem to follow some links.
Here is an example of link which is not followed :
<a href="#"
onclick="mojarra.jsfcljs(document.getElementById('usersList:userListForm'),{'usersList:userListForm:usersTable:j_idt30':'usersList:userListForm:usersTable:j_idt30'},'');return false"
class="btn btn-success">
New user
</a>
It's a link generated by the jsf code :
<h:commandLink action="#{myBean.createNewUser()}" styleClass="btn btn-success"> #{msg['new_user']}</h:commandLink>
I know it's not followed because I got in the arachni logs :
[-] BrowserCluster Worker#37948840: Job timed-out after 25 seconds: #<Arachni::BrowserCluster::Jobs::ResourceExploration::EventTrigger:32519920 @resource=#<Arachni::Page:18375440 @url="https://myserver/mywebapp/users/list.xhtml" @dom=#<Arachni::Page::DOM:18371360 @url="https://myserver/mywebapp/users/list.xhtml" @transitions=40 @data_flow_sinks=0 @execution_flow_sinks=0>> @event=:onclick @element=<a href="#" onclick="mojarra.jsfcljs(document.getElementById('usersList:userListForm'),{'usersList:userListForm:usersTable:j_idt30':'usersList:userListForm:usersTable:j_idt30'},'');return false" class="btn btn-success"> time=25.096439984 timed_out=true>
And because I don't see the page-we-should-see-if-we-follow-the-link in the arachni report "sitemap" page.
My webapp is using :
Primefaces 4.1.0
JSF 2.2.10
Tomcat 7.0.64
I'm using Arachni 1.3.1, which I'm calling by the following command line :
arachni https://myserver/mywebapp/ --scope-dom-depth-limit=10 --scope-extend-paths="/path/to/arachni-paths.txt" --audit-links --audit-forms --audit-ui-inputs --audit-ui-forms --plugin=autologin:url=https://myserver/mywebapp/login.xhtml,parameters='j_username=myusername&j_password=mypassword,check='My account' --scope-exclude-pattern=logout --checks='sql_*,http_*,os_*,xss_*,insecure_*,allowed_methods,csrf,code_injection,directory_listing,emails,form_upload,session_fixation,xpath_injection' --platforms='linux,pgsql,oracle,tomcat,java,jsf' --report-save-path=pentest.afr && \
arachni_reporter pentest.afr --reporter=html:outfile=pentest.html.zip
I don't know if there is something I forgot in order to tell
Arachni to follow this kind of links, or if it's a bug.
We had some problems using PhantomJS for our Cucumber/Selenium
tests and we are finally using HtmlUnit instead. So maybe it's kind
of related to PhantomJS.
Did someone else have this problem ?
Did I miss something in my arachni command-line ?
Is it possible to configure Arachni to use another webdriver than
PhantomJS (HtmlUnit or FirefoxDriver, for example) ?
Thanks for your help,
Marie
Comments are currently closed for this discussion. You can start a new one.
Keyboard shortcuts
Generic
? | Show this help |
---|---|
ESC | Blurs the current field |
Comment Form
r | Focus the comment reply box |
---|---|
^ + ↩ | Submit the comment |
You can use Command ⌘
instead of Control ^
on Mac
Support Staff 1 Posted by Tasos Laskos on 12 Nov, 2015 07:30 PM
Hello,
No this isn't because of a misconfiguration, there may be a JS error or something causing the page not to load.
Could you please try the nightlies to see if the issue has been fixed?
If it's still there (or if you get a new issue, the nightles are unstable after all) you can send me all the details in private and I'll try to debug this.
Cheers
Support Staff 2 Posted by Tasos Laskos on 13 Nov, 2015 03:44 AM
Hello again,
I forgot to mention, you can't use an alternative browser, it's PhantomJS 1.9.2 only.
Cheers
3 Posted by marie_lbl on 13 Nov, 2015 10:38 AM
Hello,
Thanks for your quick reply.
I tried with the nightlies : the result is quite the same (the page I want is still not in the sitemap report).
I don't see this message (I got with Arachni 1.3.1) in the logs anymore :
I don't know if it's significant - maybe it doesn't try to click on the link anymore (in this case it's "worst" than in 1.3.1), or maybe you just don't display the "BrowserCluster Worker (...) Arachni::BrowserCluster::Jobs::ResourceExploration::EventTrigger" logs anymore (?).
But, for information, I still see
So I know my link is seen by Arachni. But it is not clicked/followed by Arachni.I double-checked : manually I can click on the link and go to the desired page, and I don't have any error in the Javascript console (browser : Chrome).
Thanks a lot for offering to debug !
I can't give you access to the webapp I'm scanning, but I will try to reproduce the problem in a little project, and give you all the informations you need to reproduce (if it's possible).
Support Staff 4 Posted by Tasos Laskos on 13 Nov, 2015 07:40 PM
Yes please, a small test case on its own would actually be better than access to the full webapp.
These JS issues are hard to troubleshoot so I need to have a look at it myself.
Let me know once you've put something together.
Cheers
5 Posted by marie_lbl on 16 Nov, 2015 01:51 PM
Ok so here is my small test case :
https://github.com/lebloism/link-not-followed-by-arachni
I tried to put enough information in the README, but tell me if you need something else.
(And thanks again for your support !)
Support Staff 6 Posted by Tasos Laskos on 17 Nov, 2015 01:16 AM
Best. Bug report. Ever.
I'm on it.
Support Staff 7 Posted by Tasos Laskos on 17 Nov, 2015 04:38 AM
Hm, I just tried it and it worked:
Here's the relevant bit:
I was using the code in the nightlies, are you getting different results?
8 Posted by marie_lbl on 17 Nov, 2015 06:12 AM
I am using the nightlies too, I committed the results on github so you can see it (pentest.afr and pentest.html.zip). I read the "html" report but normally it's the same, isn't it ?
The difference between you and me is the command line. I'm using quite the same command line as before (except for the "login" part, because my webapp is simpler, without login or HTTPS). This is the command line I'm using for the small test case :
Can you try with this command line and see if you reproduce ?
9 Posted by marie_lbl on 17 Nov, 2015 06:17 AM
Maybe the problem is the "scope-extend-paths" option : I thought it was "include these paths too", but maybe it is "include only these paths" ?
Support Staff 10 Posted by Tasos Laskos on 17 Nov, 2015 06:27 AM
Not unless I somehow mixed it with the restrict-paths option which isn't out of the question, but let's start simple.
Can you verify that you're getting to that page in the test app by only using the
--checks -
option?11 Posted by marie_lbl on 17 Nov, 2015 06:35 AM
I will as soon as I am in my office (I don't have the correct environment at home), so in 2-3 hours. I'll try with different command lines "between" yours and mine, so we'll be able to identify the option which raise the problem. I'll let you know.
Support Staff 12 Posted by Tasos Laskos on 17 Nov, 2015 06:48 AM
I actually managed to reproduce the issue so it's OK.
Turns out that it has to do with Arachni auditing the inputs of the hidden form wrapped around that link, which are used to pass viewstate tokens.
That audit invalidatd the viewstate so when the browsers click the link the application returns an error instead of redirecting to the edit page.
You can skip auditing those input vectors with:
The values to these options are treated as regular expressions, so any input vector that includes them will be ignored (one of them was
j_idt8
, not sure if the integer changes so I usedj_idt
as a catch all).I verified that it works on the sample webapp you setup (thanks for that), let me know how it does against the real thing.
Cheers
13 Posted by marie_lbl on 17 Nov, 2015 05:29 PM
Thanks for the hint.
Indeed it works on the sample webapp with the audit-exclude-vector options.
in order to exclude all the generated inputs (with and without an ID defined in the h:form - all my IDs for the h:form tags end with "form" or "Form") - even if I would have some interesting inputs not scanned. But it doesn't work : Arachni still doesn't follow the links.But it doesn't work in my real webapp :
1) I can't exclude all the inputs with "j_idt' in the 'name' attribute because some of them should be scanned.
2) On some pages, there is an ID defined for the form (h:form id="myForm") and, in these cases, the generated hidden input has for name "myForm" (= not containing "j_idt")
3) There is probably others complications because I tried to run the command line with
I'll try again tomorrow, and maybe give you an other sample webapp, closer to the real webapp than the first one.
But I would like to know if it would be possible to change the condition "regex on the 'name' attribute" ?
or like thatFor example, I would be very interested in excluding all the hidden inputs (type=hidden), and someone else could be interested in a condition like "regex on the 'id' attribute".
Is it possible to have a more flexible condition for example with 2 options : --audit-exclude-vector-attribute and --audit-exclude-vector-pattern.
So they could be used like that :
?
Support Staff 14 Posted by Tasos Laskos on 18 Nov, 2015 12:23 AM
You can't do that via CLI options because the configuration would get unworkable really fast, you can however write a simple plugin that adds a global callback to determine which elements should be skipped from the audit.
Do you want to push the new sample webapp first or shall I show you an example of such a plugin?
15 Posted by marie_lbl on 18 Nov, 2015 10:52 AM
I'm very interested in the plugin you're talking about, can you give me an example ?
I created an other branch in my github project, with a similar webapp but this time with 2 lists and 2 edit pages : https://github.com/lebloism/link-not-followed-by-arachni/tree/two-l...
If I run Arachni with the audit-exclude-vector options : it works when I have only 1 list and 1 edit pages (=branch "master"), but not when I have 2 lists and 2 edit pages (= branch "two-lists").
I don't understand why the viewState is still invalidated despite the audit-exclude-vector options on the "two-lists" branch. Do you have any idea ?
I hope it will help me to understand what it doesn't work in my real webapp.
(Thanks again for helping me)
Support Staff 16 Posted by Tasos Laskos on 18 Nov, 2015 12:03 PM
That was indeed a bug caused by a deduplication mixup. The DOM events of the 2 links were identical so only one was followed, luckily it was an easy fix.
And I also managed to reproduce the original time-out issue, not sure why that happens yet though.
I need to do some more testing but I'll have some nightlies for you tomorrow.
Cheers
Support Staff 17 Posted by Tasos Laskos on 19 Nov, 2015 06:53 AM
The fix is in the nightlies, if you exclude the vectors I had mentioned is should work fine.
If you still require the plugin let me know and I'll whip something up.
Support Staff 18 Posted by Tasos Laskos on 19 Nov, 2015 07:06 AM
Actually, I think I spoke too soon.
The missing page in the sample app may have been appropriate because the edit pages only differ in text nodes, from the browser's perspective processing just one of them is enough, there's nothing to be gained by including the second one in the scan.
Can you add something "interesting" to at least one of them please?
Like an
<input>
field.Support Staff 19 Posted by Tasos Laskos on 19 Nov, 2015 07:17 AM
I added the following to
edit2.xhtml
:And the following to
edit1.xhtml
:In that case both pages were logged appropriately since they were processed as they contained some useful workload.
So I'm guessing we're back to the invalidated viewstate, I'll get working on a sample plugin to let you filter elements.
Cheers
20 Posted by marie_lbl on 19 Nov, 2015 10:10 AM
1 ---
I got the nightlies, thanks !
2---
What do you mean by "In that case both pages were logged appropriately" ? Do you mean "with that code everything works now, the both edit pages are scanned by Arachni" ? Because it's not the case for me : I added your code in the edit pages (see updated branch "two-lists"), but the pages edit1 and edit2 are still not mentionned in the logs, nor in the report. (The list pages have now both 500 status code)
So maybe I did'nt understand you correctly, or we have differents results between you and me for the same code :S
3 ---
I'm trying to understand what Arachni does (to undestand how the viewstate is invalidated) by reading its logs.
Can you explain me the following logs ?
I understand these logs as :
generates a request for "list1.xhtml".the "click" on
This is OK because the network says, when I click on the link :
POST list1 -> 302 Found location = "http://david-virtualbox:8080/users/edit1.xhtml" -> GET edit1.
So the logs say "request for list1", OK, it's the POST part. But what about the remaining part "302 -> redirection to edit1" ?
Does Arachni read the "location" part of the 302 response, in order to do the request to the provided location ?
(Sorry maybe I don't undestand how Arachni works at all, so my questions would be totally stupid. Please be indulgent, I'm just trying to understand ^^' )
4 ---
I'm still interested by the plugin because I will need it for my real webapp. But I don't think it will help for the sample webapp : I'm already excluding all the hidden inputs generated by JSF (with regex on their names). So it seems Arachni is invalidating the ViewState but with something else than scanning the JSF generated hidden inputs.
Do you have an idea which other Arachni action could do that ?
Support Staff 21 Posted by Tasos Laskos on 19 Nov, 2015 10:25 AM
1, Better wait another half an hour or so for the new ones, the nightlies worked temporarily for me because I broke them. The original behavior was the correct one I think.
2. Yeah with the changes in the edit pages both of them were in the sitemap. Although right now I couldn't reproduce it either, instead getting loads of 500 errors too (always happened with both branches but intermittently). I'm pretty sure the test cases don't accurately reflect the issue that you're getting from the real webapp and we are instead chasing bugs in the sample ones.
3. You got it right and redirects will be followed, but in this case it's not the viewstate being invalidated but the sample app exploding for some reason.
4. We better focus on the real application and go from there, the sample ones don't seem representative of it. I don't think that it's about invalidated viewstates any more but rather these errors:
I'll let you know once I've got a sample plugin ready.
22 Posted by marie_lbl on 19 Nov, 2015 10:59 AM
Ok for stopping with the sample webapp which is maybe not representative anymore.
I'll wait for your sample plugin and then try again on the real webapp !
(Thx for your work)
Support Staff 23 Posted by Tasos Laskos on 20 Nov, 2015 07:19 AM
I'm afraid I've got some bad news, the forms are sometimes extracted from browser requests instead of the HTML code of the page, depending on which component gets there first.
This means that full context (like the HTML code, attribute details etc) isn't always available.
The only information that remains constant is the input data and the action, so the only reliable way of filtering element audits are the CLI scope and audit options.
24 Posted by marie_lbl on 20 Nov, 2015 10:57 AM
Ok :-(
Actually, maybe we don't need to exclude these inputs from the audit. What we need is Arachni to follow the links and discover the other pages BEFORE auditing these inputs and invalidating the viewstate.
Would it be possible for you to change the order of the actions ? (maybe through an option)
I suppose, from the behaviour we saw, that the current order is :
find a page, auditing the page inputs, discover other pages through the links present on the current page, auditing these pages and so one.
Would it be possible to have :
find a page, discover other pages through the links present on this page (store it for later), auditing the current page inputs, and then process the discovered pages
?
Support Staff 25 Posted by Tasos Laskos on 20 Nov, 2015 11:22 AM
That won't work in the grand scheme of things but I think I found another way around it.
It's probably going to hurt performance when the server doesn't set caching options for the response, so I'll try to enable it only when necessary.
Whatever the case, if this solves the problem I'm OK with a performance penalty if it means automatically taking care of these types of issues.
I'll keep you posted on my progress.
Cheers
Support Staff 26 Posted by Tasos Laskos on 20 Nov, 2015 02:34 PM
Nightlies are up (except Windows), let me know how they work.
27 Posted by marie_lbl on 20 Nov, 2015 05:12 PM
It discovered one page which was not listed in the scope-extend-paths file (users/edit.xhtml, accessible from a link on users/list.xhtml which is a page listed in the scope-extend-paths).
But this edit page has a 500 status code. Maybe the data transfered from the list to the edit page are not correctly transferred ?
FYI we use the JSF Flash scope in order to transfer data from a page to another : http://docs.oracle.com/javaee/6/api/javax/faces/context/Flash.html
"Variables stored in the flash scope will survive a redirection and they will be discarded afterwards. This is really useful when implementing a Post-Redirect-Get pattern" "Flash scope survives redirect, exactly once"
But, for every other similar situations (profiles/list.xhtml -> profiles/edit.xhtml, entities/list.xhtml -> entities/edit.xhtml, etc ), it didn't discover the edit pages.
And, regression : I have now 0 issues (which is not correct since I hade some little issues in the previous reports, like E-mail address disclosure)
Support Staff 28 Posted by Tasos Laskos on 21 Nov, 2015 06:58 AM
I don't think I can help without access to the web application at this point, any chance you can arrange that?
29 Posted by marie_lbl on 02 Dec, 2015 03:18 PM
Hi !
Sorry for the delay. I couldn't convince my team to give you access. We finally chose to stop trying to launch the tests automatically with Arachni, and we went back to our first try, which we knew was working as we need : manual tests with ZAP.
Maybe we will have more time later to give it a new try, but currently we can't.
Thanks again for all the effort you made, it's very pleasant to have such a support !
Support Staff 30 Posted by Tasos Laskos on 02 Dec, 2015 04:46 PM
That's too bad, I hate knowing that there's a simple bug somewhere lurking around causing trouble.
Still, I understand your team's reluctance to provide access.
If anything changes please do let me know.
Cheers
Tasos Laskos closed this discussion on 02 Dec, 2015 04:46 PM.