tag:support.arachni-scanner.com,2012-07-01:/discussions/problems/4040-missing-pages-in-site-mapArachni: Discussion 2016-09-22T09:14:54Ztag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-06-29T10:43:40Z2016-06-29T10:43:40Zmissing pages in site map<div><p>Access to the webapp with a specific test case would be much
appreciated, that'd help me diagnose the issue.</p>
<p>Cheers</p>
<p>PS. Sorry for the late response.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-06-29T16:12:20Z2016-06-29T16:12:20Zmissing pages in site map<div><p>Sent some additional details in the contact form since it
includes private information.<br>
Thanks</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-06-29T16:16:05Z2016-06-29T16:16:05Zmissing pages in site map<div><p>Got it, thanks. :)</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-07-13T18:41:04Z2016-07-13T18:41:07Zmissing pages in site map<div><p>Any luck? or issue accessing the app using the info I sent?</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-07-14T09:13:20Z2016-07-14T09:13:20Zmissing pages in site map<div><p>Sorry, I've been very busy lately didn't get to it on time, the
URL you had sent is no longer valid.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-07-14T18:16:14Z2016-07-14T18:16:18Zmissing pages in site map<div><p>It should not be invalid.<br>
Maybe I mis-typed it.<br>
Please try again: <a href="https://qvaranaldb01.qad.com:22011/qad-central">https://qvaranaldb01.qad.com:22011/qad-central</a><br>
The port is necessary.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-07-14T18:19:28Z2016-07-14T18:19:28Zmissing pages in site map<div><p>I'm getting:</p>
<pre>
<code>Firefox can't find the server at qvaranaldb01.qad.com.</code>
</pre></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-07-26T21:33:57Z2016-07-26T21:33:59Zmissing pages in site map<div><p>Finally figured out that I had sent the wrong URL.<br>
I resent a contact us message with the new URL and (possibly)
updated password.</p>
<p>Sorry for the slow follow up, too much multi tasking and context
switching.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-03T16:28:17Z2016-08-03T16:28:17Zmissing pages in site map<div><p>Can you check the <a href="http://downloads.arachni-scanner.com/nightlies/">nightlies</a>
please?</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-03T18:55:37Z2016-08-03T18:55:38Zmissing pages in site map<div><p>I tried to install the most recent nightly and I'm getting the
following:</p>
<p>ruby: error while loading shared libraries: [arachni instlal
path]/arachni/arachni-2.0dev-1.0dev/bin/../system/usr/lib/libruby.so.2.2:
file too short</p>
<p>I've confirmed the 403 forbidden error and will follow up with
deployment folks to figure out why that is.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-03T20:30:21Z2016-08-03T20:30:21Zmissing pages in site map<div><p>Do you have the right package for your achitecture?<br>
It's working fine for me.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-03T22:59:20Z2016-08-03T22:59:22Zmissing pages in site map<div><p>Nevermind... I extracted from windows and tried to run in linux.
The symbolic links were not correctly created as symbolic
links.<br>
When extracting the files from linux, I was able to run
correctly.</p>
<p>It's running now to see if I can get all the urls I expect in
the site map.<br>
Will report later.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-06T00:32:39Z2016-08-06T00:32:41Zmissing pages in site map<div><p>Unfortunately, it doesn't appear that the nightly (from aug 3)
help in my ability to get url that relate to the dynamically loaded
menus.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-11T18:27:20Z2016-08-11T18:27:27Zmissing pages in site map<div><p>Any luck accessing the application with the info I sent in the
last email?</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-13T07:57:01Z2016-08-13T07:57:01Zmissing pages in site map<div><p>Sorry, I was debugging something else, I'll try to get to this
as soon as possible.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-14T13:33:38Z2016-08-14T13:33:38Zmissing pages in site map<div><p>Unfortunately the issue is cause by an incompatibility in the
current browser engine.<br>
It will however get resolved once I upgrade to a more recent
version, but this will require some time.</p>
<p>You can track the progress on this task from: <a href="https://github.com/Arachni/arachni/issues/764">https://github.com/Arachni/arachni/issues/764</a></p>
<p>Sorry for the delay in looking into this.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-15T15:46:17Z2016-08-15T15:46:20Zmissing pages in site map<div><p>Thanks for looking into it. I'll be watching the progress on
that browser engine upgrade.</p>
<p>And the delay is as least partially my fault in communicating
the environment information.</p>
<p>I'm just glad to know it's not something I was doing wrong.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-08-31T00:37:49Z2016-08-31T00:37:49Zmissing pages in site map<div><p>Perhaps it's me, but I'm still unable to get all the paths I'd
expect.</p>
<p>Here is an example command line:<br></p>
<pre>
<code> arachni https://qvaranalwb01.qad.com/qad-central --checks=trainer --browser-cluster-wait-for-element='^.*qad-central/((?!resources).)*$:#WebShellMenu' --plugin=autologin:url=https://qvaranalwb01.qad.com/qad-central/resources/login.jsp,parameters="j_username=<provided in email>&j_password=<provided in email>",check="QAD Web UI" --scope-exclude-pattern '^.*qad-central/resources.*$'</code>
</pre>
<p>And it's not finding/scanning this page (example)<br>
<code>https://qvaranalwb01.qad.com/qad-central/#/view/analytics/dashboards/display?dashboardNbr=0</code>
(available from the default top menu) Also, the menu is dynamic.
When a different option is selected in the first menu item (not
<code><a></code> tags, but <code><li></code> tags with
click event in js/angular) the menus are changed and will contains
links such as:<br>
<code>https://qvaranalwb01.qad.com/qad-central/#/view/qracore/browses/list?browseId=urn:browse:fin:BDebtor.SelectDebtor</code>
Those are the real meat of the application that would need to be
scanned for vulnerability.</p>
<p>Back to the command line example above... I can see it trying to
wait for the the element and receiving a timeout. But in a browser,
this is blazing fast. Even when I increase the timeout value on the
browser cluster, it still times out.</p>
<p>Is there something wrong with my selector syntax?</p>
<p>I used a ruby login script to try to examine the
response:<br></p>
<pre>
<code>browser.goto "https://qvaranalwb01.qad.com/qad-central/resources/login.jsp"
sleep(5)
form = browser.form( id: 'loginForm' )
form.text_field( id: 'j_username' ).set '<provided in email>'
form.text_field( name: 'j_password' ).set '<provided in email>'
form.submit
sleep(5)
output = File.open( "response1.html","w" )
output << browser.html
output.close
framework.options.session.check_url = "https://qvaranalwb01.qad.com/qad-central/#/view/webshell/home"
framework.options.session.check_pattern = /.*QAD Web UI.*/</code>
</pre>
And I found that the div where I'd expect all the menus and
additional links/urls us empty<br>
<code><div
ng-include="'view/webshell/menu'"></div></code>
<p>Could this be another phantomJS incompatibility?</p>
<p>Sorry for the long message, but I figured more details is better
than not enough.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-09-01T13:20:44Z2016-09-01T13:20:44Zmissing pages in site map<div><p>I found the bug, it has something to do with Arachni's JS
taint-tracing subsystem, it somehow interferes with the page's JS
and causes an error and prevents the page from being rendered
properly.</p>
<p>I'll keep you updated on my progress.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-09-01T14:24:00Z2016-09-01T14:24:00Zmissing pages in site map<div><p>This is the function that causes the issue:</p>
<pre>
<code>function AnyFifoArray(init) {this._values=[];for(var x=0;x<init.length;x++)this._values.push(init[x].value);}</code>
</pre>
<p>Working now to debug it.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-09-01T15:33:33Z2016-09-01T15:33:33Zmissing pages in site map<div><p>I think it has to do with the fact that
<code>AnyFifoArray</code> has a "class" structure:</p>
<pre>
<code>var AnyFifoArray = (function () {
function AnyFifoArray(init) {
this._values = [];
for (var x = 0; x < init.length; x++) {
this._values.push(init[x].value);
}
}
AnyFifoArray.prototype.add = function (value) {
this._values.push(value);
};
AnyFifoArray.prototype.remove = function (value) {
var index = this._values.indexOf(value, 0);
this._values.splice(index, 1);
};
AnyFifoArray.prototype.values = function () {
return this._values;
};
AnyFifoArray.prototype.containsValue = function (value) {
for (var i = 0; i < this._values.length; i++) {
if (this._values[i] == value)
return true;
}
return false;
};
return AnyFifoArray;
}());</code>
</pre>
<p>Digging deeper.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-09-01T16:32:28Z2016-09-01T16:32:28Zmissing pages in site map<div><p>Pushing nightlies with the fix now, will let you know once
they're up.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-09-01T18:34:58Z2016-09-01T18:34:58Zmissing pages in site map<div><p>Nightlies are up, although you may need to rethink the following
option:</p>
<pre>
<code>--browser-cluster-wait-for-element='^.*qad-central/((?!resources).)*$:#WebShellMenu'</code>
</pre>
<p>I used <code>#kMenuUserInfo</code> in my testing.</p>
<p>Let me know how the nightlies do.</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-09-01T18:44:58Z2016-09-01T18:44:58Zmissing pages in site map<div><p>You, sir, are awesome!<br>
Looks like providing so much details paid off.<br>
Downloading the nightly now (slow for me for some reason) and I
will test a few wait-for-element options.<br>
I'll report back later.</p></div>Peter-Dave Sheehantag:support.arachni-scanner.com,2012-07-01:Comment/401916872016-09-01T18:47:26Z2016-09-01T18:47:26Zmissing pages in site map<div><p>You may need to tweak the option further, I don't think it's
sufficient for a full scan.<br>
Some screenshots I took didn't have the full menu rendered and you
need a conf that will let the page render as much as possible.</p></div>Tasos Laskos