missing pages in site map

Peter-Dave Sheehan's Avatar

Peter-Dave Sheehan

24 Jun, 2016 08:10 PM

We're just starting to explore using arachni for our new web app.
I've been able to get arachni to process the login using the autologin.
But it seems that the crawler doesn't find the links in our dynamically loaded menus (angularJS ng-include).
The url would be in the form of https://host.com/appname/#/view/qraview/hybridbrowse?viewMetaUri=ur...

I tried setting up the --browser-cluster-wait_for_elements option. And I can see places in the log where it appears to wait. But those sitemap elements are still not found.

I can get a full scan by including the sitemap detected by ZAP in the --scope-extend-path option, but I'd prefer to have those elements found automatically.

Any thoughts of something else to try?

I can get you access to the app. Send me an email and I will send the details.


  1. Support Staff 1 Posted by Tasos Laskos on 29 Jun, 2016 10:43 AM

    Tasos Laskos's Avatar

    Access to the webapp with a specific test case would be much appreciated, that'd help me diagnose the issue.


    PS. Sorry for the late response.

  2. 2 Posted by Peter-Dave Shee... on 29 Jun, 2016 04:12 PM

    Peter-Dave Sheehan's Avatar

    Sent some additional details in the contact form since it includes private information.

  3. Support Staff 3 Posted by Tasos Laskos on 29 Jun, 2016 04:16 PM

    Tasos Laskos's Avatar

    Got it, thanks. :)

  4. 4 Posted by Peter-Dave Shee... on 13 Jul, 2016 06:41 PM

    Peter-Dave Sheehan's Avatar

    Any luck? or issue accessing the app using the info I sent?

  5. Support Staff 5 Posted by Tasos Laskos on 14 Jul, 2016 09:13 AM

    Tasos Laskos's Avatar

    Sorry, I've been very busy lately didn't get to it on time, the URL you had sent is no longer valid.

  6. 6 Posted by Peter-Dave Shee... on 14 Jul, 2016 06:16 PM

    Peter-Dave Sheehan's Avatar

    It should not be invalid.
    Maybe I mis-typed it.
    Please try again: https://qvaranaldb01.qad.com:22011/qad-central
    The port is necessary.

  7. Support Staff 7 Posted by Tasos Laskos on 14 Jul, 2016 06:19 PM

    Tasos Laskos's Avatar

    I'm getting:

    Firefox can't find the server at qvaranaldb01.qad.com.
  8. 8 Posted by Peter-Dave Shee... on 26 Jul, 2016 09:33 PM

    Peter-Dave Sheehan's Avatar

    Finally figured out that I had sent the wrong URL.
    I resent a contact us message with the new URL and (possibly) updated password.

    Sorry for the slow follow up, too much multi tasking and context switching.

  9. Support Staff 9 Posted by Tasos Laskos on 03 Aug, 2016 04:28 PM

    Tasos Laskos's Avatar

    Can you check the nightlies please?

  10. 10 Posted by Peter-Dave Shee... on 03 Aug, 2016 06:55 PM

    Peter-Dave Sheehan's Avatar

    I tried to install the most recent nightly and I'm getting the following:

    ruby: error while loading shared libraries: [arachni instlal path]/arachni/arachni-2.0dev-1.0dev/bin/../system/usr/lib/libruby.so.2.2: file too short

    I've confirmed the 403 forbidden error and will follow up with deployment folks to figure out why that is.

  11. Support Staff 11 Posted by Tasos Laskos on 03 Aug, 2016 08:30 PM

    Tasos Laskos's Avatar

    Do you have the right package for your achitecture?
    It's working fine for me.

  12. 12 Posted by Peter-Dave Shee... on 03 Aug, 2016 10:59 PM

    Peter-Dave Sheehan's Avatar

    Nevermind... I extracted from windows and tried to run in linux. The symbolic links were not correctly created as symbolic links.
    When extracting the files from linux, I was able to run correctly.

    It's running now to see if I can get all the urls I expect in the site map.
    Will report later.

  13. 13 Posted by Peter-Dave Shee... on 06 Aug, 2016 12:32 AM

    Peter-Dave Sheehan's Avatar

    Unfortunately, it doesn't appear that the nightly (from aug 3) help in my ability to get url that relate to the dynamically loaded menus.

  14. 14 Posted by Peter-Dave Shee... on 11 Aug, 2016 06:27 PM

    Peter-Dave Sheehan's Avatar

    Any luck accessing the application with the info I sent in the last email?

  15. Support Staff 15 Posted by Tasos Laskos on 13 Aug, 2016 07:57 AM

    Tasos Laskos's Avatar

    Sorry, I was debugging something else, I'll try to get to this as soon as possible.

  16. Support Staff 16 Posted by Tasos Laskos on 14 Aug, 2016 01:33 PM

    Tasos Laskos's Avatar

    Unfortunately the issue is cause by an incompatibility in the current browser engine.
    It will however get resolved once I upgrade to a more recent version, but this will require some time.

    You can track the progress on this task from: https://github.com/Arachni/arachni/issues/764

    Sorry for the delay in looking into this.

  17. Tasos Laskos closed this discussion on 14 Aug, 2016 01:33 PM.

  18. Peter-Dave Sheehan re-opened this discussion on 15 Aug, 2016 03:46 PM

  19. 17 Posted by Peter-Dave Shee... on 15 Aug, 2016 03:46 PM

    Peter-Dave Sheehan's Avatar

    Thanks for looking into it. I'll be watching the progress on that browser engine upgrade.

    And the delay is as least partially my fault in communicating the environment information.

    I'm just glad to know it's not something I was doing wrong.

  20. Tasos Laskos closed this discussion on 16 Aug, 2016 09:11 AM.

  21. Peter-Dave Sheehan re-opened this discussion on 31 Aug, 2016 12:37 AM

  22. 18 Posted by Peter-Dave Shee... on 31 Aug, 2016 12:37 AM

    Peter-Dave Sheehan's Avatar

    Perhaps it's me, but I'm still unable to get all the paths I'd expect.

    Here is an example command line:

     arachni https://qvaranalwb01.qad.com/qad-central --checks=trainer --browser-cluster-wait-for-element='^.*qad-central/((?!resources).)*$:#WebShellMenu' --plugin=autologin:url=https://qvaranalwb01.qad.com/qad-central/resources/login.jsp,parameters="j_username=<provided in email>&j_password=<provided in email>",check="QAD Web UI" --scope-exclude-pattern '^.*qad-central/resources.*$'

    And it's not finding/scanning this page (example)
    https://qvaranalwb01.qad.com/qad-central/#/view/analytics/dashboards/display?dashboardNbr=0 (available from the default top menu) Also, the menu is dynamic. When a different option is selected in the first menu item (not <a> tags, but <li> tags with click event in js/angular) the menus are changed and will contains links such as:
    https://qvaranalwb01.qad.com/qad-central/#/view/qracore/browses/list?browseId=urn:browse:fin:BDebtor.SelectDebtor Those are the real meat of the application that would need to be scanned for vulnerability.

    Back to the command line example above... I can see it trying to wait for the the element and receiving a timeout. But in a browser, this is blazing fast. Even when I increase the timeout value on the browser cluster, it still times out.

    Is there something wrong with my selector syntax?

    I used a ruby login script to try to examine the response:

    browser.goto "https://qvaranalwb01.qad.com/qad-central/resources/login.jsp"
    form = browser.form( id: 'loginForm' )
    form.text_field( id: 'j_username' ).set '<provided in email>'
    form.text_field( name: 'j_password' ).set '<provided in email>'
    output = File.open( "response1.html","w" )
    output << browser.html
    framework.options.session.check_url     = "https://qvaranalwb01.qad.com/qad-central/#/view/webshell/home"
    framework.options.session.check_pattern = /.*QAD Web UI.*/
    And I found that the div where I'd expect all the menus and additional links/urls us empty
    <div ng-include="'view/webshell/menu'"></div>

    Could this be another phantomJS incompatibility?

    Sorry for the long message, but I figured more details is better than not enough.

  23. Support Staff 19 Posted by Tasos Laskos on 01 Sep, 2016 01:20 PM

    Tasos Laskos's Avatar

    I found the bug, it has something to do with Arachni's JS taint-tracing subsystem, it somehow interferes with the page's JS and causes an error and prevents the page from being rendered properly.

    I'll keep you updated on my progress.

  24. Support Staff 20 Posted by Tasos Laskos on 01 Sep, 2016 02:24 PM

    Tasos Laskos's Avatar

    This is the function that causes the issue:

    function AnyFifoArray(init) {this._values=[];for(var x=0;x<init.length;x++)this._values.push(init[x].value);}

    Working now to debug it.

  25. Support Staff 21 Posted by Tasos Laskos on 01 Sep, 2016 03:33 PM

    Tasos Laskos's Avatar

    I think it has to do with the fact that AnyFifoArray has a "class" structure:

    var AnyFifoArray = (function () {
        function AnyFifoArray(init) {
            this._values = [];
            for (var x = 0; x < init.length; x++) {
        AnyFifoArray.prototype.add = function (value) {
        AnyFifoArray.prototype.remove = function (value) {
            var index = this._values.indexOf(value, 0);
            this._values.splice(index, 1);
        AnyFifoArray.prototype.values = function () {
            return this._values;
        AnyFifoArray.prototype.containsValue = function (value) {
            for (var i = 0; i < this._values.length; i++) {
                if (this._values[i] == value)
                    return true;
            return false;
        return AnyFifoArray;

    Digging deeper.

  26. Support Staff 22 Posted by Tasos Laskos on 01 Sep, 2016 04:32 PM

    Tasos Laskos's Avatar

    Pushing nightlies with the fix now, will let you know once they're up.

  27. Support Staff 23 Posted by Tasos Laskos on 01 Sep, 2016 06:34 PM

    Tasos Laskos's Avatar

    Nightlies are up, although you may need to rethink the following option:


    I used #kMenuUserInfo in my testing.

    Let me know how the nightlies do.


  28. 24 Posted by Peter-Dave Shee... on 01 Sep, 2016 06:44 PM

    Peter-Dave Sheehan's Avatar

    You, sir, are awesome!
    Looks like providing so much details paid off.
    Downloading the nightly now (slow for me for some reason) and I will test a few wait-for-element options.
    I'll report back later.

  29. Support Staff 25 Posted by Tasos Laskos on 01 Sep, 2016 06:47 PM

    Tasos Laskos's Avatar

    You may need to tweak the option further, I don't think it's sufficient for a full scan.
    Some screenshots I took didn't have the full menu rendered and you need a conf that will let the page render as much as possible.

  30. Tasos Laskos closed this discussion on 22 Sep, 2016 09:14 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac