Home → Problems →

missing pages in site map

• Edit

Peter-Dave Sheehan

24 Jun, 2016 08:10 PM

We're just starting to explore using arachni for our new web app.
I've been able to get arachni to process the login using the autologin.
But it seems that the crawler doesn't find the links in our dynamically loaded menus (angularJS ng-include).
The url would be in the form of https://host.com/appname/#/view/qraview/hybridbrowse?viewMetaUri=ur...

I tried setting up the --browser-cluster-wait_for_elements option. And I can see places in the log where it appears to wait. But those sitemap elements are still not found.

I can get a full scan by including the sitemap detected by ZAP in the --scope-extend-path option, but I'd prefer to have those elements found automatically.

Any thoughts of something else to try?

I can get you access to the app. Send me an email and I will send the details.

Thanks

1. Support Staff 1 Posted by Tasos Laskos on 29 Jun, 2016 10:43 AM

   

   Access to the webapp with a specific test case would be much appreciated, that'd help me diagnose the issue.

   Cheers

   PS. Sorry for the late response.

   • Edit

2. 2 Posted by Peter-Dave Shee... on 29 Jun, 2016 04:12 PM

   

   Sent some additional details in the contact form since it includes private information.
   Thanks

   • Edit

3. Support Staff 3 Posted by Tasos Laskos on 29 Jun, 2016 04:16 PM

   

   Got it, thanks. :)

   • Edit

4. 4 Posted by Peter-Dave Shee... on 13 Jul, 2016 06:41 PM

   

   Any luck? or issue accessing the app using the info I sent?

   • Edit

5. Support Staff 5 Posted by Tasos Laskos on 14 Jul, 2016 09:13 AM

   

   Sorry, I've been very busy lately didn't get to it on time, the URL you had sent is no longer valid.

   • Edit

6. 6 Posted by Peter-Dave Shee... on 14 Jul, 2016 06:16 PM

   

   It should not be invalid.
   Maybe I mis-typed it.
   Please try again: https://qvaranaldb01.qad.com:22011/qad-central
   The port is necessary.

   • Edit

7. Support Staff 7 Posted by Tasos Laskos on 14 Jul, 2016 06:19 PM

   

   I'm getting:

   Firefox can't find the server at qvaranaldb01.qad.com.
   

   • Edit

8. 8 Posted by Peter-Dave Shee... on 26 Jul, 2016 09:33 PM

   

   Finally figured out that I had sent the wrong URL.
   I resent a contact us message with the new URL and (possibly) updated password.

   Sorry for the slow follow up, too much multi tasking and context switching.

   • Edit

9. Support Staff 9 Posted by Tasos Laskos on 03 Aug, 2016 04:28 PM

   

   Can you check the nightlies please?

   • Edit

10. 10 Posted by Peter-Dave Shee... on 03 Aug, 2016 06:55 PM

    

    I tried to install the most recent nightly and I'm getting the following:

    ruby: error while loading shared libraries: [arachni instlal path]/arachni/arachni-2.0dev-1.0dev/bin/../system/usr/lib/libruby.so.2.2: file too short

    I've confirmed the 403 forbidden error and will follow up with deployment folks to figure out why that is.

    • Edit

11. Support Staff 11 Posted by Tasos Laskos on 03 Aug, 2016 08:30 PM

    

    Do you have the right package for your achitecture?
    It's working fine for me.

    • Edit

12. 12 Posted by Peter-Dave Shee... on 03 Aug, 2016 10:59 PM

    

    Nevermind... I extracted from windows and tried to run in linux. The symbolic links were not correctly created as symbolic links.
    When extracting the files from linux, I was able to run correctly.

    It's running now to see if I can get all the urls I expect in the site map.
    Will report later.

    • Edit

13. 13 Posted by Peter-Dave Shee... on 06 Aug, 2016 12:32 AM

    

    Unfortunately, it doesn't appear that the nightly (from aug 3) help in my ability to get url that relate to the dynamically loaded menus.

    • Edit

14. 14 Posted by Peter-Dave Shee... on 11 Aug, 2016 06:27 PM

    

    Any luck accessing the application with the info I sent in the last email?

    • Edit

15. Support Staff 15 Posted by Tasos Laskos on 13 Aug, 2016 07:57 AM

    

    Sorry, I was debugging something else, I'll try to get to this as soon as possible.

    • Edit

16. Support Staff 16 Posted by Tasos Laskos on 14 Aug, 2016 01:33 PM

    

    Unfortunately the issue is cause by an incompatibility in the current browser engine.
    It will however get resolved once I upgrade to a more recent version, but this will require some time.

    You can track the progress on this task from: https://github.com/Arachni/arachni/issues/764

    Sorry for the delay in looking into this.

    • Edit

17. Tasos Laskos closed this discussion on 14 Aug, 2016 01:33 PM.

18. Peter-Dave Sheehan re-opened this discussion on 15 Aug, 2016 03:46 PM

19. 17 Posted by Peter-Dave Shee... on 15 Aug, 2016 03:46 PM

    

    Thanks for looking into it. I'll be watching the progress on that browser engine upgrade.

    And the delay is as least partially my fault in communicating the environment information.

    I'm just glad to know it's not something I was doing wrong.

    • Edit

20. Tasos Laskos closed this discussion on 16 Aug, 2016 09:11 AM.

21. Peter-Dave Sheehan re-opened this discussion on 31 Aug, 2016 12:37 AM

22. 18 Posted by Peter-Dave Shee... on 31 Aug, 2016 12:37 AM

    

    Perhaps it's me, but I'm still unable to get all the paths I'd expect.

    Here is an example command line:
    

     arachni https://qvaranalwb01.qad.com/qad-central --checks=trainer --browser-cluster-wait-for-element='^.*qad-central/((?!resources).)*$:#WebShellMenu' --plugin=autologin:url=https://qvaranalwb01.qad.com/qad-central/resources/login.jsp,parameters="j_username=<provided in email>&j_password=<provided in email>",check="QAD Web UI" --scope-exclude-pattern '^.*qad-central/resources.*$'
    

    And it's not finding/scanning this page (example)
    https://qvaranalwb01.qad.com/qad-central/#/view/analytics/dashboards/display?dashboardNbr=0 (available from the default top menu) Also, the menu is dynamic. When a different option is selected in the first menu item (not <a> tags, but <li> tags with click event in js/angular) the menus are changed and will contains links such as:
    https://qvaranalwb01.qad.com/qad-central/#/view/qracore/browses/list?browseId=urn:browse:fin:BDebtor.SelectDebtor Those are the real meat of the application that would need to be scanned for vulnerability.

    Back to the command line example above... I can see it trying to wait for the the element and receiving a timeout. But in a browser, this is blazing fast. Even when I increase the timeout value on the browser cluster, it still times out.

    Is there something wrong with my selector syntax?

    I used a ruby login script to try to examine the response:
    

    browser.goto "https://qvaranalwb01.qad.com/qad-central/resources/login.jsp"
    sleep(5)
    form = browser.form( id: 'loginForm' )
    form.text_field( id: 'j_username' ).set '<provided in email>'
    form.text_field( name: 'j_password' ).set '<provided in email>'
    form.submit
    sleep(5)
    output = File.open( "response1.html","w" )
    output << browser.html
    output.close
    framework.options.session.check_url     = "https://qvaranalwb01.qad.com/qad-central/#/view/webshell/home"
    framework.options.session.check_pattern = /.*QAD Web UI.*/
    

    And I found that the div where I'd expect all the menus and additional links/urls us empty
    <div ng-include="'view/webshell/menu'"></div>

    Could this be another phantomJS incompatibility?

    Sorry for the long message, but I figured more details is better than not enough.

    • Edit

23. Support Staff 19 Posted by Tasos Laskos on 01 Sep, 2016 01:20 PM

    

    I found the bug, it has something to do with Arachni's JS taint-tracing subsystem, it somehow interferes with the page's JS and causes an error and prevents the page from being rendered properly.

    I'll keep you updated on my progress.

    • Edit

24. Support Staff 20 Posted by Tasos Laskos on 01 Sep, 2016 02:24 PM

    

    This is the function that causes the issue:

    function AnyFifoArray(init) {this._values=[];for(var x=0;x<init.length;x++)this._values.push(init[x].value);}
    

    Working now to debug it.

    • Edit

25. Support Staff 21 Posted by Tasos Laskos on 01 Sep, 2016 03:33 PM

    

    I think it has to do with the fact that AnyFifoArray has a "class" structure:

    var AnyFifoArray = (function () {
        function AnyFifoArray(init) {
            this._values = [];
            for (var x = 0; x < init.length; x++) {
                this._values.push(init[x].value);
            }
        }
        AnyFifoArray.prototype.add = function (value) {
            this._values.push(value);
        };
        AnyFifoArray.prototype.remove = function (value) {
            var index = this._values.indexOf(value, 0);
            this._values.splice(index, 1);
        };
        AnyFifoArray.prototype.values = function () {
            return this._values;
        };
        AnyFifoArray.prototype.containsValue = function (value) {
            for (var i = 0; i < this._values.length; i++) {
                if (this._values[i] == value)
                    return true;
            }
            return false;
        };
        return AnyFifoArray;
    }());
    

    Digging deeper.

    • Edit

26. Support Staff 22 Posted by Tasos Laskos on 01 Sep, 2016 04:32 PM

    

    Pushing nightlies with the fix now, will let you know once they're up.

    • Edit

27. Support Staff 23 Posted by Tasos Laskos on 01 Sep, 2016 06:34 PM

    

    Nightlies are up, although you may need to rethink the following option:

    --browser-cluster-wait-for-element='^.*qad-central/((?!resources).)*$:#WebShellMenu'
    

    I used #kMenuUserInfo in my testing.

    Let me know how the nightlies do.

    Cheers

    • Edit

28. 24 Posted by Peter-Dave Shee... on 01 Sep, 2016 06:44 PM

    

    You, sir, are awesome!
    Looks like providing so much details paid off.
    Downloading the nightly now (slow for me for some reason) and I will test a few wait-for-element options.
    I'll report back later.

    • Edit

29. Support Staff 25 Posted by Tasos Laskos on 01 Sep, 2016 06:47 PM

    

    You may need to tweak the option further, I don't think it's sufficient for a full scan.
    Some screenshots I took didn't have the full menu rendered and you need a conf that will let the page render as much as possible.

    • Edit

30. Tasos Laskos closed this discussion on 22 Sep, 2016 09:14 AM.