Arachni nightly scans infinitly

Vojta Polasek's Avatar

Vojta Polasek

07 Jul, 2016 04:08 PM

Hi,
I am testing a commercial application, so unfortunatelly I can not give much information.
I wanted to do only crawling so I used --checks='-*', I do not know if this is allowed or if this causes my problem.
My problem is simple - Arachni never finishes, there are still 2 jobs runninig and debug output shows following:

[2016-07-07 18:00:45 +0200 - 142.7] [!!!] [http/proxy_server/connection#on_close:195] Connection: Closed because: [Arachni::Reactor::Connection::Error::Closed] end of file reached

I used login_script to authenticate, the application is written in Java. I am using latest nightly on Windows.
Feel free to ask if you need more information. Thank you very much.

  1. Support Staff 1 Posted by Tasos Laskos on 07 Jul, 2016 07:19 PM

    Tasos Laskos's Avatar

    Hello,

    Unfortunately with these types of problems I need access to the web application to find the issue.

    You could try sending me the debugging output for the entire scan, although that could result in a few GBs of text.

    Cheers

  2. 2 Posted by Vojta POlasek on 12 Jul, 2016 09:35 AM

    Vojta POlasek's Avatar

    Well, I found an interesting thing. When Arachni runs on Windows, the mentioned problem happens. But when it runs on Linux, it does not. In both cases I download nightly package from your site. Can there be any difference between Windows and Linux package which causes this?

  3. Support Staff 3 Posted by Tasos Laskos on 14 Jul, 2016 09:11 AM

    Tasos Laskos's Avatar

    There're could be a networking issue, different OS handle sockets a bit differently.

  4. 4 Posted by Vojta Polasek on 26 Jul, 2016 08:29 AM

    Vojta Polasek's Avatar

    Hi,
    do jobs have any kind of unique numbering? I would like to find which jobs stay hanging. I searched through the debug output but I am quite confused by the numbering.

  5. Support Staff 5 Posted by Tasos Laskos on 26 Jul, 2016 08:31 AM

    Tasos Laskos's Avatar

    What do you mean by jobs? Browser jobs?

  6. 6 Posted by Vojta Polasek on 26 Jul, 2016 10:41 AM

    Vojta Polasek's Avatar

    Yes, I think so. After a long time, it shows:
    ~Browser cluster: Pending jobs 2 I need to identify those jobs.

  7. Support Staff 7 Posted by Tasos Laskos on 26 Jul, 2016 10:52 AM

    Tasos Laskos's Avatar

    Yeah they can be hard to identify.
    The best you can do is enable --output-debug and identify them by output such as:

    [2016-07-26 13:47:08 +0300 - 0.0] [!] [browser_cluster/worker#run_job:71] BrowserCluster Worker#15890740: Started: #<Arachni::BrowserCluster::Jobs::DOMExploration::EventTrigger:15886980 @resource=#<Arachni::Page::DOM:18473300 @url="http://testhtml5.vulnweb.com/#/contact" @transitions=6 @data_flow_sinks=0 @execution_flow_sinks=0> @event=:focus @element=<input type="text" class="span8" placeholder="Your Last Name" name="lastName"> time= timed_out=false>
    

    The ID such as Arachni::BrowserCluster::Jobs::DOMExploration::EventTrigger:15886980 uniquely identify each job.

  8. 8 Posted by Vojta Polasek on 27 Jul, 2016 08:35 AM

    Vojta Polasek's Avatar

    Hi,
    thanks, this helped me to filter possible states of jobs. I have found 4 states:
    Queued, Got job result, Started, Finished.
    What is please the correct order of these states?
    Thanks.

  9. Support Staff 9 Posted by Tasos Laskos on 27 Jul, 2016 08:38 AM

    Tasos Laskos's Avatar
    1. Queued
    2. Started
      1. Got job result -- optional
    3. Finished
  10. 10 Posted by Vojta Polasek on 29 Jul, 2016 11:25 AM

    Vojta Polasek's Avatar

    Hi,
    I am quite confused. I wrote a small script for parsing of debug output generated by Arachni and I watched states of jobs.
    Is it possible that a job is started and finished several times during the scan?
    Is it possible that a job does not go through the Queued state and is in Started state straight away?
    Unfortunately the hanging thing appeared also on Linux, at least on my Ubuntu 14.04 VM.

  11. Support Staff 11 Posted by Tasos Laskos on 29 Jul, 2016 11:36 AM

    Tasos Laskos's Avatar
    1. There are some special jobs that can do that yes.
    2. No, all jobs are queued first.
  12. Support Staff 12 Posted by Tasos Laskos on 02 Aug, 2016 04:15 PM

    Tasos Laskos's Avatar

    I'm pushing nightlies with a plugin that will help debug this.
    Did you by any chance have the metrics plugin enabled for your scans?

  13. Support Staff 13 Posted by Tasos Laskos on 02 Aug, 2016 05:12 PM

    Tasos Laskos's Avatar

    You can now enable the browser_cluster_job_monitor plugin to monitor active jobs and their active HTTP connections.

    Enable it like so:

    --plugin=browser_cluster_job_monitor:logfile=/tmp/browser_cluster_job_monitor.log
    

    Monitor it like so:

    watch -n1 cat /tmp/browser_cluster_job_monitor.log
    
  14. 14 Posted by Vojta Polasek on 03 Aug, 2016 11:38 AM

    Vojta Polasek's Avatar

    Hi,
    I really appreciate your work and I hope it will help with my problem.
    I tried it. My only enabled check is trainer, because for now I only want to map the application. I use 10 browsers with concurrency set to 200. Unfortunately I can see only 1 active job and it does not seem to change. It shows the same thing for long time.
    I wanted to avoid the trainer, because it somehow logs me out of the application. Do I understand it right that it is necessary to specify at least one check to get everything running properly? (vector_collector for example)
    I tried it also with --checks='-*' and it did the same.
    I tried it also on bodge IT store with --checks='sql*' and in this case it showed multiple jobs with changing information.
    I enabled the metrics plugin now, but I did not use it before. I used only autothrottle and login_script.
    I will let Arachni to finish (hopefuly) the scan with trainer enabled, but I am not sure if those very frequent logouts won't interfere with scan results.

  15. Support Staff 15 Posted by Tasos Laskos on 03 Aug, 2016 12:14 PM

    Tasos Laskos's Avatar

    For a second I thought the metrics plugin was the problem but it's not.
    There's another user with the same issue and I've been granted access to the webapp but unfortunately I can't reproduce it.

    Since he's more open about his setup I'll keep working with him to debug the issue and keep you updated with my progress.

  16. Support Staff 16 Posted by Tasos Laskos on 06 Aug, 2016 08:27 AM

    Tasos Laskos's Avatar

    Nightlies are up and should include the fix.

  17. Tasos Laskos closed this discussion on 08 Aug, 2016 09:57 AM.

  18. Tasos Laskos re-opened this discussion on 19 Aug, 2016 09:57 AM

  19. Support Staff 17 Posted by Tasos Laskos on 19 Aug, 2016 09:57 AM

    Tasos Laskos's Avatar

    Re-opening as per your e-mail.

  20. Support Staff 18 Posted by Tasos Laskos on 26 Aug, 2016 09:07 AM

    Tasos Laskos's Avatar

    Please try the nightlies again.

  21. Tasos Laskos closed this discussion on 22 Sep, 2016 09:13 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac