tag:support.arachni-scanner.com,2012-07-01:/discussions/problems/4051-arachni-nightly-scans-infinitlyArachni: Discussion 2016-09-22T09:13:59Ztag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-07T16:08:31Z2016-07-07T19:13:49ZArachni nightly scans infinitly<div><p>Hi,<br>
I am testing a commercial application, so unfortunatelly I can not
give much information.<br>
I wanted to do only crawling so I used --checks='-*', I do not know
if this is allowed or if this causes my problem.<br>
My problem is simple - Arachni never finishes, there are still 2
jobs runninig and debug output shows following:</p>
<pre>
<code>[2016-07-07 18:00:45 +0200 - 142.7] [!!!] [http/proxy_server/connection#on_close:195] Connection: Closed because: [Arachni::Reactor::Connection::Error::Closed] end of file reached</code>
</pre>
<p>I used login_script to authenticate, the application is written
in Java. I am using latest nightly on Windows.<br>
Feel free to ask if you need more information. Thank you very
much.</p></div>Vojta Polasektag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-07T19:19:26Z2016-07-07T19:19:26ZArachni nightly scans infinitly<div><p>Hello,</p>
<p>Unfortunately with these types of problems I need access to the
web application to find the issue.</p>
<p>You could try sending me the debugging output for the entire
scan, although that could result in a few GBs of text.</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-12T09:35:30Z2016-07-12T09:35:30ZArachni nightly scans infinitly<div><p>Well, I found an interesting thing. When Arachni runs on
Windows, the mentioned problem happens. But when it runs on Linux,
it does not. In both cases I download nightly package from your
site. Can there be any difference between Windows and Linux package
which causes this?</p></div>Vojta POlasektag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-14T09:11:00Z2016-07-14T09:11:00ZArachni nightly scans infinitly<div><p>There're could be a networking issue, different OS handle
sockets a bit differently.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-26T08:29:51Z2016-07-26T08:29:53ZArachni nightly scans infinitly<div><p>Hi,<br>
do jobs have any kind of unique numbering? I would like to find
which jobs stay hanging. I searched through the debug output but I
am quite confused by the numbering.</p></div>Vojta Polasektag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-26T08:31:32Z2016-07-26T08:31:32ZArachni nightly scans infinitly<div><p>What do you mean by jobs? Browser jobs?</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-26T10:41:33Z2016-07-26T10:41:34ZArachni nightly scans infinitly<div><p>Yes, I think so. After a long time, it shows:<br>
~Browser cluster: Pending jobs 2 I need to identify those jobs.</p></div>Vojta Polasektag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-26T10:52:28Z2016-07-26T10:52:28ZArachni nightly scans infinitly<div><p>Yeah they can be hard to identify.<br>
The best you can do is enable <code>--output-debug</code> and
identify them by output such as:</p>
<pre>
<code>[2016-07-26 13:47:08 +0300 - 0.0] [!] [browser_cluster/worker#run_job:71] BrowserCluster Worker#15890740: Started: #<Arachni::BrowserCluster::Jobs::DOMExploration::EventTrigger:15886980 @resource=#<Arachni::Page::DOM:18473300 @url="http://testhtml5.vulnweb.com/#/contact" @transitions=6 @data_flow_sinks=0 @execution_flow_sinks=0> @event=:focus @element=<input type="text" class="span8" placeholder="Your Last Name" name="lastName"> time= timed_out=false></code>
</pre>
<p>The ID such as
<code>Arachni::BrowserCluster::Jobs::DOMExploration::EventTrigger:15886980</code>
uniquely identify each job.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-27T08:35:46Z2016-07-27T08:35:49ZArachni nightly scans infinitly<div><p>Hi,<br>
thanks, this helped me to filter possible states of jobs. I have
found 4 states:<br>
Queued, Got job result, Started, Finished.<br>
What is please the correct order of these states?<br>
Thanks.</p></div>Vojta Polasektag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-27T08:38:48Z2016-07-27T08:38:48ZArachni nightly scans infinitly<div><ol>
<li>Queued<br></li>
<li>Started<br>
<ol>
<li>Got job result -- optional</li>
</ol>
</li>
<li>Finished</li>
</ol></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-29T11:25:26Z2016-07-29T11:25:27ZArachni nightly scans infinitly<div><p>Hi,<br>
I am quite confused. I wrote a small script for parsing of debug
output generated by Arachni and I watched states of jobs.<br>
Is it possible that a job is started and finished several times
during the scan?<br>
Is it possible that a job does not go through the Queued state and
is in Started state straight away?<br>
Unfortunately the hanging thing appeared also on Linux, at least on
my Ubuntu 14.04 VM.</p></div>Vojta Polasektag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-07-29T11:36:46Z2016-07-29T11:36:46ZArachni nightly scans infinitly<div><ol>
<li>There are some special jobs that can do that yes.<br></li>
<li>No, all jobs are queued first.</li>
</ol></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-08-02T16:15:20Z2016-08-02T16:15:20ZArachni nightly scans infinitly<div><p>I'm pushing nightlies with a plugin that will help debug
this.<br>
Did you by any chance have the <code>metrics</code> plugin enabled
for your scans?</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-08-02T17:12:19Z2016-08-02T17:12:19ZArachni nightly scans infinitly<div><p>You can now enable the <code>browser_cluster_job_monitor</code>
plugin to monitor active jobs and their active HTTP
connections.</p>
<p>Enable it like so:</p>
<pre>
<code>--plugin=browser_cluster_job_monitor:logfile=/tmp/browser_cluster_job_monitor.log</code>
</pre>
<p>Monitor it like so:</p>
<pre>
<code>watch -n1 cat /tmp/browser_cluster_job_monitor.log</code>
</pre></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-08-03T11:38:39Z2016-08-03T11:38:41ZArachni nightly scans infinitly<div><p>Hi,<br>
I really appreciate your work and I hope it will help with my
problem.<br>
I tried it. My only enabled check is trainer, because for now I
only want to map the application. I use 10 browsers with
concurrency set to 200. Unfortunately I can see only 1 active job
and it does not seem to change. It shows the same thing for long
time.<br>
I wanted to avoid the trainer, because it somehow logs me out of
the application. Do I understand it right that it is necessary to
specify at least one check to get everything running properly?
(vector_collector for example)<br>
I tried it also with --checks='-*' and it did the same.<br>
I tried it also on bodge IT store with --checks='sql*' and in this
case it showed multiple jobs with changing information.<br>
I enabled the metrics plugin now, but I did not use it before. I
used only autothrottle and login_script.<br>
I will let Arachni to finish (hopefuly) the scan with trainer
enabled, but I am not sure if those very frequent logouts won't
interfere with scan results.</p></div>Vojta Polasektag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-08-03T12:14:05Z2016-08-03T12:14:05ZArachni nightly scans infinitly<div><p>For a second I thought the <code>metrics</code> plugin was the
problem but it's not.<br>
There's another user with the same issue and I've been granted
access to the webapp but unfortunately I can't reproduce it.</p>
<p>Since he's more open about his setup I'll keep working with him
to debug the issue and keep you updated with my progress.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-08-06T08:27:13Z2016-08-06T08:27:13ZArachni nightly scans infinitly<div><p><a href="http://downloads.arachni-scanner.com/nightlies/">Nightlies</a> are
up and should include the fix.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-08-19T09:57:02Z2016-08-19T09:57:02ZArachni nightly scans infinitly<div><p>Re-opening as per your e-mail.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/402767462016-08-26T09:07:29Z2016-08-26T09:07:29ZArachni nightly scans infinitly<div><p>Please try the nightlies again.</p></div>Tasos Laskos