tag:support.arachni-scanner.com,2012-07-01:/discussions/problems/3970-arachni-seems-hangingArachni: Discussion 2016-04-14T11:59:29Ztag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-03T14:09:24Z2016-04-03T14:09:24ZArachni seems hanging<div><p>2GB of RAM isn't actually all that much for that site. Just the
size of the string objects involved can eat it up, some of the
pages are quite large and when you process them that size gets
multiplied by a lot.</p>
<p>About the distributed stuff, it doesn't distribute the crawl,
only the audit so what you're seeing is expected.</p>
<p>Since I fixed this I decided to also profile the system with
that site, hoping that there would be a memory leak I can fix to
reduce the RAM but no dice, so far everything is operating
properly.</p>
<p>Btw, v1.3.2 did have leaks which were fixed in v1.4, so you
can't really compare the 2, the browser parts of the system were
seriously overhauled.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-03T14:47:38Z2016-04-03T14:47:38ZArachni seems hanging<div><p>Fixed: <a href="https://github.com/Arachni/arachni/commit/8a9c8bbc36367e0a461cc9a047a274b3fe47b8a7">
https://github.com/Arachni/arachni/commit/8a9c8bbc36367e0a461cc9a04...</a></p>
<p>Will let you know once the nightlies are up.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-03T15:55:03Z2016-04-03T15:55:04ZArachni seems hanging<div><p>Heeyy... nice job, I'm looking forward to try it out.</p>
<p>Is there any way I can limit the ram consumption on the scan
server ?<br>
Maybe reduce --browser-cluster-pool-size or
--browser-cluster-worker-time-to-live<br>
Or even better: make it fail in a graceful non-fatal way ?</p>
<p>I'll get back you asap with the results of the test of most
recent nightly.</p>
<p>Best Regards<br>
Dave</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-03T16:44:20Z2016-04-03T16:44:20ZArachni seems hanging<div><p><a href="http://downloads.arachni-scanner.com/nightlies/">Nightlies</a> are
up, let me know how they work.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-03T16:55:44Z2016-04-03T16:55:45ZArachni seems hanging<div><p>Cool, I'm starting a scan right away.</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-05T19:06:15Z2016-04-05T19:06:16ZArachni seems hanging<div><p>ok, I've run tests with both short scans (minutes) and long
scans (several hours), but I'm not seeing significant differences
in the number of timed out requests.<br>
Each scan still ends with a lot (thousands) of queued requests and
most of them times out.</p>
<p>I'll let you know if I find something interesting, but for now,
it looks like the problem is still here.</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T09:30:25Z2016-04-06T09:30:25ZArachni seems hanging<div><p>You mean browser jobs right?<br>
The requests don't actually time out.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T11:03:28Z2016-04-06T11:03:30ZArachni seems hanging<div><p>Yes, browser jobs.</p>
<p>I'll start a full scan with the nightly later today to get the
final data.</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T13:09:36Z2016-04-06T13:09:36ZArachni seems hanging<div><p>Hm, what's your browser job timeout set at?</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T14:51:21Z2016-04-06T14:51:22ZArachni seems hanging<div><p>The browser job timeout is set to 30 sec.</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T19:43:30Z2016-04-06T19:43:30ZArachni seems hanging<div><p>You should set it to something like 120, I'm not sure what the
right setting is for your application but it can't hurt.<br>
I was running a scan for quite some time and only got a handful of
timed out jobs with the above.</p>
<p>The odd time out now and then is to be expected, but now you
should see the jobs being processed much more reliably.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T19:59:13Z2016-04-06T19:59:14ZArachni seems hanging<div><p>Ok, I'll try that as well.<br>
Just out of curiosity: 120 sec is a long time for loading and
processing a web page, what is Arachni doing with all that time
?</p>
<p>About the memory consumption: I'm running with
--http-max-response-size=500000 (as per default) with
--browser-cluster-pool-size=4, that uses a about 1100 MB including
the OS.<br>
If double the --http-max-response-size and half the
--browser-cluster-pool-size the ruby maxes out the memory every
time on my 2GB virtual machine, and eventually the process dies
before it finishes.<br>
How can it be that increasing the --http-max-response-size has such
a dramatic effect ?<br>
And can something be done to avoid it ?</p>
<p>Best Regards</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T20:13:06Z2016-04-06T20:13:06ZArachni seems hanging<div><p>The system deals with page snapshots which include transitions
(you see these printed in the output), not just loading a page by
its URL -- which it tries in case it works.<br>
These transitions are basically a list of events that need to be
triggered on the root page to restore it to the snapshot's
state.<br>
It also waits for timers, in the case of your webapp the max timer
is 45s which will be capped to the value of the HTTP request
timeout option (not that they're related, but the request timeout
lets you glean how patient the system should be in general and
having a gazillion configurable time outs can get complicated).<br>
Then there's whether or not you're dealing with a cold cache or not
and the odd timed-out HTTP request due to server stress or a
dropped connection or whatever.</p>
<p>I'll check to verify that there's nothing fishy going on for
those few timed-out jobs, but a few failed jobs are to be expected
just like a few failed HTTP requests are, regardless of
configuration,</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T20:17:41Z2016-04-06T20:17:41ZArachni seems hanging<div><p>About the RAM, that's what the
<code>--http-max-response-size</code> option is there for, keeping
RAM consumption in check for these circumstances.<br>
Your site includes large pages with huge lists, ergo huge amount of
HTML that needs to be parsed and processed which results in many
times more RAM.<br>
I'll try a few optimizations and let you know.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T20:32:35Z2016-04-06T20:41:12ZArachni seems hanging<div><p>Ok, thank you, that's good to know.</p>
<p>The large inventory page has almost 2000 <code><a></code>
tags, 2/3 of them are ajax that don't generate a new page. I assume
that this can make that one page a very big task to process.</p>
<p>Also the cart page has some ajax (post action) not nearly as
much, but the inputs are named by the product id, which gives about
2000 possible variations to test.<br>
I can see that Arachni tests all of them, would it be possible to
have a redundancy limit like --scope-auto-redundant applied to post
as well as get ?<br>
That would certainly speed up the scanning.</p>
<p>Thanks again for all your help</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-06T20:45:24Z2016-04-06T20:45:24ZArachni seems hanging<div><p><code>--scope-auto-redundant</code> doesn't apply to GET
requests but resource locations regardless of HTTP method. I'm not
sure whether or not that helps in your case though, I'll have to
see one of the requests in question and I'm not sure which one it
is.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-08T13:25:58Z2016-04-08T13:26:01ZArachni seems hanging<div><p>Ok, I've got some more test results:<br>
arachni 2.0 nightly --browser-cluster-pool-size='4'
--browser-cluster-job-timeout='30' is actually about 30% faster has
30% less timeouts than arachni 1.4 with the same settings, on a
full scan. So that is very nice.</p>
<p>arachni 2.0 nightly --browser-cluster-pool-size='4'
--browser-cluster-job-timeout='120'<br>
and<br>
arachni 2.0 nightly --browser-cluster-pool-size='4'
--browser-cluster-job-timeout='60'<br>
Both ran out of memory before completion. But only colleted 1 time
out in comparisson to 5-6000 for the shorter timeout.</p>
<p>Now trying: arachni 2.0 nightly --browser-cluster-pool-size='2'
--browser-cluster-job-timeout='60'... will get back with the
results.</p>
<p>This is good progress.<br>
If only there was a less than fatal way to handle memory overruns
:)</p>
<p>Best regards</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-08T15:45:04Z2016-04-08T15:45:04ZArachni seems hanging<div><p>The kernel kills the process so there's not much that can be
done by Arachni.<br>
The recommended requirements state at least 2GB of available RAM
for Arachni and 2GB for the entire OS means you're bellow the
threshold.</p>
<p>And the defaults are meant to handle about 95% of cases using
those recommended requirements and your case both falls in the 5%
and runs on a system with less resources -- regardless of how this
goes, you should increase your RAM.</p>
<p>Btw, after a certain point the timeout is meant to act as a last
resort when something goes wrong, like in v1.4, so that one bug
won't freeze the entire scan.<br>
A large timeout for your case is a good thing, if the system needs
more time to do its job you should allow it; sacrificing coverage
to save some RAM isn't a good idea.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-11T18:56:11Z2016-04-11T18:56:11ZArachni seems hanging<div><p>May I please run a scan with DOM checks? (XSS, unvalidated
redirect, etc.)<br>
I did a crawl for a few hours and got no time outs, so I'd like to
make sure that jobs from the checks don't time out either.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-12T08:51:44Z2016-04-12T08:51:45ZArachni seems hanging<div><p>Yes of course, you can run a scan.</p>
<p>Thank you for your previous reply.<br>
I understand what you are saying (and mechanics of allocating
memory resources), my point was that it's hard to guess how many
browser cluster workers I can assign when starting the scan, and
given that a full scan can run for days it would be nice to trade
performance for completion.</p>
<p>My idea to achieve this was to check available memory prior to
the scanning of each new page (or other meaning full interval)
making sure that at least X amount of memory is available.<br>
This would effectively make the number of active phantomjs workers
dependent on the available memory resources and provide autoscaling
in a simple form.</p>
<p>Best Regards</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-12T10:44:52Z2016-04-12T10:44:52ZArachni seems hanging<div><p>I have thought of that, although I don't remember why I didn't
implement it.<br>
It's worth a shot though and could also work for many other parts
of the system, if you know that resources are running out you can
change a lot of other settings to automatically try and make due
with what's available.</p>
<p>I actually have the resource monitoring code available because I
was working on something similar for the new WebUI so that's a
plus.</p>
<p>You can follow the progress of this feature at: <a href="https://github.com/Arachni/arachni/issues/695">https://github.com/Arachni/arachni/issues/695</a></p>
<p>It's going to take a while to be implemented though.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-13T15:34:20Z2016-04-13T15:34:20ZArachni seems hanging<div><p>Current timed out jobs seem to be due to occasional HTTP
time-outs due to server stress.<br>
When running with all checks I get the occasional timed-out job
because there are a lot of requests being performed, when running
only with DOM checks I get no time-outs after running for a few
hours.</p>
<p>I'll perform a few more scans to profile the system, see if I
can optimize it.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-14T07:59:11Z2016-04-14T07:59:11ZArachni seems hanging<div><p>It would really be awesome if you can implement this way of
handling memory consumption, the scanning process would be much
more robust. I'll be looking forward to that.</p>
<p>Should I create a github issue for it ?</p>
<p>So it sounds like the remaining time outs are limited to target
server responsiveness, that's good news.</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-14T08:01:16Z2016-04-14T08:01:16ZArachni seems hanging<div><p>I beat you to it: <a href="https://github.com/Arachni/arachni/issues/695">https://github.com/Arachni/arachni/issues/695</a></p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-14T08:06:56Z2016-04-14T08:06:57ZArachni seems hanging<div><p>Perfect :)</p></div>Davetag:support.arachni-scanner.com,2012-07-01:Comment/393196332016-04-14T11:59:28Z2016-04-14T11:59:28ZArachni seems hanging<div><p>You may also want to grab the nightlies, they include some more
optimizations.</p></div>Tasos Laskos