tag:support.arachni-scanner.com,2012-07-01:/discussions/suggestions/945-memory-managementArachni: Discussion 2013-10-30T03:39:38Ztag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-19T12:03:46Z2013-09-19T12:03:46ZMemory management<div><p>Is there any way I can get some numbers? Like how much RAM does
the VM have? How many Instances you have running and how much RAM
they consume?</p>
<p>Also, what do you mean by "the instances are staying alive"? Are
their processes still there are the scan finishes?<br>
If you're talking about running scans not dying out when you close
the WebUI then that's the intended behavior, so that if the WebUI
crashes for whatever reason you won't lose your progress and be
able to grab the report once you fire-up the WebUI again.</p>
<p>Or are you using a Dispatcher? In that case that's the
Dispatcher's job, to maintain a pool of Instances. If you don't
want that the you can perform direct scans, where Instances are
spawned as needed.</p>
<p>So let's start by you giving me that info and we'll figure this
out.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T19:26:57Z2013-09-29T19:26:57ZMemory management<div><p>Hy tasos,</p>
<p>Sorry for my late answer.<br>
I found this in my /var/log/messages :<br>
<code>Sep 16 15:03:39 openvas kernel: OOM killed process 7152
(ruby) vm:1849208kB, rss:194448kB, swap:407644kB Sep 29 17:59:44
openvas kernel: : OOM killed process 2003 (ruby) vm:6109920kB,
rss:1832860kB, swap:2991764kB Sep 29 20:49:26 openvas kernel: : OOM
killed process 8196 (ruby) vm:1692656kB, rss:42328kB, swap:409276kB
Sep 29 20:52:03 openvas kernel: : OOM killed process 8200 (ruby)
vm:1418896kB, rss:167280kB, swap:64720kB Sep 29 20:52:19 openvas
kernel: : OOM killed process 8169 (ruby) vm:7557696kB,
rss:1969348kB, swap:3945872kB</code></p>
<p>I think that this is a Virtual issue.</p>
<p>By the way, is it normal that no date appears in the arachni
logs?</p>
<p>Thanks</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T19:54:45Z2013-09-29T19:54:45ZMemory management<div><p>Yeah your VM is killing the process because they're eating a lot
of its memory. The one that worries me is PID 2003, it hit 1.8GB
RAM usage.</p>
<p>Do you happen to know what that process was doing? Was it
performing a scan? If so, can you try performing the scan again and
checking if the RAM usage is similar?<br>
If I can reproduce this I'm pretty sure I'll be able to fix it.</p>
<p>Also, to which logs are you referring to? I'm pretty sure all of
Arachni's log files have timestamps for each logged message.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T20:16:40Z2013-09-29T20:16:40ZMemory management<div><p>Thanks for your answer tasos.</p>
<p>In my production.log I have things like that in the webui
part:<br>
<code>DispatcherManager#refresh Dispatcher Load (0.2ms) SELECT
"dispatchers".* FROM "dispatchers"</code> No date appears.<br>
Moreover , I didn't found logs for arachni framework. The folder is
empty</p>
<p>Regarding to the memory issue, I relaunched a have a look to the
processes. I'll let you know</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T20:19:41Z2013-09-29T20:19:41ZMemory management<div><p>Ah those logs, those are generated by Ruby-on-Rails and they're
just for the WebUI, you won't find any scan related info in
there.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T20:23:31Z2013-09-29T20:23:31ZMemory management<div><p>Ok,</p>
<p>So where are the arachni logs stored?</p>
<p>Regards</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T20:37:19Z2013-09-29T20:37:19ZMemory management<div><p>Depends of how you run the scan, if you were using a Dispatcher
to get an Instance for the scan you'd have found logs for the
Dispatcher under the "framework" directory, otherwise there's
nothing to log. If you were using a Dispatcher you'd also be able
to log all output of the scanners (as if you were running it from
the CLI) by using <code>--reroute-to-logfile</code>.</p>
<p>None of the logs would help debug this though. If you can
reproduce the RAM consumption issue I'll then have to try it for
myself and give it a very close look.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T20:58:55Z2013-09-29T20:58:55ZMemory management<div><p>for the moment I launch the process like this<br>
<code>bin/arachni_web -D --host 10.0.130.190</code> Do you mean
that launching t with <code>bin/arachni_web -D --host 10.0.130.190
--reroute-to-logfile /path/to/log/file</code> will be enough?<br>
Sorry if I didn't understood correctly.</p>
<p>regards</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T21:07:39Z2013-09-29T21:07:39ZMemory management<div><p>No, you'll need to:</p>
<ul>
<li>Start a Dispatcher using <code>bin/arachni_rpcd
--reroute-to-logfile</code>.</li>
<li>Add that Dispatcher to the WebUI.</li>
<li>Select "Remote" scan type using that Dispatcher via the
advanced options when starting a new scan.</li>
</ul>
<p>You'll then find 2 types of logs in the "framework" directory,
one from the Dispatcher and a few from the Instances the Dispatcher
has in its pool. One of those logfiles will belong to the Instance
you're using for the scan.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T21:20:08Z2013-09-29T21:20:08ZMemory management<div><p>Hy tasos,</p>
<p>To be sure, the dispatchers have to be started by CLI?<br>
Then, in the webUI, what port do I have to set?<br>
Does the dispatcher need to be on a different machine?<br>
Does a dispatcher could be started at boot?</p>
<p>Regards</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T21:25:13Z2013-09-29T21:26:04ZMemory management<div><ol>
<li>Yep, it's like a server which keeps track of scanner Instances.
You generally don't need it in simple-ish deployments but provides
improved logging, among other things.<br></li>
<li>Default address is <code>localhost:7331</code>.<br></li>
<li>Nope.<br></li>
<li>Sure, you just have <code>bin/arachni_rpcd
--reroute-to-logfile</code> run at boot.</li>
</ol></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-29T21:50:46Z2013-09-29T21:50:46ZMemory management<div><p>so ....</p>
<p>Let's log all that stuff ;-)</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-30T05:59:15Z2013-09-30T05:59:15ZMemory management<div><p>So I think I have another issue.<br>
This morning no more UI is available.<br>
Ruby proc still running.<br>
No OOM appears in the logs that could kill the processes.<br>
I restarted the UI by CLI and the scan still running.<br>
The VM has enough resources for this scan</p>
<p>I'll try everything in a full virtualized VM. Maybe it could be
better.<br>
What is amazing for me is that I didn't succeed to scan a site. My
conf should be weird!</p>
<p>regards</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-30T11:34:07Z2013-09-30T11:35:00ZMemory management<div><p>@mailinglist, as for the memory usage increasing on long scans,
i proposed Tasos to add disk buffer file for temporary storing data
so freeing more ram but the idea was not accepted (maybe in the
future who knows) and i respect his decision.</p>
<p>Currently open issues relayed to this:<br>
<a href=
"https://github.com/Arachni/arachni/issues/389">https://github.com/Arachni/arachni/issues/389</a><br>
<a href=
"https://github.com/Arachni/arachni/issues/366">https://github.com/Arachni/arachni/issues/366</a></p>
<p>A few points that might help reducing memory usage though :</p>
<ul>
<li>use a filter for binary content, so the crawler doesn't store
all that data, like
exclude='.a3c|.ace|.aif|.aifc|.aiff|.arj|.asf|.asx|.attach|.au|.avi|.avi|.bin|.bmp|.cab|.cache|.class|.djv|.djvu|.dwg|.es|.esl|.exe|.fif|.fvi|.gif|.gz|.hqx|.ice|.ico|.ief|.ifs|.iso|.jar|.jpe|.jpeg|.jpg|.kar|.mdb|.mid|.midi|.mov|.movie|.mp|.mp2|.mp3|.mp4|.mpeg|.mpeg2|.mpg|.mpg2|.mpga|.msi|.pac|.pdf|.png|.ppt|.psd|.qt|.ra|.ram|.rar|.rm|.rpm|.snd|.svf|.tar|.tgz|.tif|.tiff|.tpl|.uff|.wav|.wma|.wmv|.zip'</li>
</ul>
<p>-in case ur using the trainer module, that also have a big
impact</p></div>user021tag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-09-30T11:55:22Z2013-09-30T11:55:22ZMemory management<div><p>@user021 Like I've already explained, it is very unlikely that
this has anything to do with data the system stores. If you
experience high RAM consumption then something's leaking memory,
it's a bug that needs to be fixed. A disk buffer generally wouldn't
do you any good.</p>
<p>Also, the crawler doesn't store response bodies, that filter
will prevent the crawler from following links that match the
filter, and will save you time and bandwidth, but has none to very
little effect on RAM consumption.</p>
<p>However, you may be onto something with the <code>Trainer</code>
though. The <code>Trainer</code> <strong>does</strong> store new
elements in RAM, and those elements are part of a page, so if a lot
of new elements appear during the audit then those pages will stay
in RAM until they're audited. This is probably the only data
structure in the system which can cause those issues, given that a
memory leak has been ruled out. In that case, a disk buffer would
indeed help.</p>
<p>However, this is all conjecture until I have a reproducible
case.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-10-10T19:28:01Z2013-10-10T19:28:01ZMemory management<div><p>Guys, any news on this? I'd really like to get this sorted.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-10-10T19:43:14Z2013-10-10T19:43:14ZMemory management<div><p>Hy Tasos,</p>
<p>Thanks for your following. My planning is a bit overload. I
should have a look to the VM that hosts your soft in the next 10
days. I'll inform you immediately,</p>
<p>regards</p></div>mailinglisttag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-10-10T23:15:04Z2013-10-10T23:52:51ZMemory management<div><p>I just <a href=
"https://github.com/Arachni/arachni/commit/d37fb1da197c058f278cbaee7d9de1321295f381">
pushed</a> an optimization that leads to pages being consumed ASAP
and not be stored in RAM for extended periods of time.</p>
<p>If the problem was caused by the Trainer, this should take care
of it.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-10-10T23:54:23Z2013-10-10T23:54:23ZMemory management<div><p>Well, there may be one more case that I may have missed so I'll
see if I can do more.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-10-11T01:36:34Z2013-10-11T01:36:34ZMemory management<div><p>OK, the page queue is now offloaded to disk. If that was indeed
the problem it should now be fixed.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-10-11T03:14:33Z2013-10-11T03:40:05ZMemory management<div><p>More good news, I tweaked the part of the HTTP library that has
to do with how many requests can be queued at a time and this has
greatly reduced RAM consumption.</p>
<p>For example, a very simple scan which requires 94MB of RAM with
the current stable version, requires 64MB with the code in the
experimental branch.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/289019612013-10-30T03:38:42Z2013-10-30T03:39:38ZMemory management<div><p>Hey folks,</p>
<p>I've optimized the hell out of Arachni and the changes can be
found in the nightlies.<br>
Differences for a sample scan follow.</p>
<p><strong>experimental</strong></p>
<p>RAM usage:</p>
<ul>
<li>After crawl: 48.068 MB</li>
<li>After audit: 65.116 MB</li>
<li>After plugins: 82.7 MB</li>
<li>
<p>After reports: 86.824 MB</p>
<pre>
<code>[~] Sent 53861 requests.
[~] Received and analyzed 53861 responses.
[~] In 00:20:59
[~] Average: 42 requests/second.</code>
</pre></li>
</ul>
<p><strong>v0.4.5.2</strong></p>
<p>RAM usage:</p>
<ul>
<li>After crawl: 48.068 MB</li>
<li>After audit: 85.12 MB</li>
<li>After plugins: 102.732 MB</li>
<li>
<p>After reports: 105.088 MB</p>
<pre>
<code>[~] Sent 78657 requests.
[~] Received and analyzed 78657 responses.
[~] In 00:29:32
[~] Average: 44 requests/second.</code>
</pre></li>
</ul></div>Tasos Laskos