tag:support.arachni-scanner.com,2012-07-01:/discussions/problems/177-memory-management-with-large-formsArachni: Discussion 2013-11-11T19:32:58Ztag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-07T21:29:50Z2013-11-07T21:29:50ZMemory management with large forms<div><p>Hey Mike,</p>
<p>The stacktrace shows that the memory ran out when the issue data
were being copied prior to a processing operation. That could be
because the issues were too many for the machine to hold in memory
or because there was no memory left because it used it all to
generate form mutations (the latter sounds more reasonable).</p>
<p>How much RAM does the machine have?</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-08T19:23:38Z2013-11-08T22:38:28ZMemory management with large forms<div><p>It is a VMWare VM and has 4GB available... The RAM utilization
never gets above 3GB however during the audit.</p></div>Miketag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-08T19:27:26Z2013-11-08T19:27:26ZMemory management with large forms<div><p>3GB RAM usage is pretty unacceptable anyways no matter the
reason.</p>
<p>I skipped your case while I was performing memory optimizations
recently because I figured that no-one would run into something
like that but clearly you're affected.</p>
<p>I'm half-done with the fix, will let you know once its
ready.</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-08T23:07:18Z2013-11-08T23:07:18ZMemory management with large forms<div><p>I'm now running the full test suite to make sure the fix didn't
introduce any regressions, so I've got some time to explain the
issue in some detail.</p>
<p>What was going on in the system was that it needed to generate
fuzzing mutations for that element and as it had 2000 inputs there
were an astounding amount of permutations to be generated and held
in memory.</p>
<p>What I've done now is update the audit to use pairs of
generate-consume operations, instead of generating mutations in
bulk first (and thus requiring a lot of memory and introducing
latency to the audit) and consuming later. This way only the
absolutely minimum and necessary amount of elements will remain in
memory and the audit moves along much faster.</p>
<p>However, because the form is so big and a few permutations of it
will have to remain in memory, the system will still require a
significant amount of memory. Not as much as before mind you, but
still plentiful (half a GB, possibly...hopefully).</p>
<p>Also, I've added a new option (<code>--http-queue-size</code>)
which lets you control the maximum amount of requests to be stored
in the HTTP queue before performing a run. More requests means
better I/O scheduling and thus better performance, fewer requests
means less memory (that's because each element remains in memory
until its associated response has arrived and been processed).</p>
<p>After the test suite has finished I'll see if I can optimize
this further and give you some hints about fine-tuning the above
option in order to keep RAM consumption at bay.</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-08T23:38:40Z2013-11-08T23:38:41ZMemory management with large forms<div><p>Great thanks for working on this...</p>
<p>By the way I made another discussion that I think got caught in
your spam filter regarding the AutoLogin plugin... I went ahead and
registered for an account finally so maybe my posts won't be
flagged so often.</p>
<p>Let me know how it goes</p>
<p>Thanks again!</p></div>Miketag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-09T00:20:38Z2013-11-09T00:20:38ZMemory management with large forms<div><p>No worries.</p>
<p>The spam filter does seem to have a few issues with you, sorry
about that.</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-10T03:34:31Z2013-11-10T03:34:31ZMemory management with large forms<div><p>Problem sort of solved, although I don't see what else I can
do.</p>
<p><code>experimental</code> branch (code can be found in the
<a href=
"http://downloads.arachni-scanner.com/nightlies/">nightlies</a>):</p>
<ul>
<li><code>--http-queue-size=20</code>: RAM around 110MB.</li>
<li>Default (<code>--http-queue-size=500</code>): RAM around
500MB.</li>
</ul>
<p><code>master</code> branch: RAM 910MB after mutation generation,
just before the audit starts and increasing as the audit goes
on.</p>
<p>So you can massively improve RAM consumption by lowering the
default value of <code>http-queue-size</code> but the audit will
still be pretty slow due to the design of the webapp. A form with
2000 inputs means a lot of HTML code to be analyzed by Arachni and
a lot of audit workload, but it also probably means a big workload
for the server as well.</p>
<p>Overall, this is a bad situation, be prepared for a massive hit
in performance.</p>
<p>I won't close this issue until I hear how it works for you so
please do let me know.</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-11T15:55:17Z2013-11-11T15:55:17ZMemory management with large forms<div><p>Btw, you'll need to disable the <code>sqli_blind_rdiff</code>
module because it really needs to generate the mutations prior to
performing the analysis, no way to get around that.</p>
<p>Cheers</p></div>Tasos Laskostag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-11T19:24:46Z2013-11-11T19:24:46ZMemory management with large forms<div><p>Arachni is no longer crashing when I set the http-queue-size to
100. The RAM utilization for ruby is somewhere around 1GB with the
limit on as opposed to the 3 or so without it.</p>
<p>I agree there is nothing the scanner can do to improve the bad
design of the Web application so this fix works for me.</p>
<p>Thanks,</p></div>Miketag:support.arachni-scanner.com,2012-07-01:Comment/298558422013-11-11T19:32:57Z2013-11-11T19:32:57ZMemory management with large forms<div><p>No worries man, glad you got it working.</p>
<p>Cheers</p></div>Tasos Laskos