browser cluster problem / ram usage problem
hello, after a long-time work (24h+) arachni stops respawning browsers, now i have only 10 browsers of the expected number of 36 (6 instances/6 browser per instance). Also got a problem with RAM - some instances use too much (10Gb+), does arachni really needs so much RAM ?
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
1 Posted by John on 18 May, 2017 12:44 PM
attached a screenshot
Support Staff 2 Posted by Tasos Laskos on 18 May, 2017 12:59 PM
Looks like you've stressed your system too much, try running fewer scans at the same time.
As for the RAM, are you saying that a single instance uses 10GB+ of RAM? That's hard to believe.
3 Posted by John on 18 May, 2017 01:10 PM
see screenshot: 10634Mb ; 8th line from the top.
Support Staff 4 Posted by Tasos Laskos on 18 May, 2017 01:18 PM
That's surprising to say the least, any chance you can give me the scan details so that I can try to debug the issue? I'll be needing to perform an identical scan against the same target.
5 Posted by John on 18 May, 2017 01:22 PM
sent via email
6 Posted by John on 18 May, 2017 02:14 PM
without any critical CPU load (30-70%) arachni still has a problem with respawning browsers, now running 5 instances (expected 30 browsers), but have only 26
Support Staff 7 Posted by Tasos Laskos on 18 May, 2017 02:17 PM
You're trying to run 30 processes that require considerable resources, unless you've got 30 cores then this could indeed be the issue.
8 Posted by John on 18 May, 2017 02:23 PM
you say it's a hardware problem. system could handle 30 browsers (at start) with max load at 70% (CPU) and 15% (RAM), but it can't respawn browsers due to lack of recources (which doesn't exist) - it seems for me to be illogically. Or i had misunderstood your message ?
Support Staff 9 Posted by Tasos Laskos on 18 May, 2017 02:37 PM
There's a way to settle this, try running only 1 instance, if you're still getting the same issue then it's a bug in Arachni, otherwise it's system stress.
Support Staff 10 Posted by Tasos Laskos on 18 May, 2017 03:33 PM
Another thing to watch out for is to check the PIDs of the browsers, they could be respawning successfully but slowly, thus appearing fewer at any given time.
11 Posted by John on 19 May, 2017 05:59 AM
No, they don't respawn at all (during few hours of scan the number of browsers just decrease). One instance works stable, but 2nd and 3d are already unstable. Current system has 6 physical and 12 logical cores (2.3Ghz each), maximum load running 3 instances is 50%, average is 30% . Tried both on Linux and windows - problem persists. System also has more than 20Gb of free ram (ddr3). Bandwidth isn't overloaded too. I don't even guess what the matter is, it's hard for me to believe this system can handle only 1-2 instances stably.
Support Staff 12 Posted by Tasos Laskos on 19 May, 2017 01:09 PM
I don't know what to tell you, this is out of my hands and there are plenty of folks out there running multiple scans without issue.
This sounds like an env issue.
About the RAM consumption, I got your e-mail and I'll have a look at it when I find some free time.
13 Posted by John on 19 May, 2017 02:34 PM
can you plase try running 3 or more scans ? On the screenshot you can see that even after 30 minutes arachni gets a problem with respawning ( in cmd is the number of phantomjs processes running (17 of 18 expected) - for easier counting)
14 Posted by John on 19 May, 2017 03:30 PM
+when all browsers crashed, instances get a fatal error => crash
15 Posted by John on 19 May, 2017 04:58 PM
tested on the third system (win 8.1 / x64 / i7-5960x / 32GB ddr4 /3 INSTANCES (6 browser per instance) / ) - the same situation. attached a file with logged numbers of running browsers during ~1 hour scan. i've tested on debian x64 and win serv 2016 before. Also, any of these three systems had a different internet provider. So it can't be system or hardware problem. Do you have any ideas? Maybe you should add a re-respawn in case of respawning failure?
16 Posted by John on 19 May, 2017 07:14 PM
problem with respawn occures even to single scans (see screenshot). Looking forward to getting any reply:)
Support Staff 17 Posted by Tasos Laskos on 27 May, 2017 03:50 PM
I haven't forgotten about this but I've been a little preoccupied as of late. I'll get to it when I find some free time.