Arachni seems hanging
I am running the latest version (Arachni v1.4 - WebUI v.0.5.10).
The pages discovered is 2364; the requests performed and the responses received are both 2047725. The running time is more than 39 hours and it is still running. However, Arachni seems hanging without further requests and response. The other information is standing still without changing. It is showing "scanning" too.
I do not want to quit it as it is running over 39 hours. Any idea?
Comments are currently closed for this discussion. You can start a new one.
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
Support Staff 1 Posted by Tasos Laskos on 03 Mar, 2016 04:53 PM
That's strange, are there any errors printed in the scan progress page?
2 Posted by samiux on 03 Mar, 2016 05:13 PM
The last message is over 10 hours ago which is about giving up to audit a page after 5 tries as it has no data and no headers returns.
Support Staff 3 Posted by Tasos Laskos on 03 Mar, 2016 05:16 PM
I'm not sure what's going on, if I were you I'd abort the scan and retry via the CLI.
If there is indeed an issue then it'll be easier to spot using the CLI's progress messages.
4 Posted by samiux on 03 Mar, 2016 05:19 PM
How to insert custom request headers in cli?
Support Staff 5 Posted by Tasos Laskos on 03 Mar, 2016 05:19 PM
See: https://github.com/Arachni/arachni/wiki/Command-line-user-interface...
6 Posted by samiux on 03 Mar, 2016 05:21 PM
How about multiple request headers?
Support Staff 7 Posted by Tasos Laskos on 03 Mar, 2016 05:23 PM
You can invoke the option multiple times to add as many headers as you want.
8 Posted by Dave on 14 Mar, 2016 03:41 PM
I have the same problem with the same version, even with much fewer pages to scan.
At some point after several hours of successful scanning, it seems like the browser workers stop responding and most of the requests times out after 25 secs.
The site I'm scanning is still responding.
This effectively prohibits the scan from completing, hence the very long scan time.
I'm on Linux Debian 8.1 and ruby 2.2.3.
I've sent you some debug info in an email, I hope that helps.
Support Staff 9 Posted by Tasos Laskos on 15 Mar, 2016 12:45 AM
I couldn't find an issue with it, all I had to do to avoid the time-outs was specify a larger allowance with
--browser-cluster-job-timeout=60.I'm certain that the same time-outs were occurring during the entire scan but you only noticed them at the end because there was nothing else going on.
So, the browsers were responding, they just needed more time to perform their tasks.
Also, I've noticed that response times tend to periodically get pretty high, which adds to the problem.
I'll keep testing to make sure, but since I increased the time allowance there haven't been any time-outs.
10 Posted by Dave on 15 Mar, 2016 10:51 AM
Interesting, I'm running another scan to try that right away.
I'll let you know, how it works out.
11 Posted by Dave on 16 Mar, 2016 08:38 AM
Good morning
I ran a scan with --browser-cluster-job-timeout 60
The first one maxed out the memory, so I reduced the --browser-cluster-pool-size from 4 to 2 and ran another one. It's been running for >13 hours now and most of the remaning browser jobs exceed the 60-second limit with messages like these:
[~] BrowserCluster: Pending jobs: 45352 [-] BrowserCluster Worker#25672880: Job timed-out after 60 seconds: #[~] BrowserCluster: Pending jobs: 45351 [-] BrowserCluster Worker#25672880: Job timed-out after 60 seconds: #[~] BrowserCluster: Pending jobs: 45350 [-] BrowserCluster Worker#25672880: Job timed-out after 60 seconds: #[~] BrowserCluster: Pending jobs: 45349
Unfortunately, I can't see which jobs are failing, can I get that information somehow?
The Site I'm scanning has a lot of mp3 files, could that be causing the problems?
Best regards
Dave
Support Staff 12 Posted by Tasos Laskos on 16 Mar, 2016 03:11 PM
I saw one of these somewhere down the line too so I think I've got an idea.
I'll keep digging.
13 Posted by Dave on 16 Mar, 2016 03:18 PM
Cool, let me know if I can do something.
I'm currently running a new scan with debug enabled, I'll be happy to send you the output file.
Btw. I've tried scanning testfire.net, which worked without problems. So that should rule out the most basic OS and config issues.
Support Staff 14 Posted by Tasos Laskos on 16 Mar, 2016 03:20 PM
The debug output won't work for this job and the issue is specific to your website so don't worry about it, I'll keep you updated on my progress.
Support Staff 15 Posted by Tasos Laskos on 16 Mar, 2016 09:11 PM
Btw, I'll also look into the RAM usage, I don't like Arachni using more than 2GB, although in your particular case it may be warranted due to the large number of huge pages -- still, it's worth a look.
I've also identified another optimization that should severely limit the amount of browser jobs.
Support Staff 16 Posted by Tasos Laskos on 17 Mar, 2016 07:47 PM
When performing a timed-out job individually there was no problem so it may be workload related, I'll keep digging.
17 Posted by Dave on 17 Mar, 2016 08:06 PM
Your effort is very much appreciated.
Speaking of workload, I saw that the cart page was subjected to >463.000 checks, which seems like a lot for one page. The cart page has some ajax functionality in addition to the dynamic nature of a cart, not sure if that makes a difference in this case but now you know.
Best regards
Dave
18 Posted by Dave on 21 Mar, 2016 01:56 PM
Hi Tasos
Any news about the hanging scans?
Best regards
Dave
Support Staff 19 Posted by Tasos Laskos on 21 Mar, 2016 04:31 PM
The only common thing I could find was that once the time allowance has been increased adequately, this only happens for extremely large pages and only when as part of the scan, individually the jobs complete without issue.
This is a tricky one.
20 Posted by Vojta on 23 Mar, 2016 06:57 PM
Hi, I experience the same. I am scanning prety complex web application full of Ajax, it stopped working when reached about 2100000 requests.
I am running this on Windows through web interface. Couldn't the database be the problem? I use default database. I could run this with command line but I think that the Windows version lacks the status screen, right?
Another thing - when I have tried to suspend this scan, it have been showing "suspending" for several hours.
I have set job timeout to 60. Should I even increase it?
Thanks,
Vojta
Support Staff 21 Posted by Tasos Laskos on 23 Mar, 2016 10:02 PM
@Vojta
Suspending the scan has to wait for the queued browser jobs to complete, so if there are a lot of them it will take time.
And you're right about the missing CLI status screen on Windows.
Your situation could help me diagnose the issue, would you mind sending me the details in private so that I can try and reproduce it? ([email blocked])
Cheers
22 Posted by Vojta on 24 Mar, 2016 08:08 AM
Hi, your email was redacted, I contacted you at the email address provided on Arachni home page. Thanks for your time.
Support Staff 23 Posted by Tasos Laskos on 28 Mar, 2016 04:36 PM
@Dave
By the way, there seems to be a 45s
setTimeout()call in the page which makes loading the pages significantly slower, Arachni will cap this based on the HTTP request timeout setting -- so by default it'll wait for 10s rather than 45, but still pretty slow.Still trying to isolate the problem, will keep you updated.
24 Posted by Dave on 28 Mar, 2016 05:07 PM
Hi Tasos
Thank you for getting back to me.
I'm not sure what you mean, is that a setTimeout() call in Arachni or my website ? and in the latter case, which page are you referring to ?
My numbers in Newrelic shows that server response time (for Http requests) is <100ms in the 99 percentile.
Meanwhile i've been doing some testing of my own and found a few things:
My page didn't handle non-existing pages properly in all cases, so for instance /DE (which we don't have) returned a 302 Http status rather than a 404, that might have caused some extra crawling time. I've fixed that now on the development environment, I can send you a link in an email.
I've run scans with different subsets of checks to try to figure out if a specific check could be linked to the problems. So far the Trainer check seems to be a candidate, I'm currently running a full scan with all checks except the Trainer, to see if I can validate that hypothesis.
I'm more than happy to help figuring this out, let me know if I can provide any additional info.
Thank you again for looking into this.
Kind Regards
Dave
Support Staff 25 Posted by Tasos Laskos on 28 Mar, 2016 05:14 PM
It's a
setTimeout()in http://festklaveret.dk/melodi-til-din-festsang.html and probably all pages, most likely set by a JS file.Newrelic's metrics don't have anything to do with that though, Arachni will wait for the largest timeout to trigger in order to let the page get to its real ready state as JS is executed asynchronously.
I've also managed to reproduce the issue more easily so that should help significantly.
Btw, it has nothing to do with specific checks (I don't think so), even if you disable all checks and just do a crawl you'll see some jobs timeout.
26 Posted by Dave on 31 Mar, 2016 12:20 PM
That's interesting. We don't use setTimeout() in our own js code, but it is used a few times in jQuery and underscore.js. So that's probably where that is originating.
Also I'm running a scan with Arachni 1.3, it's been running for more than 2 days now, but it has neither chrashed or stalled, so thats good.
Any news about the 1.4 ?
Support Staff 27 Posted by Tasos Laskos on 31 Mar, 2016 01:09 PM
Progress is slow, unfortunately it takes a long time to reproduce which makes it hard to debug.
Support Staff 28 Posted by Tasos Laskos on 03 Apr, 2016 12:43 PM
At last, I've identified the issue, it has to do with pending HTTP requests in Arachni's proxy.
Some requests are still shown as in progress even after they've completed so the browsers waits indefinitely and the job times out.
Not sure if it's the proxy itself or the underlying library that handles the connections yet but I'm getting closer.
Cheers
Support Staff 29 Posted by Tasos Laskos on 03 Apr, 2016 01:46 PM
OK, I'm pretty sure I fixed it now, SSL connections which were abruptly closed didn't clean-up properly.
I'll have a nightly for you to test soon.
30 Posted by Dave on 03 Apr, 2016 02:00 PM
That is great news, which actually reasons with what I've been noticing: the number of spawned Arachni process seems to grow during the scan, most of them are passive but still taking up memory.
The scan I was running in Arachni 1.3.2 stopped after running for 10 days. It ran out of memory, just like the 1.4 scans.
Today I've been running the same scan with RPC/RPCD, with similar results.
I was hoping to split the workload be providing two RPCD, but Apparently all the work is done by the RPCD that I'm calling with the RPC client.
Is that how it is supposed to work ?
Ultimately I got the same results: sooner or later Arachni runs out of memory and the process is killed by the OS.
A good takeaway is that I found a way to replicate the problem much faster than before. I'll send you an email with the way to do it.
All test are run on 2GB instances.
Looking very much forward to the update :)