Workload exhausted

Malween Le Goffic's Avatar

Malween Le Goffic

14 Dec, 2017 03:54 PM

Hi,

I have a problem with Arachni when I scan an application, here the log file :

[Thu Dec 14 13:35:43 2017] [status]  Got control verification response for cookie variable 'MY_COOKIE' with action 'https://domain/connexion/XUI/libs/requirejs-2.1.14-min.js'.
[Thu Dec 14 13:35:43 2017] [status]  Got control verification response for cookie variable 'varl' with action 'https://domain/connexion/XUI/libs/requirejs-2.1.14-min.js'.
[Thu Dec 14 13:35:44 2017] [line]
[Thu Dec 14 13:35:44 2017] [status]  Workload exhausted, waiting for new pages from the browser-cluster...
[Thu Dec 14 13:35:44 2017] [info]  Pending jobs: 12789

After the scan reach this, the status is always "status=scanning and busy=true" but stuck for more than 2 hours and never resume it. For information, the scan started at 9am and was stuck at 1pm, manually shutdown at 3pm.

I don't know if this will help you but the javascript file mentionned as attach file.

Best regards,

  1. Support Staff 1 Posted by Tasos Laskos on 19 Dec, 2017 04:17 PM

    Tasos Laskos's Avatar

    Do the "Pending jobs" keep going down or is it just stuck there?

  2. 2 Posted by Malween Le Goff... on 20 Dec, 2017 09:00 AM

    Malween Le Goffic's Avatar

    Arachni seems stuck when the "Pending jobs" state is reached. When I request the Arachni REST server with summary order, the job is : scanning and busy true. But on the log file (arachni_path/system/logs/framework no activity after the Workload exhausted entry.

    Regards,

  3. 3 Posted by Malween Le Goff... on 21 Dec, 2017 08:40 AM

    Malween Le Goffic's Avatar

    Hi,

    I did more tests on the targeted application to try to understand where the problem is coming from. The last test carried out had a very light profile with only a few checks. The analysis still stopped (stucked) after a few hours of operation. On the profile I did however exclude the javascript file indicated above, it does not seem to be the root cause of the problem. Here are the last lines of the analysis log:

    [Wed Dec 20 15:53:06 2017] [info]  Pending jobs: 44483
    [Wed Dec 20 15:53:19 2017] [info]  Pending jobs: 44482
    [Wed Dec 20 15:53:31 2017] [info]  Pending jobs: 44481
    [Wed Dec 20 15:53:44 2017] [info]  Pending jobs: 44480
    [Wed Dec 20 15:53:56 2017] [info]  Pending jobs: 44479
    [Wed Dec 20 15:54:09 2017] [info]  Pending jobs: 44478
    [Wed Dec 20 15:54:22 2017] [info]  Pending jobs: 44477
    [Wed Dec 20 15:54:35 2017] [info]  Pending jobs: 44476
    [Wed Dec 20 15:54:47 2017] [info]  Pending jobs: 44475
    [Wed Dec 20 15:55:59 2017] [info]  Pending jobs: 44474
    

    However, the summary analysis is still active (active and busy).

    I also made a call to the REST API to delete the scan, and the log on the REST API side indicated the following information:

    [Thu Dec 21 09:18:23 2017] [status]  Shutting down...
    [Thu Dec 21 09:18:23 2017] [error]  [Arachni::BrowserCluster::Error::AlreadyShutdown] Cluster has been shut down.
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/browser_cluster.rb:473:in fail_if_shutdown'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/browser_cluster.rb:276:indone?'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework/parts/browser.rb:54:in wait_for_browser_cluster?'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:179:inblock in audit'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:177:in loop'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:177:inaudit'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework.rb:117:in block in run'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/utilities.rb:425:incall'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/utilities.rb:425:in exception_jail'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework.rb:117:inrun'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/rpc/server/framework.rb:156:in block in run'
    [Thu Dec 21 09:18:23 2017] [error]
    [Thu Dec 21 09:18:23 2017] [error]  Parent:
    [Thu Dec 21 09:18:23 2017] [error]  Arachni::RPC::Server::Framework
    [Thu Dec 21 09:18:23 2017] [error]
    [Thu Dec 21 09:18:23 2017] [error]  Block:
    [Thu Dec 21 09:18:23 2017] [error]  #<Proc:0x000000055d14e8@/arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework.rb:117>
    [Thu Dec 21 09:18:23 2017] [error]
    [Thu Dec 21 09:18:23 2017] [error]  Caller:
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/utilities.rb:425:inexception_jail'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/framework.rb:117:in run'
    [Thu Dec 21 09:18:23 2017] [error]  /arachni-1.5.1-0.5.12/system/gems/gems/arachni-1.5.1/lib/arachni/rpc/server/framework.rb:156:inblock in run'
    [Thu Dec 21 09:18:23 2017] [error]  --------------------------------------------------------------------------------
    
    Is it normal that the error indicating that the cluster is already shut down while the scan is supposed to be still active ?

    I hope this can help move forward on the issue.

    Best regards,

  4. 4 Posted by Malween Le Goff... on 01 Mar, 2018 10:04 AM

    Malween Le Goffic's Avatar

    Hi,

    Some news about the scan of the target which end with a "workload exhausted" message (and an infinite loop). After a certain amount of tests I found out that the option called "Audit UI forms" in "Audit" section causes this error/infinite loop (Worlkload exhaustive message). Without this option, the analysis of the target ends correctly after few hours.

    Can you explain what exactly do this option and if you have an idea of why this option causes an this infinite loop ?

    Best regards,

  5. Support Staff 5 Posted by Tasos Laskos on 26 Mar, 2018 08:03 AM

    Tasos Laskos's Avatar

    That option basically turns all orphan page inputs (like textareas and selects that don't belong to a specific form) and submit buttons into virtual-forms, sort of.

    So if there are a lot of those, the scan can take a long time, although I don't know about inf loops, there are limits to basically every procedure that could cause one, but bugs are unavoidable as well.

    About the error, this:

    [Thu Dec 21 09:18:23 2017] [status]  Shutting down...
    

    Means that the scan wasn't actually active, you deleted it, i.e. shut it down, so pending jobs were rejected by the cluster.
    The error looks scary but nothing went wrong, although catching this post-shutdown case prior to it getting to the point where the system thinks it's in error is something I need to do.

    Cheers

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Already uploaded files

  • requirejs-2.1.14-min.js 14.9 KB

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac