Normal Arachni progress

Ranus's Avatar

Ranus

29 Dec, 2017 01:24 PM

Hi Tasos,
I was wondering if you could explain what is a "normal" scan progress.
I explain, I run a scan and after 16 hours I see it's scanning a page. When I watch now, after more than 20 hours (4 hours latter), it scan the same web page.
I though maybe Arachni was crawling all the website for getting a map and scan all the webpages after.

Could you please tell me if I'm wrong ?

One more question, I'm seeing 1315 pages snapshots during the scan but when it interrupt I'm only seeing between 100 and 400 on the report. Is it normal ?

Regards,

  1. Support Staff 1 Posted by Tasos Laskos on 29 Dec, 2017 07:40 PM

    Tasos Laskos's Avatar

    The 2 are different, let's start by defining a couple of things:

    • Snapshot: Page + DOM state (starts as no DOM)
    • Page: Resource per URL

    You may see the same "page" being audited, but actually it's a different snapshot (DOM state) of the page, for example, the list of transitions will probably be different -- not necessarily, but usually new states come after DOM events which are recorded as page DOM transitions.

    Now, regarding your issue of the same page being audited for so long, as I said, it's probably different states and it seems like there might be a lot of them or the server could be very slow, resulting in large scan durations, have you tried this? http://support.arachni-scanner.com/kb/general-use/optimizing-for-fa...

    Also, the system doesn't crawl first and keep a list (or map or tree) of resources to audit later, it's all on the fly and in a way interconnected in a feedback-loop -- the crawl and audit are basically complementary processes.

    Not sure if the above makes sense, I haven't had my coffee yet and the first draft of my reply really did not make any.

  2. 2 Posted by Ranus on 02 Jan, 2018 12:36 PM

    Ranus's Avatar

    Hi,
    Thank you for your answer. I understand much better now.

    Yes, I followed the guide but couldn't end a test for now :'(
    I've tried to only check allowed methods or CSRF but couldn't end it anyway.

    A weird thing is that it get stuck on a different page every time.
    The website is really big, but it get stuck after like 10-20 hours.

    I tried with 1-50 browsers cluster.

    I think my problem is similar to this one : http://support.arachni-scanner.com/discussions/problems/4922-worklo...

    If you want, I've sent you a capture of the problem on your support email address ( december 06 2017 17:09).

    Thank you for your response before your coffee and happy new year

  3. 3 Posted by Ranus on 04 Jan, 2018 12:31 PM

    Ranus's Avatar

    Hi,
    I tried with output-debug=5

    Here is last console screen when it get stuck :

    [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/bridge.rb:170:in `execute'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/oss/bridge.rb:581:in `execute'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/oss/bridge.rb:549:in `find_element_by'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/search_context.rb:62:in `find_element'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/browser/element_locator.rb:70:in `locate'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/element/dom/capabilities/locatable.rb:22:in `locate'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/element/ui_form/dom.rb:46:in `trigger'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/element/dom/capabilities/submittable.rb:64:in `submit_with_browser'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/element/dom/capabilities/auditable.rb:47:in `audit_handle_submit'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/browser_cluster/jobs/browser_provider.rb:26:in `call'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/browser_cluster/jobs/browser_provider.rb:26:in `run'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/browser_cluster/job.rb:109:in `configure_and_run'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/browser_cluster/worker.rb:80:in `run_job'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:88] Worker: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-dfdc70df98a8/lib/arachni/browser_cluster/worker.rb:244:in `block in start'
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser_cluster/worker#browser_respawn:259] Worker: Re-spawning browser (TTD?: false - alive?: true) ...
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser#shutdown:397] Worker: Shutting down...
     [2018-01-04 08:54:34 +0100 - 81.4] [!!] [browser#shutdown:399] Worker: Killing process.
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [browser#shutdown:408] Worker: Shutting down proxy...
     [2018-01-04 08:54:34 +0100 - 82.7] [!!] [http/proxy_server#shutdown:95] ProxyServer: Shutting down...
     [2018-01-04 08:54:34 +0100 - 82.0] [!!!] [http/proxy_server/connection#on_close:221] Connection: Closed because: [Arachni::Reactor::Connection::Error::Closed] end of file reached
     [2018-01-04 08:54:34 +0100 - 82.0] [!!!] [http/proxy_server/connection#on_close:221] Connection: Closed because: [Arachni::Reactor::Connection::Error::Closed] end of file reached
     [2018-01-04 08:54:34 +0100 - 0.0] [!!!] [http/proxy_server/connection#on_close:221] Connection: Closed because: [NilClass]
     [2018-01-04 08:54:34 +0100 - 81.9] [!!!] [http/proxy_server/connection#on_close:221] Connection: Closed because: [NilClass]
     [2018-01-04 08:54:34 +0100 - 81.9] [!!!] [http/proxy_server/connection#on_close:221] Connection: Closed because: [NilClass]
     [2018-01-04 08:54:34 +0100 - 81.6] [!!!] [http/proxy_server/connection#on_close:221] Connection: Closed because: [NilClass]
     [2018-01-04 08:54:34 +0100 - 81.7] [!!!] [http/proxy_server/ssl_interceptor#on_close:34] SSLInterceptor: Closed because: [NilClass]
     [2018-01-04 08:54:34 +0100 - 81.5] [!!!] [http/proxy_server/ssl_interceptor#on_close:34] SSLInterceptor: Closed because: [NilClass]
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [http/proxy_server#shutdown:102] ProxyServer: ...shutdown.
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [browser#shutdown:410] Worker: ...done.
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser#shutdown:420] Worker: ...shutdown complete.
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [browser#start_webdriver:1356] Worker: Starting WebDriver...
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser#spawn_phantomjs:1247] Worker: Spawning PhantomJS...
     [2018-01-04 08:54:34 +0100 - 0.0] [!] [browser#start_proxy:1338] Worker: Booting up...
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [browser#start_proxy:1340] Worker: Starting proxy...
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [http/proxy_server#start_async:61] ProxyServer: Starting...
     [2018-01-04 08:54:34 +0100 - 0.1] [!!] [http/proxy_server#start_async:78] ProxyServer: ...started at: http://127.0.0.1:19812
     [2018-01-04 08:54:34 +0100 - 0.1] [!!] [browser#start_proxy:1352] Worker: ... started proxy at: http://127.0.0.1:19812
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [browser#spawn_phantomjs:1262] Worker: Attempt #0, chose port number 44692
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [browser#spawn_phantomjs:1266] Worker: Spawning process: /sltcfrf85953/arachni/arachni-2.0dev-1.0dev/bin/../system/usr/bin/phantomjs
     [2018-01-04 08:54:34 +0100 - 0.0] [!!] [browser#spawn_phantomjs:1287] Worker: Process spawned, waiting for WebDriver server...
     [2018-01-04 08:55:34 +0100 - 60.1] [!] [browser#spawn_phantomjs:1305] Worker: Spawn timed-out.
     [2018-01-04 08:55:34 +0100 - 60.0] [!!] [browser#spawn_phantomjs:1309] Worker: 112023: Started
    PID: 112028
    [ERROR - 2018-01-04T07:54:35.291Z] GhostDriver - main.fail - {"line":85,"sourceURL":"phantomjs://code/main.js","stack":"global code@phantomjs://code/main.js:85:56"}
    
      phantomjs://platform/console++.js:263 in error
    112023: EOF
    112023: Exiting
    
     [2018-01-04 08:55:34 +0100 - 0.0] [!!] [browser#spawn_phantomjs:1317] Worker: Killing process.
     [2018-01-04 08:55:34 +0100 - 0.0] [!] [browser_cluster/worker#shutdown:162] Worker: Shutting down (wait: true) ...
     [2018-01-04 08:55:34 +0100 - 0.0] [!!] [browser_cluster/worker#shutdown:171] Worker: Signaling done.
     [2018-01-04 08:55:34 +0100 - 0.0] [!!] [browser_cluster/worker#shutdown:175] Worker: Waiting for done signal...
     [2018-01-04 08:55:34 +0100 - 0.0] [!!] [browser_cluster/worker#shutdown:179] Worker: ...done.
     [2018-01-04 08:55:34 +0100 - 0.0] [!!] [browser_cluster/worker#shutdown:181] Worker: Waiting for kill check...
     [2018-01-04 08:55:34 +0100 - 0.0] [!!] [browser_cluster/worker#shutdown:183] Worker: ...done.
     [2018-01-04 08:55:34 +0100 - 0.0] [!!] [browser_cluster/worker#shutdown:186] Worker: Killing consumer thread...
     [2018-01-04 08:55:34 +0100 - 0.0] [!] [browser_cluster/worker#run_job:126] Worker: Finished: #<Arachni::BrowserCluster::Jobs::BrowserProvider:69940590209340 callback= time= timed_out=false>
     [2018-01-04 08:55:34 +0100 - 142.4] [!] [browser_cluster#job_done:213] BrowserCluster: Job done: #<Arachni::BrowserCluster::Jobs::BrowserProvider:69940590209340 callback= time= timed_out=false>
     [~] BrowserCluster: Pending jobs: 9849
     [2018-01-04 08:55:34 +0100 - 0.1] [!] [framework/parts/audit#audit:192] BrowserCluster: Current jobs:
     [2018-01-04 08:55:34 +0100 - 0.0] [!] [framework/parts/audit#audit:194] BrowserCluster:
    

    Here is the command

    ./bin/arachni http://mywebsite.com --plugin=login_script:script=loginscript.rb --scope-exclude-pattern=deconnexion.do --scope-exclude-pattern=service.do --http-request-concurrency=1 --browser-cluster-pool-size=1 --platforms=linux,db2,java,tomcat --checks=sql_injection*,code_injection*,response_splitting,rfi,unvalidated_redirect*,xpath_injection,xss*,allowed_methods,http_put,webdav,xst,private_ip,interesting_responses,html_objects,directory_listing,mixed_resource,insecure_cookies,password_autocomplete,origin_spoof_access_restriction_bypass,cookie_set_for_parent_domain,insecure_cors_policy,insecure_cross_domain_policy_access,insecure_client_access_policy,csrf --output-debug=5 --snapshot-save-path=bin/snaps/ --http-response-max-size=50000000000
    

    I hope it could help solve the problem.

    edit : We investigated and found a problem of common port. We currently try to replace PhantomJS for a test. Will know more in some days.

    Thank you again for all your time

  4. 4 Posted by Ranus on 15 Jan, 2018 10:52 AM

    Ranus's Avatar

    Hi Tasos, we finally succeded to scan the entire website (2.5 millions lines of code) :) .
    We found a workaround by downgraded phantomjs version. From what we understood, it was a shared port issue by phantomjs and selenium.

    Unfortunatly, we had turn off scans : xss_dom* and unvalidated_redirect*
    From what I saw, with dom redirects it stayed stuck in loop.

    I can't give you access to our code/website, but if you like we could help you for debugging (beta test).

    today issue is in protocol.rb:158:in 'rescue in rbuff_fill'
    will update if we found a solution for it.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac