Home → Problems →

Arachni isn't crawling

• Edit

chris

11 Aug, 2014 11:35 AM

Hi!

I'm trying to use arachni but none site is being crawled/scanned.

I always have the output below, any site I try.:

[~] Crawling, discovered 0 pages and counting.

[~] Sent 1 requests. [~] Received and analyzed 1 responses. [~] In 00:00:01 [~] Average: 0 requests/second.

And I'm issuing the command:

bin/arachni -fv http://my-site.com --report=afr:outfile=my-site.com.afr

I have arachni installed locally (both git clone and MacOS package) without apparent errors, but it's not working.

Am I doing something wrong?

Thanks!

1. Support Staff 1 Posted by Tasos Laskos on 11 Aug, 2014 11:39 AM

   

   Can you please enable the --debug flag and show me the output?

   Cheers

   • Edit

2. Support Staff 2 Posted by Tasos Laskos on 11 Aug, 2014 11:39 AM

   

   Also, shall I close http://support.arachni-scanner.com/discussions/problems/2055-arachn... ?

   • Edit

3. 3 Posted by chris on 11 Aug, 2014 12:57 PM

   

   Hello!

   Yes, you can close the previous issue. Let’s work with the last one.

   And here the debug output. This is a test in a local Rails application using a clone from github, experimental branch. Same thing happens for any site that I try.

   The MacOS binary package started to work. I don’t know what happened. I was off to lunch, my computer entered in sleep mode… and now it’s working.
   

   $ bin/arachni -fv --debug http://localhost:3000 --report=afr:outfile=localhost.afr
   
   
   
   
   [~] No modules were specified.
   [~]  -> Will run all mods.
   
   
   
   
   [~] No audit options were specified.
   [~]  -> Will audit links, forms and cookies.
   
   
   
   
   [*] Initialising...
   [*] Waiting for plugins to settle...
   [!] HTTP: ------------
   [!] HTTP: Queued request.
   [!] HTTP: ID#: 0
   [!] HTTP: URL: http://localhost:3000/
   [!] HTTP: Method: get
   [!] HTTP: Params:
   [!] HTTP: Headers: {"Accept"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding"=>"gzip, deflate", "User-Agent"=>"Arachni/v1.0dev"}
   [!] HTTP: Train?: false
   [!] HTTP: ------------
   [!] HTTP: ------------
   [!] HTTP: Got response for request ID#: 0
   [!] HTTP: Status: 200
   [!] HTTP: Error msg: No error
   [!] HTTP: URL: http://localhost:3000/
   [!] HTTP: Headers:
   HTTP/1.1 200 OK
   X-Frame-Options: SAMEORIGIN
   X-Xss-Protection: 1; mode=block
   X-Content-Type-Options: nosniff
   X-Ua-Compatible: chrome=1
   Content-Type: text/html; charset=utf-8
   Etag: "4cc60a91f58fe0cdbfd7c58030c83173"
   Cache-Control: max-age=0, private, must-revalidate
   X-Request-Id: c84ac0de-cd7e-4521-b5bb-503927ec896b
   X-Runtime: 0.011453
   Server: WEBrick/1.3.1 (Ruby/2.1.1/2014-02-24)
   Date: Mon, 11 Aug 2014 12:41:12 GMT
   Content-Length: 9949
   Connection: Keep-Alive
   
   
   
   
   [!] HTTP: Parsed headers: {"X-Frame-Options"=>"SAMEORIGIN", "X-Xss-Protection"=>"1; mode=block", "X-Content-Type-Options"=>"nosniff", "X-Ua-Compatible"=>"chrome=1", "Content-Type"=>"text/html; charset=utf-8", "Etag"=>""4cc60a91f58fe0cdbfd7c58030c83173"", "Cache-Control"=>"max-age=0, private, must-revalidate", "X-Request-Id"=>"c84ac0de-cd7e-4521-b5bb-503927ec896b", "X-Runtime"=>"0.011453", "Server"=>"WEBrick/1.3.1 (Ruby/2.1.1/2014-02-24)", "Date"=>"Mon, 11 Aug 2014 12:41:12 GMT", "Content-Length"=>"9949", "Connection"=>"Keep-Alive"}
   [!] HTTP: ------------
   [*] Spider: [HTTP: 200] http://localhost:3000/
   [!]
   [!] Waiting on the following (6) plugins to finish:
   [!] autothrottle, healthmap, discovery, timing_attacks, uniformity, resolver
   [!]
   [*] Resolver: Resolving hostnames...
   [*] Resolver: Done!
   [!]
   [!] Waiting on the following (5) plugins to finish:
   [!] healthmap, discovery, timing_attacks, uniformity, resolver
   [!]
   
   
   
   
   [*] Dumping audit results in 'localhost.afr'.
   [*] Done!
   
   
   
   
   [~] 0.0% [=>                                                             ] 100%
   [~] Est. remaining time: --:--:--
   
   
   
   
   [~] Crawling, discovered 1 pages and counting.
   
   
   
   
   [~] Sent 1 requests.
   [~] Received and analyzed 1 responses.
   [~] In 00:00:01
   [~] Average: 0 requests/second.
   
   
   
   
   [~] Burst response time total    0
   [~] Burst response count total   0
   [~] Burst average response time  0
   [~] Burst average                0 requests/second
   [~] Timed-out requests           0
   [~] Original max concurrency     20
   [~] Throttled max concurrency    20
   

   • Edit

4. Support Staff 4 Posted by Tasos Laskos on 11 Aug, 2014 01:05 PM

   

   I can't guarantee that the code from the repo will work in your env but the self-contained package should work so let's just stick with that.

   What's the state of it now? You mentioned it started working, was it not in the past?

   • Edit

5. 5 Posted by chris on 11 Aug, 2014 01:25 PM

   

   My Ruby environment is simple.

   Ruby 2.1.1
   RVM
   Isolated gemset to Arachni
   

   I've downloaded the package because my clone wasn't working. I'm pretty sure that the package has not worked previously. But now is working.

   • Edit

6. Support Staff 6 Posted by Tasos Laskos on 11 Aug, 2014 01:34 PM

   

   Arachni has a lot of system dependencies as well (via its gems mostly) that need to be met. The Wiki is very clear: https://github.com/Arachni/arachni/wiki/Installation#source-based

   And because I can't guarantee that the codebase by itself will work everywhere, I've made available packages which include the right dependencies in the right configuration.

   Realistically, it'd be impossible for me to support anything other than the packages.

   As for the package not working in the past, I'm not sure what to make of that, seems like an env issue that somehow went away when your OS changes stated.
   I haven't gotten any complaints regarding the packages.

   If the package stops working again please let me know, otherwise everything seems OK.

   • Edit

7. Tasos Laskos closed this discussion on 11 Aug, 2014 01:34 PM.

8. chris re-opened this discussion on 11 Aug, 2014 01:49 PM

9. 7 Posted by chris on 11 Aug, 2014 01:49 PM

   

   With the package seems to be everything ok.

   And two things that I've noted.

   1 - Redirected url's are failling, i.e:
   http://my-site.com is failling
   http://www.my-site.com is ok
   both urls point to the same site.

   2 - Clone with Ruby 2.1.1 is failling. Running with Ruby 1.9.3 is ok.
   /Users/chris/dev/ruby/workspace/arachni/lib/arachni/ruby/object.rb:31: warning: Arachni::Element::Cookie#respond_to?(:marshal_dump) is old fashion which takes only one parameter /Users/chris/dev/ruby/workspace/arachni/lib/arachni/element/cookie.rb:340: warning: respond_to? is defined here /Users/chris/dev/ruby/workspace/arachni/lib/arachni/ruby/object.rb:31: warning: Arachni::Element::Cookie#respond_to?(:_dump) is old fashion which takes only one parameter /Users/chris/dev/ruby/workspace/arachni/lib/arachni/element/cookie.rb:340: warning: respond_to? is defined here /Users/chris/dev/ruby/workspace/arachni/lib/arachni/ruby/object.rb:31: warning: Arachni::Element::Cookie#respond_to?(:marshal_dump) is old fashion which takes only one parameter /Users/chris/dev/ruby/workspace/arachni/lib/arachni/element/cookie.rb:340: warning: respond_to? is defined here /Users/chris/dev/ruby/workspace/arachni/lib/arachni/ruby/object.rb:31: warning: Arachni::Element::Cookie#respond_to?(:_dump) is old fashion which takes only one parameter /Users/chris/dev/ruby/workspace/arachni/lib/arachni/element/cookie.rb:340: warning: respond_to? is defined here /Users/chris/dev/ruby/workspace/arachni/lib/arachni/ruby/object.rb:31: warning: Arachni::Element::Cookie#respond_to?(:marshal_dump) is old fashion which takes only one parameter /Users/chris/dev/ruby/workspace/arachni/lib/arachni/element/cookie.rb:340: warning: respond_to? is defined here /Users/chris/dev/ruby/workspace/arachni/lib/arachni/ruby/object.rb:31: warning: Arachni::Element::Cookie#respond_to?(:_dump) is old fashion which takes only one parameter /Users/chris/dev/ruby/workspace/arachni/lib/arachni/element/cookie.rb:340: warning: respond_to? is defined here

   • Edit

10. Support Staff 8 Posted by Tasos Laskos on 11 Aug, 2014 01:55 PM

    

    1. That's really not the same thing, the www subdomain does matter, you'll have to use the -f flag for that to work.

    2. Yep, I'm aware. I'm now preparing the new release which should work on pretty much anything (Well, there are some issues with Rubinius and JRuby but I hope to work with the maintainers to fix them).

    Cheers

    • Edit

11. Tasos Laskos closed this discussion on 11 Aug, 2014 01:55 PM.

12. chris re-opened this discussion on 11 Aug, 2014 02:01 PM

13. 9 Posted by chris on 11 Aug, 2014 02:01 PM

    

    Ok.

    From my side we can close this discussion/issue.

    Thanks. You're a very handy person.

    • Edit

14. Support Staff 10 Posted by Tasos Laskos on 11 Aug, 2014 02:04 PM

    

    No problem, if anything else comes up please let me know.

    Closing.

    Cheers

    • Edit

15. Tasos Laskos closed this discussion on 11 Aug, 2014 02:04 PM.