Home → Suggestions →

Array of proxys, array of user agent in argument

• Edit

Beunwa

05 Oct, 2012 02:28 PM

Hi,

I'm still on my way hacking arachni-scanner for seo audit. One common thing in such audits is changing user-agent and ip to test the website reaction.

As I can see it in the options and http class there is no way to choose a proxy or user agent dynamicly.

I think that providing a path to a proxy list / user agent list in addition of the classical arg --proxy and --user-agent could be a good thing.

Then in http class detect if there is an array of proxy / user-agent and then use a array.sample to change the proxy (or use an array rotation)

I know that this will cause trouble with the sessions but maybe there is a simple solution.

1. Support Staff 1 Posted by Tasos Laskos on 05 Oct, 2012 02:56 PM

   

   I've got good news and bad news...

   Bad news:

   • I won't support that functionality.

   Good news:

   • The global HTTP options actually take effect per request.
   • You can intercept and modify ALL queued HTTP requests, thus you can alter the user-agent and proxy options for any request, even for ones that other components perform.

   So basically you can do what you want from your plugin using something like:

   http.add_on_queue do |request, _|
       request.proxy = 'my.proxy.com:8080'
       request.headers['User-Agent'] = 'MyUA/0.1'
   end
   
   http.get 'http://test.com' do |response|
       p response.request.proxy
       #=> "my.proxy.com:8080"
   
       p response.request.headers['User-Agent']
       #=>"MyUA/0.1"
   end
   
   http.run
   

   Sound good?

   • Edit

2. Support Staff 2 Posted by Tasos Laskos on 05 Oct, 2012 02:58 PM

   

   Forgot to mention that it won't affect the session as cookies will remain intact -- unless the web devs decided to depend on the IP address or the User-Agent for session maintenance for some reason.

   • Edit

3. Tasos Laskos closed this discussion on 05 Oct, 2012 02:58 PM.

4. Beunwa re-opened this discussion on 05 Oct, 2012 03:06 PM

5. 3 Posted by Beunwa on 05 Oct, 2012 03:06 PM

   

   that's good news :)

   • Edit

6. Tasos Laskos closed this discussion on 05 Oct, 2012 03:07 PM.

7. beunwa re-opened this discussion on 05 Nov, 2012 01:23 PM

8. 4 Posted by beunwa on 05 Nov, 2012 01:23 PM

   

   This work fine a month ago, but now it seems like the http.add_on_queue was never called :( .
   I use the distributed crawl branch, maybe its a clue.
   I call arachni this way :
   bundle exec arachni http://mysite.tld --module=- -g --plugin=submarine

   This is my plugin :

   ` class Arachni::Plugins::Submarine < Arachni::Plugin::Base

   @uas = []
   @proxys = []
   
   def run      
     @uas = open('uas.txt').read.split
     @proxys = open('mpp.txt').read.split
   
     #check la proxy validity
     @proxys.each do |pro|
       pr = pro.split(':')
   
       proxy_class = Net::HTTP::Proxy(pr[0], pr[1], pr[2], pr[3])
   
       proxy_class.start('http://www.google.com') {|http|
         response = http.head('/index.html')
         @proxys.delete(pro) if response.code.to_i != 200          
       }
     end
   
     http.add_on_queue do |request, _|
       pr = @proxys.sample  
       request.proxy = pr
       request.headers['User-Agent'] = @uas.sample
     end
   
     http.run
   end
   
   def self.info
       {
           name:        'Submarine',
           description: %q{intent to keep under the radar by changing the ua and ip randomly, warning this plugin must be load the first},
           author:      'Benoit Chevillot <[email blocked]>',
           version:     '0.1'
       }
   end
   

   end
   `

   • Edit

9. Support Staff 5 Posted by Tasos Laskos on 05 Nov, 2012 01:37 PM

   

   Nice catch, will sort it out today.

   • Edit

10. Support Staff 6 Posted by Tasos Laskos on 05 Nov, 2012 04:04 PM

    

    First of all, your plugin had some issues, here's a cleaned up version:

    class Arachni::Plugins::Submarine < Arachni::Plugin::Base
        # run it on all instances
        is_distributable
    
        def prepare
            # pause the framework while we setup our hooks
            framework.pause
    
            @uas     = open('uas.txt').read.split
            @proxies = open('mpp.txt').read.split
    
            # check la proxy validity
            @proxies.dup.each do |pro|
                begin
                    Net::HTTP::Proxy( *pro.split( ':' ) ).
                        start( 'http://www.google.com' ) do |http|
                            next if http.head('/index.html').code.to_i == 200
                            raise "Proxy #{pro} seems dead."
                    end
                # if we got here either the proxy server is dead or something
                # more horrible has happened, bottom line is the proxy needs to
                # be removed.
                rescue
                    @proxies.delete( pro )
                end
            end
        end
    
        def run
            http.add_on_queue do |request, _|
                request.proxy = @proxies.sample
                request.headers['User-Agent'] = @uas.sample
            end
        end
    
        def cleanup
            framework.resume
        end
    
        def self.info
            {
                name:        'Submarine',
                description: %q{intent to keep under the radar by changing the ua and ip randomly, warning this plugin must be load the first},
                author:      'Benoit Chevillot <[email blocked]>',
                version:     '0.1',
                priority:    0 # run first
            }
        end
    
    end
    

    Secondly, did you expect to run a distributed crawl using the command you mentioned?
    Because that won't happen, the arachni CLI will just run a simple single-node direct scan.
    To perform a distributed use this code example: https://github.com/Arachni/arachni/issues/207#issuecomment-10066220

    Just to be clear, I'm not sure how it'll work with a distributed scan because I haven't gotten to that point yet.
    And to be even clearer I didn't actually test the code I pasted either but it looks about right.

    • Edit

11. 7 Posted by Beunwa on 05 Nov, 2012 04:44 PM

    

    Nice refactoring, your my master !

    2 details :
    framework.resume is needed at the end of prepare, if not the crawl never start

    start( 'www.google.com' ) instead of start( 'http://www.google.com' ) , if not it give error 400 (dunno why, didnot take the time to check)

    For the moment I just use it like before with just a single node, and with the 2 above adjustement it work like a charm !

    Feel free to distribute this plugin if you want, it can be very usefull (it is for me)

    • Edit

12. Support Staff 8 Posted by Tasos Laskos on 05 Nov, 2012 04:58 PM

    

    Lol my bad, that method should have been called clean_up so do put framework.resume there to be safe, otherwise you might miss some requests.

    As for these sort of plugins working properly with the new crawler I'll take care of that tomorrow.

    However, I don't like to encourage people to use proxies with Arachni so I won't be including it in the repo.

    • Edit

13. Support Staff 9 Posted by Tasos Laskos on 06 Nov, 2012 04:40 PM

    

    OK, fixed the problem, plugins are now distributed properly even during the crawl phase.
    Of course, the code is still considered unstable so if you come across issues please let me know.

    • Edit

14. Tasos Laskos closed this discussion on 06 Nov, 2012 04:40 PM.