login_script.rb fails to parse the proper element in the login page

Luca's Avatar


23 Dec, 2015 11:51 AM

I am trying to use the --plugin=login_script:script='./test.rb option over a SSO login page.
I have managed to debug the whole requests issued by Arachni with both --output-debug=3 --http-proxy= and BurpSuite.
As far as I can see, even if I set in the new.rb script the SSO login page URL, somehow Arachni tries to retrieve all contents (e.g. JS, Images, icons, etc) of the page, and tries to parse for the "EmailAddress" (in the wrong response).
However instead of parsing only the html of the main webpage (as is supposed to do?!), is trying to parse also the other contents (e.g. ads-tracking, images, etc).

And of course it fails:

 [!!!] Client: Performer: #<Arachni::Browser pid=2533 last-url=nil transitions=0>
 [!!!] Client: Status: 200
 [!!!] Client: Code: ok
 [!!!] Client: Message: No error
 [!!!] Client: URL: https://www.google.com/ads/ga-audiences?v=1&aip=1&t=sr&_r=4&tid=UA-17568443-1&cid=947824252.1450869118&jid=1607298792&_v=j40&z=1391033097
 [!!!] Client: Headers:
HTTP/1.0 200 Connection established

HTTP/1.1 200 OK
Content-Type: image/gif
Date: Wed, 23 Dec 2015 11:12:49 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
X-Content-Type-Options: nosniff
Server: adclick_server
Content-Length: 42
X-XSS-Protection: 1; mode=block
Connection: close

 [!!!] Client: Parsed headers: {"Content-Type"=>"image/gif", "Date"=>"Wed, 23 Dec 2015 11:12:49 GMT", "Pragma"=>"no-cache", "Expires"=>"Fri, 01 Jan 1990 00:00:00 GMT", "Cache-Control"=>"no-cache, no-store, must-revalidate", "X-Content-Type-Options"=>"nosniff", "Server"=>"adclick_server", "Content-Length"=>"42", "X-Xss-Protection"=>"1; mode=block", "Connection"=>"close"}
 [!!!] Client: ------------
 [!!] ProxyServer: [perform_proxy_request] Completed: GET /ads/ga-audiences?v=1&aip=1&t=sr&_r=4&tid=UA-17568443-1&cid=947824252.1450869118&jid=1607298792&_v=j40&z=1391033097 HTTP/1.1
[2015-12-23 13:12:49] DEBUG CONNECT: 375 byte from
[2015-12-23 13:12:50] DEBUG close:
[2015-12-23 13:12:50] DEBUG CONNECT: 102 byte from
[2015-12-23 13:12:50] DEBUG CONNECT closed
[2015-12-23 13:12:50] DEBUG close:

[-] Login script: [Watir::Exception::UnknownObjectException] unable to locate element, using {:id=>"EmailAddress", :tag_name=>"input or textarea", :type=>"(any text type)"}

--- test.rb ---
browser.goto 'https://SSO.foo.com/SSO/Account/Login?ReturnUrl=%2f%3fwa%3dwsignin1.0%26wtrealm%3dhttps%253a%252f%252ftarget.foo.com'
browser.text_field( id: 'EmailAddress' ).set '[email blocked]'
browser.text_field( id: 'Password' ).set 'XXXXXXX'
browser.goto 'https://target.foo.com'
--- test.rb ---

Any suggestions how I can fix the issue?

  1. Support Staff 1 Posted by Tasos Laskos on 23 Dec, 2015 12:15 PM

    Tasos Laskos's Avatar

    It doesn't really work that way, the browser just loads whatever the page says it needs, that includes resources like JS and other assets.
    The form is then located in the loaded page.

    At first I'd suggest trying the nightlies and we can go from there.


  2. 2 Posted by Luca on 23 Dec, 2015 01:00 PM

    Luca's Avatar

    Firstly, Thanks for the uber-fast reply.
    I have just tried the dev version suggested, and I get the same behaviour.
    Considering that I am rather newbie about arachni's core, while inspecting the debug 3 level, I can see this:
    [!!] [browser#request_handler:1478] Browser: Request: https://SSO.foo.com/SSO/Account/Login?ReturnUrl=%2f%3fwa%3dwsignin1.0%26wtrealm%3dhttps%253a%252f%252ftarget.foo.com [!!] [browser#ignore_request?:1597] Browser: Checking: https://SSO.foo.com/SSO/Account/Login?ReturnUrl=%2f%3fwa%3dwsignin1.0%26wtrealm%3dhttps%253a%252f%252ftarget.foo.com

    [!!] [browser#ignore_request?:1600] Browser: Allow: Scope enforcement disabled.

    [!!] [browser#request_handler:1520] Browser: Request can proceed to origin.

    [!!!] [http/proxy_server/connection#handle_request:99] SSLInterceptor: -- Handler approves, running... [!!!] [http/proxy_server/connection#handle_request:107] SSLInterceptor: -- ...completed in 0.113469: HTTP/1.0 200 Connection established [!!!] [http/proxy_server/connection#handle_request:120] SSLInterceptor: Processed request. [!!!] [http/proxy_server/connection#handle_response:131] SSLInterceptor: Preparing response. [!!!] [http/proxy_server/connection#handle_response:141] SSLInterceptor: -- Has special handler: #<Proc:0xa900a24@/root/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-879f032cc945/lib/arachni/browser.rb:188> [!!] [browser#response_handler:1550] Browser: Got response: https://SSO.foo.com/SSO/Account/Login?ReturnUrl=%2f%3fwa%3dwsignin1.0%26wtrealm%3dhttps%253a%252f%252ftarget.foo.com [!!] [browser#response_handler:1567] Browser: Injected custom JS.

    [!!] [browser#response_handler:1579] Browser: Outside of domain scope, will not store.

    Might be that I have to set the target in scope (in my case https://SSO.foo.com for login and https://target.foo.com for the scanning) by using some options in order for Arachni to parse only the SSO login page and stop trying parsing the wrong assets?

    Here below another snippet of the logs, exactly where the parser fails to grep for EmailAddress text field (as mentioned in my first post).

    [!!!] [http/proxy_server/connection#initialize:33] SSLInterceptor: Starting new connection: 89134040
    [!!!] [http/proxy_server/connection#initialize:38] SSLInterceptor: Incoming request.
    [!!!] [http/proxy_server/connection#initialize:51] SSLInterceptor: Request received: GET /ads/ga-audiences?v=1&aip=1&t=sr&_r=4&tid=UA-17568443-1&cid=1419214905.1450873664&jid=467562653&_v=j40&z=295020460
    [!!!] [http/proxy_server/connection#handle_request:86] SSLInterceptor: Processing request.
    [!!!] [http/proxy_server/connection#handle_request:90] SSLInterceptor: -- Has special handler: #<Proc:0xa900a38@/root/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-879f032cc945/lib/arachni/browser.rb:185>
    [!!] [browser#request_handler:1478] Browser: Request: https://www.google.com/ads/ga-audiences?v=1&aip=1&t=sr&_r=4&tid=UA-17568443-1&cid=1419214905.1450873664&jid=467562653&_v=j40&z=295020460
    [!!] [browser#ignore_request?:1597] Browser: Checking: https://www.google.com/ads/ga-audiences?v=1&aip=1&t=sr&_r=4&tid=UA-17568443-1&cid=1419214905.1450873664&jid=467562653&_v=j40&z=295020460
    [!!] [browser#ignore_request?:1600] Browser: Allow: Scope enforcement disabled.
    [!!] [browser#request_handler:1520] Browser: Request can proceed to origin.
    [!!!] [http/proxy_server/connection#handle_request:99] SSLInterceptor: -- Handler approves, running...
    [!!!] [http/proxy_server/connection#handle_request:107] SSLInterceptor: -- ...completed in 0.469363: HTTP/1.0 200 Connection established
    [!!!] [http/proxy_server/connection#handle_request:120] SSLInterceptor: Processed request.
    [!!!] [http/proxy_server/connection#handle_response:131] SSLInterceptor: Preparing response.
    [!!!] [http/proxy_server/connection#handle_response:141] SSLInterceptor: -- Has special handler: #<Proc:0xa900a24@/root/arachni-2.0dev-1.0dev/system/gems/bundler/gems/arachni-879f032cc945/lib/arachni/browser.rb:188>
    [!!] [browser#response_handler:1550] Browser: Got response: https://www.google.com/ads/ga-audiences?v=1&aip=1&t=sr&_r=4&tid=UA-17568443-1&cid=1419214905.1450873664&jid=467562653&_v=j40&z=295020460
    [!!] [browser#response_handler:1567] Browser: Injected custom JS.
    [!!] [browser#response_handler:1573] Browser: Asset detected, will not store.
    [!!!] [http/proxy_server/connection#handle_response:169] SSLInterceptor: Sending response.
    [!!!] [http/proxy_server/connection#on_flush:200] SSLInterceptor: Response sent.
    [!!!] [http/proxy_server/tunnel#on_read:47] Tunnel: <- Forwarding 369 bytes to client.
    [-] [utilities#exception_jail:428] Session: [Watir::Exception::UnknownObjectException] unable to locate element, using {:id=>"EmailAddress", :tag_name=>"input or textarea", :type=>"(any text type)"}
  3. Support Staff 3 Posted by Tasos Laskos on 25 Dec, 2015 02:59 AM

    Tasos Laskos's Avatar

    Sorry for the delay, your response got caught by the spam filter.

    Like I mentioned previously, Arachni is not parsing the wrong assets, it's behaving just like a browser should and loading what the page says it needs.

    About your issue, you can try capturing screenshots of the relevant pages to ensure that they are loading properly, for that see: http://watirwebdriver.com/screenshots/

    There can be a few reasons for the missing form, maybe the relevant form isn't rendered by the time the DOM says it's ready ready and you may need to tell the browser to wait for it to appear or there may be a bug somewhere else causing the page to fail to load.

    Let's start with the screenshots though.


  4. 4 Posted by Luca on 25 Dec, 2015 11:08 AM

    Luca's Avatar

    Please consider I am not able to access the environment now.

    However, I was experimenting with WATIR and I have developed a standalone version of the test.rb script which uses also screenshots and save cookies in order to debug the issue.
    Overall the standalone script works like charm (get logged in, cookies, right screenshots, etc.).

    I will try to set some sleep() in the script to see if a problem of DOM rendering (as you just wrote).

    Besides those sleep()... Do you reccomend to set other delays somewhere in Arachni sources? Or sleep() will be enough?

    BR and merry Xmas,

  5. Support Staff 5 Posted by Tasos Laskos on 26 Dec, 2015 08:03 AM

    Tasos Laskos's Avatar

    Since your standalone script worked then the issue probably is in Arachni.
    I'm afraid I won't be able to be of further assistance unless I can try the script myself in order to debug the issue.
    Any chance you can send me the details via e-mail? tasos[dot]laskos[at]arachni-scanner.com

    Happy holidays to you as well. :)

  6. 6 Posted by Luca on 26 Dec, 2015 08:18 AM

    Luca's Avatar

    Any pgp pubkey available? :)


  7. Support Staff 7 Posted by Tasos Laskos on 19 Jan, 2016 12:42 PM

    Tasos Laskos's Avatar

    Solved, the webapp needed a common user-agent to be set.

  8. Tasos Laskos closed this discussion on 19 Jan, 2016 12:42 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac