Arachni spidering REST APIs

Zukky's Avatar

Zukky

16 May, 2017 09:06 AM

Hi,

First of all, thank you for your quick, swift responses, they have been very helpful and it is most appreciated.

Me and my team whom are trialling Arachni are having the same problem, we want it to be able to spider into subdomains. We are using arachni directly for REST API endpoints, i.e. http://test-endpoint:8081/, we want it to go into http://test-endpoint:8081/test1, test2, test3, so on and so forth. I am having trouble running such a scan because when I do the following via Arachni CLI - "arachni --scope-include-subdomains http://test-endpoint:8081/" - I feel that it only scans as far as the endpoint URL provided and doesn't spider through to other sub endpoints within. The reason we feel this way, when I do a scan against http://test-endpoint:8081/test1, the results are different than just doing the former endpoint. How do we get Arachni to spider through the URLs provided and feel confident that it has spidered the endpoint and all of its subtree?

Thank you,

Zukky

  1. 1 Posted by Zukky on 16 May, 2017 10:03 AM

    Zukky's Avatar

    Ok, what I noticed, the endpoint that is provided, if it returns a 404, Arachni doesn't spider it.

    http://test-endpoint:8081/ has nothing, so we return a 404. It makes logical sense to start the scan from this start point because it can then spider into all areas. Whereas, starting from test1 will mean that it may not reach test2/test3/ so on. How can I still get Arachni to run even when response code is 404?

  2. Support Staff 2 Posted by Tasos Laskos on 16 May, 2017 10:12 AM

    Tasos Laskos's Avatar

    It has nothing to do with the response code, are you sure that there are paths that lead to the other resources from the target you provided?

  3. 3 Posted by Zukky on 16 May, 2017 10:24 AM

    Zukky's Avatar

    [HTTP: 404] http://app:8081/ [~] Analysis resulted in 0 usable paths.

    I think it is todo with response code, if it returns a 200 (I changed the "/" resource to return a 200) it spiders and continues the scan.

  4. Support Staff 4 Posted by Tasos Laskos on 16 May, 2017 10:28 AM

    Tasos Laskos's Avatar

    Are you absolutely certain that only the response code is different? Did you check via a browser?

  5. Support Staff 5 Posted by Tasos Laskos on 16 May, 2017 11:07 AM

    Tasos Laskos's Avatar

    Also, if http://test-endpoint:8081/has nothing it makes no sense to provide that as a starting point since it exposes no other paths to follow.

  6. 6 Posted by Zukky on 16 May, 2017 11:25 AM

    Zukky's Avatar

    Yeah i'm sure, I changed the microservice to return a 200 instead of a 404 on the "/" resource. And it spiders but this is hacky, I don't want to change the application to make it scannable. Is the original behaviour of Arachni to not run on a 404?

    I thought it would've been able to spider into other parts of the url, I'm not too sure how the underlying implementation works but I thought it would just pump in different url endings /AB, /AC etc etc... even if a 404 was present. I think you're right, if there is a 404, it doesn't expose any usable paths, thus, Arachni only scans one page. I wish this was more explicitly stated, maybe it is just my understanding.

    How should I do this then? Is there a way of automatically spidering into URL's from a 404 "/"?

  7. 7 Posted by Zukky on 16 May, 2017 11:26 AM

    Zukky's Avatar

    I can start from "/health" which returns a 200 but I fear it may not be able to reach other sibling resources.

  8. Support Staff 8 Posted by Tasos Laskos on 16 May, 2017 12:07 PM

    Tasos Laskos's Avatar

    The response code is irrelevant, if the response has paths that can be followed then they will be followed.
    I think what confused you are the checks that discover directories, those aren't part of the crawl and will not run on non-200 codes, so you will see a difference in behavior.

    The important part is that the scan will not run if the seed URLs hasn't got any usable paths, you either need to provide a different target URL or extend the paths manually.

  9. 9 Posted by Zukky on 16 May, 2017 04:23 PM

    Zukky's Avatar

    That's what i'm doing now, i'm creating a file with the paths it should go under. Can regex expressions be used in the file or relative paths? Or must they be simple absolute paths of the URLs?

  10. Support Staff 10 Posted by Tasos Laskos on 17 May, 2017 09:51 AM

    Tasos Laskos's Avatar

    Either relative or absolute paths, if they're relative they're going to be converted to absolute using the target URL as a reference.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac