Optimizing for faster scans
Left to its own devices, Arachni will try to optimize itself to match any given circumstance, but there are limitations to what it can do. If a scan is taking too long, chances are that there are ways to make it go much faster by taking a couple of minutes to configure the system to closer match your needs.
- Only enable security checks that concern you
- Tailor the audit process to the platforms of the web application
- Ensure server responsiveness
- Balance RAM consumption and performance
- Reduce RAM consumption by avoiding large resources
- Don't follow redundant pages
- Increase the amount of browser workers
Only enable security checks that concern you
By default, Arachni will load all checks, which may not be what you want. If you are interested in high severity vulnerabilities or don't care for things like discovery of common files and directories, and the like, you should disable the superfluous checks.
Command-line user interface
You can enable only active checks with:
--checks=active/*
Or skip the inefficient passive ones with:
--checks=*,-common_*,-backup_*,-backdoors
More info on CLI checks options
Web user interface
Tailor the audit process to the platforms of the web application
By default, the system will fingerprint the web application in order to deduce what platforms power it, thus enabling it to only send applicable payloads (instead of everything) which results in less server stress and bandwidth usage.
However, it is a good idea to explicitly set the platforms, if you know them, so as to play it safe and get the best possible results -- especially since database platforms can't be fingerprinted prior to the audit and their payloads are a large part of the overall scan.
Command-line user interface
You can specify platforms with:
--platforms=linux,mysql,php,apache
More info on CLI platform options
Web user interface
Ensure server responsiveness
By default, Arachni will monitor the response times of the server and throttle itself down if it detects that the server is getting stressed. This happens in order to keep the server alive and responsive and maintain a stable connection to it.
However, there are times with weak servers when they die before Arachni gets a chance to adjust itself.
Command-line user interface
You can bring up the scan statistics screen by hitting Enter, in which case you'll see something like:
We can see that the server is having a hard time from the following values:
- Average: 3 requests/second
- Burst average response time 1.7590072539682537
- Burst average: 0 requests/second
- Throttled max concurrency: 2
The response times were so high (1.75 seconds) that Arachni had to throttle its HTTP request concurrency from 20 requests to 2 requests, which would result in a drastically increased scan time.
You can lower the default HTTP concurrency and try again to make sure that the server at no point gets a stressful load:
--http-request-concurrency=13
More info on the CLI HTTP request concurrency option
Web user interface
When monitoring the progress of a Scan you are presented with a small table of statistics. The relevant ones are:
- Timed out requests: 12
- Request concurrency: 2 (From the default of 20)
- Response times: 0.983 seconds
You can set the "Http req limit" option by editing the Profile you plan to use for the scan, under the "HTTP" section.
Red cells indicate trouble, yellow cells are so-and-so, green ones are good.
More info on WebUI Scan monitoring
Balance RAM consumption and performance
Most excessive RAM consumption issues are caused by large (or a lot of) HTTP requests, which need to be temporarily stored in memory in order for them to later be scheduled in a way that achieves optimal network concurrency.
To cut this short, having a lot of HTTP requests in the queue allows Arachni to be better at performing a lot of them at the same time, and thus makes better use of your available bandwidth. So, a large queue means better network performance.
However, a large queue can lead to some serious RAM consumption, depending on the website and type of audit and a lot of other factors.
As a compromise between preventing RAM consumption issues but still getting decent performance, the default queue size is set to 100.
You can adjust this number to better suit your needs depending on the situation.
Command-line user interface
You can adjust the HTTP request queue size via the --http-request-queue-size option.
More info on the CLI HTTP queue size option
Web user interface
You can set the "Http request queue size" option by editing the Profile you plan to use for the scan, under the "HTTP" section.
Reduce RAM consumption by avoiding large resources
Arachni performs a large number of analysis operations on each web page. This is usually not a problem, except for when dealing with web pages of large sizes.
If you are in a RAM constrained environment, you can configure Arachni to not download and analyze pages which exceed a certain size limit -- by default, that limit is 500KB.
Command-line user interface
You can adjust the maximm allows size of HTTP response via the --http-response-max-size option.
More info on the CLI HTTP max response size option
Web user interface
You can set the "Http response max size" option by editing the Profile you plan to use for the scan, under the "HTTP" section.
Don't follow redundant pages
A lot of websites have redundant pages like galleries, calendars, directory listings etc. which are basically the same page with the same inputs but just presenting different data. Auditing the first (or first few) of such pages is often enough and trying to follow and audit them all can sometimes result in an infinite crawl, as can be the case with calendars.
Arachni provides 2 features to help deal with that:
- Redundancy filters: Specify
patternandcounterpairs, pages matching thepatternwill be followed the amount of times specified by thecounter. - Auto-redundant: Follow URLs with the same combinations of query parameters a limited amount of times.
Command-line user interface
Web user interface
You can set these options by editing the Profile you plan to use for the scan, under the "Scope" section.
Increase the amount of browser workers
Arachni uses real browsers to support technologies such as HTML5, AJAX and DOM manipulation and perform deep analysis of client-side code.
Even though browser operations are performed in parallel using a pool of workers, the default pool size is modest and operations can be time consuming. By increasing the amount of workers in the pool, scan durations can be dramatically shortened, especially when scanning web applications that make heavy use of client-side technologies.
Finding the optimal pool size depends on the resources of your machine (especially the amount of CPU cores) and will probably require some experimentation. On average, 1-2 browsers for each logical CPU core serves as a good starting point.
Command-line user interface
More info on --browser-cluster-pool-size
Web user interface
You can set the option by editing the Profile you plan to use for the scan, under the "Browser cluster" section.