Home → Problems →

Grid distributing

• Edit

user021

17 Jul, 2013 11:31 AM

I did ran this scan once : ./arachni_multi --server localhost:9184 --reroute-to-logfile --audit-link --audit-forms --audit-headers --audit-cookies --spawns=1 --modules=trainer,sqli --http-timeout=28000 --link-count=11111 --auto-redundant=2 --exclude='jpg|mp4|js|css|pdf' --http-req-limit=30 'http://www.bcbg.com/'

i was not at pc, when came back for some reason was stuck at "Crawling" "Discovered 10834 pages" network usage 0, CPU usage 0, i look at logs, in one, last comment was "done crawling at least for now" while on other1 there wasn't anything interesting at end of it, so i thought to reproduce it, this time however, it did not stuck but i noticed something:

Crawl finished, progressing to distribution of audit workload.
[Wed Jul 17 07:05:54 2013] [info] Found 10315 pages with a total of 88097 elements.
[Wed Jul 17 07:11:09 2013] [info] /tmp/arachni-59592 (Master)
[Wed Jul 17 07:11:09 2013] [info] * 5158 URLs
[Wed Jul 17 07:11:09 2013] [info] * 38690 elements
[Wed Jul 17 07:11:09 2013] [info] /tmp/arachni-instance-slave-10962 (Slave)
[Wed Jul 17 07:11:09 2013] [info] * 5157 URLs
[Wed Jul 17 07:11:09 2013] [info] * 38689 elements

88097 divided by 2 = 44048.5, what is goin on ? i was cracking some hashes in the background so the CPU activity was almost all time 100% but i doubt that could be the reason for this

1. Support Staff 1 Posted by Tasos Laskos on 17 Jul, 2013 12:06 PM

   

   The 88097 include duplicate elements, during the distribution they are deduplicated. As for the stuck thing...no idea. What was at the end of the other log?

   • Edit

2. 2 Posted by user021 on 17 Jul, 2013 12:15 PM

   

   nothing, just usual crawled pages, ill keep testing, an update though, is this memory usage normal? just making sure

   • BackBox_x64-2013-07-17-15-08-29.png 307 KB
   • Edit

3. Support Staff 3 Posted by Tasos Laskos on 17 Jul, 2013 12:18 PM

   

   Holy crap, no not really.

   • Edit

4. 4 Posted by user021 on 17 Jul, 2013 12:21 PM

   

   I have no idea how to inspect it ...if is uses more memory than normal or what's the bug that makes it use more, could i ask how you do it?xD

   • Edit

5. Support Staff 5 Posted by Tasos Laskos on 17 Jul, 2013 03:23 PM

   

   I can't really scan a site without permission so not sure how to repro this either...Did that memory usage happen during the crawl at least? I wouldn't want to audit them.

   • Edit

6. 6 Posted by user021 on 17 Jul, 2013 03:28 PM

   

   update: ran the scan gain, this time is stuck again (at crawling after limit of crawled URLs was reached) and CPU activity is null
   ps: stopped any application that demands high CPU usage on host so it doesn't interfere. Yes, right now the memory usage seem normal (around 130mb each) but when it did not get stuck, it jumped to 1 gb during the audit,don't worry ill look more into it trying to find out exactly what's causing it as soon as you fix the "stuck" thing

   • Edit

7. Support Staff 7 Posted by Tasos Laskos on 17 Jul, 2013 03:52 PM

   

   Ok so I'll least be able to repro that one and hopefully fix it and we'll go from there.

   • Edit

8. 8 Posted by user021 on 17 Jul, 2013 07:50 PM

   

   Let me know if it get stuck for you too or if not...how could i debug it hmm

   • Edit

9. 9 Posted by user021 on 18 Jul, 2013 12:55 PM

   

   update:about the memory usage, i think is nothing wrong with it, considering that i enabled trainer and it pokes a specific URL that contains a long list of images like this one

   http://www.bcbg.com/NEW-MARKDOWNS/sale-new-markdowns,default,sc.html?cm_sp=_arachni_trainer_949df831534f30be2e6f34a3ebefc71adc9acd0cc8c679457c28ba963f1cef2b

   i did run the scan with --exclude='jpg|mp4|js|css|pdf', did open images and they seem to be jpg but when i hover mouse above one of them i get a link like this

   http://www.bcbg.com/Guy-Tuxedo-Jacket/ZBA4G077-101,default,pd.html?dwvar_ZBA4G077-101_color=101&cgid=sale-new-markdowns

   i was wondering if there's any way to stop Arachni caching useless images into memory

   • Edit

10. Support Staff 10 Posted by Tasos Laskos on 18 Jul, 2013 01:05 PM

    

    It doesn't, at least not intentionally.

    • Edit

11. Support Staff 11 Posted by Tasos Laskos on 18 Jul, 2013 02:54 PM

    

    Can you remove the content_types plugin out of plugin/defaults/ and retry? It keeps some response data around and that could be making it hard on the garbage collector, leading to keeping a lot of response data in memory.

    • Edit

12. 12 Posted by user021 on 18 Jul, 2013 03:08 PM

    

    Sure, doing it now.

    • Edit

13. 13 Posted by user021 on 18 Jul, 2013 03:43 PM

    

    apparently there's no effect, memory usage still risses (analized 1000 responses and the ruby process uses right now 280mb and goin up)

    ./arachni --audit-link --audit-forms --plugin=rescan:afr=a.afr --audit-headers --audit-cookies --modules=trainer --http-timeout=28000 --link-count=11111 --auto-redundant=2 --http-req-limit=30 'http://www.bcbg.com/'

    ps: used the afr report this time to avoid crawling again and trainer module alone

    • a.afr 1.49 MB
    • Edit

14. 14 Posted by user021 on 18 Jul, 2013 03:47 PM

    

    I can make new thread on github for this one if you want. since it doesn't affect the grid alone

    • Edit

15. Support Staff 15 Posted by Tasos Laskos on 18 Jul, 2013 03:49 PM

    

    Yeah that'd be helpful thanks.

    • Edit

16. Support Staff 16 Posted by Tasos Laskos on 18 Jul, 2013 08:22 PM

    

    Closing this since the discusssion has been moved to GitHub.

    • Edit

17. Tasos Laskos closed this discussion on 18 Jul, 2013 08:22 PM.

18. Tasos Laskos closed this discussion on 22 Sep, 2013 02:09 PM.