At work we have one VM which hosts all our project management software like Git, SVN, Trac and Bugzilla. However, recently it has been taking a hammering and essentially crashing. The issue was it was running out of RAM, and swapping like crazy. A little investigation into the situation uncovered that there were some Trac 0.10 CGI processes which were using >500MB of memory.
Using the ps ewww -p <pid> command to look at the environment variables of the processes we determined that each of them was serving the same kind of request, SVN changesets. And essentially all of them were being hit by spiders.
Essentially the issue was that the spiders were managing to create a large number of overlapping changeset requests, which were chewing up all the RAM on the VM. Increasing the amount of RAM wouldn’t only offset the problem. So, we now just block the spiders from the changeset areas of our tracs (robots.txt):
User-agent: *
Request-rate: 1/5
Disallow: /projects/data-activities/changeset
...In the weeks since then we’ve had no issues. Besides, who wants their gaffs showing up on google ;)
Interestingly we’ve not seen the same thing happen with Trac 0.11, so perhaps the issue is resolved now.