Interesting things from last week

It was an interesting last week to say the bit and just my luck I was the one there to deal with them.  I figured I’d write about them and the solutions I came up with to solve them.

The first one I encountered was yet another customer being hit by a massive surge of traffic.  The coolest part of this was the customer was not on digg, reddit or anything like that.  They were actually mentioned on tv which resulted in a lot of traffic via google.  The customer unfortunately was not prepared for the surge of traffic as their site was entirely dynamic.  Unfortunately we had no choice but to suspend their site for the sake of the server as it was not able to keep up with serving the PHP pages which were probably inefficient as well.  The customer was notified and contacted us about it and I pointed them to the very useful article Cody posted earlier: Digg.. Reddit.. Stumbleupon .  The customer read the article switched their pages to html files and the traffic was no longer an issue at all.

The second one actually may be useful for those out there using mod_python and I learned some problems that come up with it.  I can only assume there is a rise in it’s use due to Rvskin now using it for their Subversion + Trac integration system.  The problem a lot of hosts deal with is unclean restarts of httpd due to various reasons (crashing, tons of requests ect. ect.).  Well over time there are semaphore arrays sitting which are not cleaned out.  This problem becomes more apparent when running mod_python and it hit a few of our machines knocking apache offline as there was no space to load up the modules again on restart.  I cleared out the semaphore arrays then I had to come up with a solution to this problem.  Our monitoring is great and all but without the cleaning out of the semaphore arrays we’d run into this problem again.  My solution was instead of calling the standard restart script when apache fails a request we’d call a script that cleans everything out then restarts apache.  Here it is:

#!/bin/bash

ipcs -s | grep nobody | perl -e ‘while () { @a=split(/\s+/); print `ipcrm sem $a[1]`}’

/scripts/restartsrv httpd

Yep it really was that simple the ipcs portion gets rid of them then apache is restarted.  This script obviously only would work on a cpanel server due to the call to the /scripts/restartsrv.  This could however be changed to call something else to restart apache.

Well I hope someone finds this useful :)  Otherwise it’s just another rant added to our blog.