Software Updates Are Never Fun

As a web host we’re expected to keep things stable, but at the same time introduce new features which means updating software.  This can be problematic as we do not actually develop any of the major software we use which means we are reliant on the developers of the software.  The majority of the developers of the software do not run these on production operations themselves which means we need to hope they introduce new features that don’t break things.  This unfortunately was not the case recently as we had two pieces of software that run on all machines break in someway.

The first was our beloved RVSkin which an auto update on it resulted in it no longer functioning on almost all our servers.  The only one that was not broken was one we did not have on auto update which we proceeded to break when we updated it.  So we had to think fast and as always the solution had to be found on a forum and this case it was the rvskin one at least.  So we updated Rvskin manually using the instructions provided and problem solved.  This one was not major we maybe had rvskin broken for 10 hours and with x3 and other skins still running it was not a fatal issue.

The fun did not just end with Rvskin we also had cPanel introduce some strange functionality last week as well.  In this case they had complaints of customers complaining about how remote domains were being set as local within cPanel when they were created or transferred.  So they introduced a feature to detect this and if the domain was not pointed to the servers name servers it set it as a remote.  It sounds great in theory, however web hosts setup accounts all the time for domains that’s name servers are not set to them yet.  As a result of this we had numerous new accounts being unable to receive mail as the mail server was dropping it on mail entry. Once again another search on a forum and found cPanel introduced it and then they reverted it days later.  Of course by then we already had some accounts we had to fix that were setup this way.

In the cPanel case we updated in order to introduce features like roundcube webmail, new statistic software monitor, speed improvements and various other features.  All things customers have been begging us to add for a while, but we’re very conservative and run a RELEASE build which is considered production ready and should be really stable.  I suppose it’s not always the case or features that are bug free can still cause issues elsewhere.

Well I figured I’d share some of the fun we’ve dealt with last week with regards to software.  I’d like to point out both cPanel and Rvskin and solid products and we would not get rid of them for issues like this.  Every piece of software has bugs, it’s just a matter of how fast they fix them.  Of course here’s hoping we don’t see any unexpected features or bugs in the near future.