static.chombo.com.au. It’s a virtual machine running the bare essentials. CentOS 5, lighttpd. No PHP, no MySQL, no Webmin. I’ve even got lighttpd stripped down, running only the modules that it needs. It’s not capable of vhosts or URL rewriting, so no fancy URLs or different content for different hostnames. All it has is the ability to respond to HTTP requests and the ability to compress JavaScript and CSS on the fly. Why have I done that? For speed. What’s it going to be hosting? Images, JavaScript, CSS.

I’ve been told that when you’re on a fast computer and fast network link not far from the server (less than 20ms latency), there can be a slightly noticeable difference in loading times between Apache and lighttpd when serving the same content. For ages we’ve had Apache and lighttpd running on the server that hosts chombo.com.au, lighttpd hosting all the public content, Apache hosting our ‘preview’ website – the place where we can test website revisions before they go public. I do have a problem with lighttpd, however. It’s a pain in the backside if you’re wanting to be running any URL rewriting. Could be my config file causing the problems, but I think it’s just fussy. Time to run Apache for the public content, I think!

My reason for setting up static.chombo.com.au was so that I could have the flexibility of Apache for the main website and still get the fast response times of lighttpd. As I said, static.chombo.com.au is going to be for images/CSS/JavaScript only, meaning Apache will really only do the PHP stuff. Make sense?

VMware ESXi was the virtualisation platform that we decided to run with. Got some VMs running on it and it seemed fine. ‘Barney’ was born. Named after the character in How I Met Your Mother, we had some hope invested in it/him. It was going to be more reliable than Mulder – which ironically became good after it wasn’t our sole shared hosting server – and potentially faster. I got some customers to volunteer to test out the new server and it seemed fine.

Further down the track Barney becomes the worst server on our network. No downtime or anything on it, the speed of it was just a killer. You’d try to open a WHM session, type in your password, then you’d have a white screen open for about a minute or two before anything really happened. While waiting for it to start working properly, the load would skyrocket to about 15 to 20 from 0. Once WHM had loaded for that first time the pages would be loading fine and everything would perform nicely. It’d just be that first time to load every separate session. Seemed as though the hard disk was spinning down, even though it wasn’t.

Naturally the most logical thing to do is to work out why it’s going slow. Log into an SSH session, make sure it’s all working, then load up WHM and wait for the load to spike. Yep, I did exactly that. SSH was always fast, WHM would always be initially sluggish. I’d have a top session running and nothing would seem odd. No resources sitting at 100% CPU usage. Plenty of RAM free, with a reasonable amount cached. Plenty of disk space. No other VMs on the same node hogging resources.

Ranjit was set up on the same node as a DNS server and was eventually going to be used as our primary cPanel DNS server for customers. It had the same problems as Barney. We were never able to work out the cause of this problem. Disappointing, I know. cPanel/WHM just weren’t liked. I had a Webmin server going on there for some of my own testing stuff and it seemed to work fine overall – a few patches of slowness here and there, but nothing like encountered with Barney & Ranjit.

At the start of this month we provisioned a new server called Dugong. This was to coincide with the change of ownership, going from Curtis to Michael. Michael’s a big fan of Dugongs for some reason. If you’ve seen him on Twitter you’d think he was an absolute lunatic. Some of his tweets (prior to the acquisition) looked like this:

Just you wait; Dugong Bank, Dugong Cars, Dugonalds.

@chombo I demand you rename to Dugong Hosting.

I wanted to choose some sort of awesome name for the new server, but apparently I didn’t have a choice. Dugong has so far been an awesome server. Our best yet, in fact. Incredibly fast. What would surprise you is that it runs the exact same hardware as Barney did – only there’s no virtualisation involved here. We gradually moved everyone off Barney and on to Dugong. So far I’ve moved half of the people who were on Mulder onto Dugong. The exceptions being resellers and dedicated IP address customers.

What’s happening with Barney now? I turned off the VM earlier this evening after the last person on it moved off. I’ll admit, I did feel a little sentimental when running the shutdown -h now command. I had rebooted it many times in the hope of new kernels fixing the slowness, but nothing ever worked. We should soon have that node back up and running soon as ‘Dolphin’. Exactly the same hardware as Dugong. Core 2 Quad, 8 GB of RAM, dual 500 GB hard drives in RAID-1. Same software will be running on it as Dugong – CentOS 5 (64-bit), cPanel, Installatron, PHP 5, MySQL 5 etc. etc. There will be a slight difference to this server in terms of software configuration. What is that difference going to be? I’ll make an announcement when it happens. I can tell you now that Dolphin will be used for reseller customers.

I recently came up with the awesome idea of a blog site where I’m able to share my excitement about everything with everyone AND go into detail about it. Twitter only lets me tell people some of the stuff that’s going on. If I do get too carried away I’d probably lose 100 followers or something. That doesn’t excite me like this blog should.

Hopefully I’ll find some time soon to write up some posts on what’s been happening and why. Watch this space.