A better round-robin DNS server
Load balancing on the cheap is easy, especially if you don't require state.
Stick two servers in different places, and create a DNS entry with both IP addresses in it.
Your DNS server will then give out first one address, and then the other, balancing the load across them. Round robin. Yay.
So what's wrong with round-robin then?
Where this falls down, is when one of the boxes falls down. Then, 50% (or 33% or 25% etc) of the requests for the DNS name are sent to a box that isn't up. The viewers of your site would be miffed.
What you need is a system whereby when the box falls over, the DNS server removes the record for that address.
To manage this, you need a system where the servers continuously tell the DNS server that they're still available.
I thought I'd knock up a little simple DNS server that did this. I called it the very Dynamic DNS server.
How it works
It's a non-recursive DNS server, so you don't point your IP DNS settings at it.
What you do, is you delegate a subdomain to it. In BIND, you add a record like this to the mydomain.com zone file:
dynamic IN NS 3.4.5.6 (where 3.4.5.6 is the IP address you'll be running the DNS server on).
Then, all requests for *.dynamic.mydomain.com will be sent to the vDDNS server.
Once this is done, you run the server.
java -jar vDDNS.jar
Because of the annoying way that Unixes want you to be root to bind to ports <1024, you'll need to run it as root if you want the DNS server to listen on 53. (It doesn't do anything bad. It really doesn't. But run it in a VM, use GRSec, whatever you like.)
It requires a config file in the same directory called vddns.properties with a few simple properties in it.
Simplicity
So, once the server is running, it has no DNS entries in it.
To register an entry, you connect to the server on a TCP port, and send the name you want to register. That's it. Super simple.
I did it like this to keep things simple.
This way doesn't require any special software on the hosts, and you can script the updates with netcat and crontab.
You want access control? Use IPTables to block access to the port.
For instance, the following line in crontab on all the hosts you are trying to loadbalance should be enough:
* * * * * echo lb.dynamic.mydomain.com | nc 3.4.5.6 53053
They'll all register their IP against lb.dynamic.mydomain.com, and, if you've delegated the dynamic.mydomain.com subdomain, you should be able to dig/nslookup that name, and get a random IP each time.
Each updated entry only exists for (by default) 65 seconds. This means that if a host hasn't re-registered its entry, traffic will stop going to it after 65 seconds. If all hosts go down, all records disappear.
It currently doesn't support any records apart from A, although I will add IPv6 record support shortly.
Try it out
If you want to have a look, download these two files vDDNS.jar and vddns.properties, and run this:
java -jar vDDNS.jar
dig A foo.com @127.0.0.1 (should not return)
echo foo.com | nc 127.0.0.1 53053
dig A foo.com @127.0.0.1 (should return a record)
It's very simple, it works, and it does what it's meant to.
It ignores things it doesn't understand, and it's written in Java, which means the JVM will protect against buffer overflows, and the like.
TODO
Make the DNS packet parsing nicer.
Make it support IPv6 AAAA records.