Installing a HAProxy loadbalancing cluster

At work, the datacenter has been expanding with more work and more traffic. As part of the expansion plan I’ve been researching different loadbalancing solutions. Traditionally, we’ve used the Windows Network Load balance service built into Windows Server but it is a Layer4 load balancer. It isn’t very intelligent and, sometimes, isn’t reliable. The web service on Windows server 1 could be stopped, yet Windows server 2 is still operating and half your visitors will get the downed server. This is the difference between Network (layer 4/TCP) and application (layer 7/HTTP) load balancers. We’ve had a good experience with Cisco so were looking forward to try their ACE load balancer. Sadly, they shut that down. Barracuda was a little expensive as was the F5. I’ve heard a lot of great things about HAProxy so I decided to give that a shot.

In this lab we’ll be creating a HAProxy loadbalancer connected to 2 webservers. All of this is virtualized in our datacenter. The datacenter is comprised of a cluster of VMware ESXi servers with 1 management and 1 public network.

  1. Grab your favourite Linux distribution. I alternate between Debian and CentOS, a lot. Debian has been my go to for ages so I’ll be using Debian for this post. We won’t need the full install since it will be a stripped down appliance, basically.
  2. Create your virtual machine. Set the cluster to using guest OS “Debian 6 GNU/Linux (64-bit)”, 1 CPU, 2 GB RAM, use VMXNET3 (the Intel E1000 driver is CPU constrained), with LSI Logic SAS, and 10GB thin provisioned disk.
  3. Boot your virtual machine with the Debian Netinstall ISO and complete the installation.
  4. Once complete install the VMware tools. In the vSphere Windows console select guest and “Install/Upgrade VMware Tools.” Then in the console server console:
    # mount /dev/cdrom /media/cdrom;
    # cp /media/cdrom/VMwareTools-x.x.x.tar.gz /tmp/
    # cd /tmp/; tar -xzvf VMwareTools-x.x.x.tar.gz
    # apt-get install gcc make
    # apt-cache search linux-header
    # apt-get install linux-headers-x.x.x-amd64
    # /tmp/vmware-tools-distrib/

Once VMware tools is installed you may need to restart the server.
HAProxy isn’t in the main library of Debian Wheezy. To get it will need the backports repo. So create the file
# nano /etc/apt/sources.list.d/backports.list
And add the line
deb wheezy-backports main

And update your catalog and install HAProxy
# apt-get update
# apt-get install haproxy

After install haproxy change the service default to enabled
# nano /etc/default/haproxy

We won’t be creating the failover just yet. Once the main haproxy server is configured we can clone it using the vSphere console and update the IP address.

For the moment install keepalived.
# apt-get install keepalived

In part 2 we’ll continue working on the load balancer appliance by configuring web server delegation (round robin, etc)

Thanks, Digital Ocean

$5 per month for a VPS? What could possibly go wrong? Well lots actually. As the saying goes you get what you pay for, so when I decided to create my first droplet with Digital Ocean I wasn’t expecting a whole lot. Some users on HackerNews had rave reviews about them and me being frugal was always looking for a cheaper VPS host. So I created a droplet and copied this site over from AWS to them.

Things were working swell for a very a while, then my internet at home went on the fritz. While using a hotspot to make my last post I also decided to login to my droplet and do an apt update. And that’s when the problems arose.

See, I wanted to configure a proxy that I could use from any hotspot in Paris, since I don’t trust hotspots and I need the internet for banking and accounts (see DNS spoofing) I wanted a connection I could trust. So I installed some proxy onto my droplet. For $5 more I should have bought another. And then the connection died. No http, ssh, nor ping. I could no longer reach my droplet, my website was gone, and I was trying to fix this over a wireless hotspot that jumped APs every few minutes. No problem, I told myself I’ll log into the web console and fix it. The web console never worked on my Mac. So I did the same from a PC running Firefox and that just showed me a black screen.

Rebooting the server showed the server was stuck at init-bottom. This is the point in the boot sequence where grub should exit and the kernel is supposed to take over, and it wasn’t happening. Back at the droplet control panel I tried a few different kernels and stopped at the same point. I sent a ticket to Digital Ocean and the support team suggested I boot using the DO-Recovery kernel. That recovery kernel dropped me into a busybox shell but it didn’t have the tools I needed to access a borked LVM volume and it didn’t have network drivers. So not only could I not access the partition, but even if I did I wouldn’t be able to do a whole lot.

I opened another ticket to get a LiveCD boot (any distro). The support team came through again and got me into a Ubuntu LiveCD shell where I was able to open the volume with this site. Then I realized I had no network access and raised it with DO who straightened it out almost immediately.

With my volume mounted and reachable from the network I created another droplet with the same configuration as the broken server then using SCP I copied over my files, databases, and configurations to the new droplet. Some file permissions had to be edited but everything is right where I left it. And that is where we are today.

So I’d like to thank Digital Ocean and their invaluable support team for helping me get things back on track!

Corporate Inanity

This is going to be short because well I don’t do long rants. But my ISP, SFR, is a steaming pile of shit. We’ve been using them for almost 7 years, and even longer if you count that the company they bought, Neuf, bought AOL France. They’ve been shedding customers left and right (up to 4 million/year) and now I can see why.

So a few weeks ago we received an SMS saying the bill hasn’t been paid. Not too strange, we normally wait until the last minute because we have 2 mobiles and internet with them, easier to pay them when all the bills arrive rather than one this week and another 2-3 weeks later. So bill paid, done. The next day the phone doesn’t work. Call support and the rep says he’ll take care of it, gives us credits, and it should be sorted in a few days. 10 days later the phone is still not sorted, but we weren’t using it and simply forgot about it. 3 days ago the internet was turned off. Called SFR again and the reps were completely useless as they say the internet has been turned off because the bill wasn’t paid. But the bill was paid. “okay, it will take a few days, up to 72 hours. In the meantime you can go to the Agence SFR and get a 3G dongle until its turned on.” So the misses marches down to the agence and they can’t give it to here because we haven’t paid the bill. .

Apparently none of the reps can fix the situation. We’ve already accumulated over 6 months worth of credit with all the calling, complaining, and glad handling. We’ll use that free internet to close the account, find the nearest office to drop off their equipment, and search for a new ISP.

I’m still boggled it takes 72 hours to “turn on internet.” I can order a Macbook from Shanghai and it will be here in 24 hours, more or less. For some companies, the digital economy is a buzzword in their PR releases and not something they grasp all that well internally.

Raspberry Pi and SDR, Getting started

Due to the limited availability of free time I’ve had the raspberry pi has been sitting in a box unused, and unloved for a very long time. Looking for a project that was inexpensive and didn’t require a lot of loose wires hanging around I got involved in using SDR. SDR stands for Software Defined Radio and is where traditional radio components (tuners, clocks, modulators) are handled in software. Most popular SDR uses are USB TV Sticks such as this one.

After thinking long and hard about this (okay, 2 minutes) I put in an order for this USB DVB-T stick. It was even cheaper than the previous link and the reseller was a Prime reseller so it would get here quick. I should have done more research, according to the SDR wiki what I really wanted was a chipset that used the E4000 or R820T tuner chip. The RTL software lists mine as a Fitipower FC0013 which has half the range of the E4000. No worries, you learn and you learn.

So now that I’ve got the USB stick I needed to get the software that would interface with it. With the Raspberry Pi plugged in and fully booted I proceeded to install all the necessary files.

Here are the steps I used:
Step 0. Update and upgrade your system.
# sudo apt-get update; sudo apt-get upgrade

Step 1. Install the build tools
# sudo apt-get install git cmake build-essential libusb-1.0-dev

Step 2. Download the SDR files
# cd /tmp
# git clone git://

Step 3. Compile and install
# cd rtl-sdr
# mkdir build; cd build
# make
# sudo make install
# sudo ldconfig

Step 4. Test
# rtl_test -t
Found 1 device(s):
0: Sweex DVB-T USB

Using device 0: Sweex DVB-T USB
usb_open error -3
Please fix the device permissions, e.g. by installing the udev rules file rtl-sdr.rules
Failed to open rtlsdr device #0.

Hmm, okay appears to be a problem. Let’s test with root permissions

# sudo rtl_test -t
Found 1 device(s):
0: Sweex DVB-T USB

Using device 0: Sweex DVB-T USB
Found Fitipower FC0013 tuner
Supported gain values (23): -9.9 -7.3 -6.5 -6.3 -6.0 -5.8 -5.4 5.8 6.1 6.3 6.5 6.7 6.8 7.0 7.1 17.9 18.1 18.2 18.4 18.6 18.8 19.1 19.7
No E4000 tuner found, aborting.

Yes, definitely permissions. I did everything correct in /etc/udev/rules.d so not sure why this doesn’t work with a normal account.

The radio is on and working with Raspberry Pi. Now I can try some other tests
# sudo rtl_adsb
Found 1 device(s):
0: Realtek, RTL2838UHIDIR, SN: 000000041

Using device 0: Sweex DVB-T USB
Found Fitipower FC0013 tuner
Tuner gain set to automatic.
Tuned to 1090000000 Hz.
Sampling at 2000000 Hz.
Exact sample rate is: 2000000.052982 Hz

This pipes all ADS-B traffic to the screen. And if you have the right software you can decode the string to get position, aircraft.

Now that rtl_sdr is installed we have some alternatives that can use the USB stick now.

# git clone git://
# cd dump1090
# make

Then start dump1090 to get a formatted ADS-B table
# ./dump1090 –interactive –enable-agc
Hex Flight Altitude Speed Lat Lon Track Messages Seen .

The RTL library includes a basic server that can be accessed over telnet. Now you can stick your Pi and USB stick in a remote location and process the results elsewhere.
# sudo rtl_tcp
Found 1 device(s).
Found Fitipower FC0013 tuner
Using Sweex DVB-T USB
Tuned to 100000000 Hz.
Use the device argument 'rtl_tcp=' in OsmoSDR (gr-osmosdr) source
to receive samples in GRC and control rtl_tcp parameters (frequency, gain, ...).

For many other tutorials and information visit the project wiki

Goodbye, Nexus 7

Back in October on a trip through London I picked up the Google tablet, the Asus Nexus 7 from the duty-free. I also had an iPad 2 and a Nook Color that was beginning to show its age. With my son, wife, and niece making extensive use of the iPad I was really looking forware to having another tablet around the house. And I wanted to check out and see if Android 4 was as good as everyone says it was. Well, I can say I was not disappointed.

ICS was an excellent OS but no one uses just the OS and once I got past the basics, gmail, Youtube, web browser I really needed more. And what I needed I couldn’t find on the app store and the equivalents were always a little lower in quality. There are a lot of free games in Play. Some of the paid apps in the iTunes App Store were free in the Play store. And it showed. Usually with ads along the top or bottom of the app and sometimes at the startup. Then one day I started getting charges to my credit card. I had enabled Google Wallet and assumed it would behave like any other online store and ask for a password, first. Luckily the receipts arrived quickly in my inbox and I was able to cancel the transactions. Then I deleted Google Wallet. But 5 year olds and Play store don’t mix. I realized I could enable a pin on the device but that would be like getting a credit card from the bank then calling them to set up the pin. Some things are just automatic for a reason.

After a while I started to use it less and everyone else started to use it less as well. It became routine to squabble over the iPad and reach for the Nexus 7 only as the runner up. The games that were being played the most simply weren’t on the Nexus, the educational apps I bought on the iPad had no equivalent on the Play store.

Then it hit me while I was on vacation. Talking to my cousin on why I prefer Apple over Google, and him knowing I was a geek so it should be the other way. I simply told him, “I’ve invested 10 years into Apple. I signed up for the iTunes store when it opened it 2003 (I should have bought some stock then, dammit).” The iPhones and iPads are the least of my concern, they are just the terminal. I’ve bought 1000s of music and a few dozen apps through that store. For me to switch would mean I’d have to do it all over again. And I haven’t found a strongly compelling to throw that money away. He quite liked my reasoned argument rather than the typical fanboy rant.

So one day while the Nexus was sitting on the banquet, completely discharged, I plugged it in and did a reset and wipe. An hour later I listed it on the local version of Craigslist, Le Bon Coin. A few days later I was 120€ richer. It took about 4 days before anyone else in the house even realised it was missing. My son finally asked about the motorcycle game he loved so much.

So, goodbye, Nexus 7. You will be missed, a little bit.

The server in your home

A few days ago a friend of mine gave me a fantastic deal on a PowerEdge server coming out of his datacenter. A deal that was so low it could not be missed so after checking the bank account I sprang on it. I walked away with 3 PE 2950s for less than price of a point-and-shoot camera. So after a week of figuring out how to get 3 30kg servers home I finally got them into my apartment and wired up.

I manage our datacenter ESX servers at the office but I wanted to see what else is out there and compare to the VMWare solution. I’ve had limited exposure to Microsoft’s Hyper-V. Some with KVM. And little of Xen. The Hyper-V experience in the past has never been all that great. Disk space, memory, CPU, when you are virtualizing these things are important. KVM didn’t have good Windows server performance. Xen I hear is better and some of our clients use it.

After getting the servers plugged in and powered up the first thing that you notice is the noise. These beasts are built for the datacenter and the fans spin fast through a very small, metallic enclosure. Lots of noise. Also, they put out a good amount of heat. The passwords were unknown to me so I couldn’t login to ESX. I created a Linux boot CD and booted each server from a USB key.

mount /mnt/Hypervisor1
cp state.tgz /tmp/
tar -xzf state.tgz
tar -xzf local.tgz
vi etc/shadow

In vi completely delete the root password. Don’t change any other setting and don’t try and enter a new password since it would need to be encrypted. Save the file and copy it back to the mounted Hypervisor partition

rm local.tgz
tar -czf local.tgz etc
rm state.tgz
tar -czf state.tgz local.tgz
mv /mnt/Hypervisor1/state.tgz /mnt/Hypervisor1/state.tgz.bak
cp state.tgz /mnt/Hypervisor1/state.tgz

Reboot the server and remove the USB key. At the ESX console hit F2 to change the settings. Login as root and leave the password field blank. You’ll now be able to update the password and network settings. I downloaded the vSphere client using the browser and logged in. These machines have old ESX 4.1 Standard licenses. Fairly expensive but not what I planned to use anytime soon. So after recording the keys I formatted the top one.

The first server has 6x 146GB 10k SAS HDs with a Perc5i controller that has seen better days. The battery is kaput so I won’t be getting the performance I’d expect since Writeback caching is disabled. On this machine I decided to install Windows Hyper-V Server 2012 Core. This is the closest approximation that Microsoft has to ESXi. They do include a lot of functionality in the free version that VMware requires in their more expensive options.

The out of box experience was not nearly as nice as ESXi 5.0. ESXi only requires a 512MB USB key to install and then can install the hypervisor back onto that same VM. Hyper-V can technically do this but the instructions state that you need to download a lot of files and create a VHD. So I formatted the disk array to RAID 5 and installed the server to the local disk.

Hyper-V Server restarts a few times during the install. This was surprising to me since I had stepped out to grab some tea and come back to the USB menu. So, remember to remove the USB key after the initial file install. The first time you connect you’ll be asked to enter a password. Make it a good one because the local policy by default won’t accept “password” as the password. Once you’re in you’ll be presented with the text-based menu. Here you can configure networking, enable remote desktop, and run system updates. You should do all three in that order.
Screen Shot 2013-02-22 at 4.57.12 PM
Once the updates have completed and the server rebooted you can disconnect from the server and resume the session through Remote Desktop.

I’m trying to bring a server instance up using only Powershell. Hyper-V Management tools for Server 2012 require Windows 8 and I only have Windows 7. My plan is to install Server 2012 as a bootstrap server than try the different aspects of Hyper-V and see how it really stacks up. So far the experience hasn’t been enjoyable but I’m willing to slog through it and see this as minor stumbling blocks. Microsoft has invested a lot of time improving Hyper-V so I’ll do my part likewise.

A number >1 is 1 too many



My hard drive on my Mac was acting a little strange lately. More specifically I was getting spinners and sometimes beachballs when I tried to open a folder. This hard drive is the Users storage area for all the accounts on this Mac so I can’ afford to have it fail. Through a lot of careful planning I was able to create a new RAID1 disk on the Mac from the old RAID disks recovered from the Ubuntu servers (a topic I’ll cover in another post). SmartReporter displayed nothing and Disk Utility still had the hard drive’s SMART status as verified. Being cautious about the last “verified” hard drive that is sitting in a drawer waiting for me to save enough to take it to a recovery service I decided to be a bit more proactive and get the actual SMART results. Using SmartReporter again, I had it read out the SMART attributes and this is what I saw.Screen Shot 2012-12-16 at 6.49.30 PMFor hard drives you need to pay attention of a few things: Reallocated Sector Count, Current Pending Sector Count, and Offline Uncorrectable. The first will tell you that the media is failing so be cautious; you can continue to use the hard drive but it will fail, soon. The second will tell you if data is being copied from that failed sector to another currently (flashing yellow lights). The third is saying it’s already too late, you are losing data as we speak. The first 2 are critical, the last can make backups a nightmare because the OS will try to read a bad sector, fail, and stop any operation including backups in progress.

Knowing what was going down I immediately started an rsync
sh# sudo rsync -av /Volumes/Data/Users /Volumes/RAID\ Drive/

What proceeded was hours of frustration as things got progressively worse. Rsync would start the files would begin to copy and after a few minutes or hours the drive would disconnect. Figuring the drive might be overheating I isolated it from the others and installed another fan directly in front of it to keep cool air moving over it. Every few hours I would have to restart the Mac because the drive would stop responding and the only way to get it back is to reboot. Eventually, I started triaging data, deciding what needed to be copied right away and what could be done later. The wife and kids folder went first, they were smaller and took just a few minutes. My Library, Photos, and Documents folder was my priority; Dropbox was safe since it was in the cloud. My music folder was backed by iTunes Match (which expired a few days later) and Videos folder was the lowest.

After many hours (a day I would say) I was finally able to transfer the entire volume onto a RAID-backed array. With everything back where it belongs I relinked the Users folder and all is right in the world. I was planning to copy the data to this drive so I could format the SSD into a Fusion Drive. For now, it will have to wait while I create a new strategy to backup, format and restore to the SSD.


I sold out, a little bit.



On the way back home, while passing through the airport I stopped at the Dixon’s and found this baby, The Google Nexus 7. I’ve had a poor experience with previous Android tablets so wanted to see if Android had really gotten any better or was it more fanboy drivel. After having it for a week I can honestly say it’s pretty good. There are differences between Apple and Android OS philosophy and there are some things I enjoy more on the iPad 2 than on the Nexus 7. I’ve been forced to make this my daily driver since my son has assumed control of the family iPad. So I’ll discuss my thoughts on each aspect below.

Apple pays a great deal of attention to the out of the box experience in all their products. The Nexus came in a cardboard box with a sleeve that was almost impossible to open without tearing it. Inside was the Nexus wrapped in plastic, a box with the cable and charger inside, and warranty card. While not as great a packaging as the iPad at least it wasn’t blister packed.

I turned it on for the first time and after doing it’s initial cold boot, which took about a minute, we go to the language selection screen. I selected English of course then moved through the wizard in selecting an access point, entering my Gmail details and then entering the play store where Transformers 3 was ready for me to watch. It was thrown in as freebie.

I do a lot of reading and the Google Books app looked sorta nice. But you can’t side load books; it only takes content you purchased through the Play store. It downloaded some copy-expired books, Jules Verne, Victor Hugo, etc, but I wasn’t interested in reading those (they are good books just not what I’m looking for at the moment). Since Google’s competition is the Kindle, the Nook, and the iPad, 3 devices with first party readers that do support undreamed ebooks this is something they will need to work out, soon.

Next I tried Google Music. The online version is still not available in France and I haven’t plugged a tablet into a PC in ages. But when activated that was my only option. I looked at my iPhone, conveniently loaded up with music that hasn’t been copied from a PC since I bought it, and shrugged. Google may rule the cloud but it appears the cloud doesn’t work everywhere.

So books and music are a wash, surely video works. Netflix isn’t available in my country (still feels weird saying that) nor was it presented as an option to download in the Play store. I do subscribe to so hopefully those guys were able to do some DNS magic for the Play store. I went into wifi settings, selected advanced options and changed my IP and DNS from DHCP to static. The dynamic IP remained, this was good, the DNS was blank so I entered the IP to the DNS servers playmo provides. I rebooted the Nexus and entered the Play store again. Sure enough the Netflix app was available to download. I Installed and logged in and was soon watching the show from where I left it. The Netflix app for Android would pause and stutter at the selection screen. It clearly wasn’t as smooth as the iPad but having Netflix is better than nothing.

The Nexus 7 also includes 2 native players. The Gallery app and the Play Videos app. Play Videos would only display the files if they were local. I wasn’t interested in plugging it into the USB port of my PC. I did watch a few minutes of Transformers 3. It was pretty good. The Gallery app was able to play some formats; mp4 it seems. It doesn’t natively handle network shares, I downloaded another app (ES File Explorer) to connect to my NAS and pipe the data to Gallery. The results were good but Gallery is sort of bare when it comes to video control. I ended up using Mobo Player to watch the subsequent videos. I’m not a fan of Mobo’s interface (I think it’s not pretty) but it has all the controls I need.

Games are another thing. They are actually pretty good on the Nexus 7. For casual gaming, like Bad Piggies or Angry Birds, you wouldn’t be able to tell the difference between this and the iPad. FPS games, I felt, weren’t as smooth as they are on the iPad. This is a subjective observation and I don’t have the tools to make a completely unbiased opinion. The zombie game, Dead Trigger, was just alright.

The Nexus 7 is a great device. It’s just marred by the geo-locks Google puts in place for the moment. And some first-party apps that I feel are incomplete compared to the competition. For now, Apple still rules the roost. The iTunes, iPad, iPhone integration is much better implemented and I’m still waiting on Android’s response to Airplay (it’s not DLNA; snort). I am reading books on it. The main thing that keeps it from being returned is the fantastic screen, lightness, and the Kindle app. It’s ironic that one of the best stores on Android is not from Google but their competitor. And the price is hard to beat. If you’re invested in the iTunes ecosystem then the iPad mini is a better option. If you try to purchase DRM-free material when available then this is also a great option. I have reservations about Google’s motives, they hacked around Safari’s privacy settings because their tracking was more important than your privacy. Their were a  lot of features that I wanted to use but are still “unavailable in this country” like Wallet and Music.

Amazon EC2 and surprises

I’ve been running this blog on the micro instance of amazon EC2 for a few months now and I’m still trying my best to optimize it. The first month wasn’t so bad; I moved all the content, reattached the databases and redirected the domain name. I was in for about 7€ that month. The second month was more but I assumed a private instance should cost about 20€. There were a ton of problems. MySQL and apache were being stopped due to not enough memory or not enough swap. And sometimes the site would be completely unresponsive. This month, now that the site is getting more traffic, the bill was $30. Almost twice what I expected to pay for micro instance.

I’ve converted this server to a reserve instance. It should have been more clear when I signed up that this would be a smarter option I’d it was going to be running 24-7. But c’est la vie.

Apple TV, IPv6, and a problem

For the past month I’ve been beta testing the DNS services of to access my Netflix account while outside the US. It’s very simple, sign up, and change the DNS settings on your compatible devices. In the network settings of the Apple TV I switched the LAN configuration from DHCP to manual, kept the IP and subnet but updated the DNS address (originally pointing to the router which was downstream from my ISP DNS) with the DNS servers provided with my playmo account. After rebooting the AppleTV I was surprised to find I had no internet service at all. Clicking on the network configuration again should that it was manual IP, Subnet, and router, but now the DNS was changed not to the one I specified but to the IPv6 DNS of the router.

I repeated this 2 more times hoping it was some kind of error but the results were the same. DNS I entered was gone, replaced by the automatically completed IPv6 DNS address. My family was starting to get bothered (since they only complain when things break) and as a quick fix I disabled the IPv6 DHCP service on the router. Rebooting the Apple TV one more time got me connected to the iTunes Store and now Netflix.

So in summary if you want to use the Apple TV with a DNS forwarding service like playmo you’ll need to disable IPv6 services on your router. For the moment it isn’t a big problem but I’d like for Apple to fix this bug in their network stack.