Installing a HAProxy loadbalancing cluster

At work, the datacenter has been expanding with more work and more traffic. As part of the expansion plan I’ve been researching different loadbalancing solutions. Traditionally, we’ve used the Windows Network Load balance service built into Windows Server but it is a Layer4 load balancer. It isn’t very intelligent and, sometimes, isn’t reliable. The web service on Windows server 1 could be stopped, yet Windows server 2 is still operating and half your visitors will get the downed server. This is the difference between Network (layer 4/TCP) and application (layer 7/HTTP) load balancers. We’ve had a good experience with Cisco so were looking forward to try their ACE load balancer. Sadly, they shut that down. Barracuda was a little expensive as was the F5. I’ve heard a lot of great things about HAProxy so I decided to give that a shot.

In this lab we’ll be creating a HAProxy loadbalancer connected to 2 webservers. All of this is virtualized in our datacenter. The datacenter is comprised of a cluster of VMware ESXi servers with 1 management and 1 public network.
Management: 10.0.5.0/24
Public: 10.0.0.0/24

  1. Grab your favourite Linux distribution. I alternate between Debian and CentOS, a lot. Debian has been my go to for ages so I’ll be using Debian for this post. We won’t need the full install since it will be a stripped down appliance, basically.
  2. Create your virtual machine. Set the cluster to using guest OS “Debian 6 GNU/Linux (64-bit)”, 1 CPU, 2 GB RAM, use VMXNET3 (the Intel E1000 driver is CPU constrained), with LSI Logic SAS, and 10GB thin provisioned disk.
  3. Boot your virtual machine with the Debian Netinstall ISO and complete the installation.
  4. Once complete install the VMware tools. In the vSphere Windows console select guest and “Install/Upgrade VMware Tools.” Then in the console server console:
    # mount /dev/cdrom /media/cdrom;
    # cp /media/cdrom/VMwareTools-x.x.x.tar.gz /tmp/
    # cd /tmp/; tar -xzvf VMwareTools-x.x.x.tar.gz
    # apt-get install gcc make
    # apt-cache search linux-header
    # apt-get install linux-headers-x.x.x-amd64
    # /tmp/vmware-tools-distrib/vmware-install.pl

Once VMware tools is installed you may need to restart the server.
HAProxy isn’t in the main library of Debian Wheezy. To get it will need the backports repo. So create the file
# nano /etc/apt/sources.list.d/backports.list
And add the line
deb http://cdn.debian.net/debian wheezy-backports main

And update your catalog and install HAProxy
# apt-get update
# apt-get install haproxy

After install haproxy change the service default to enabled
# nano /etc/default/haproxy
ENABLED=1

We won’t be creating the failover just yet. Once the main haproxy server is configured we can clone it using the vSphere console and update the IP address.

For the moment install keepalived.
# apt-get install keepalived

In part 2 we’ll continue working on the load balancer appliance by configuring web server delegation (round robin, etc)

The server in your home

A few days ago a friend of mine gave me a fantastic deal on a PowerEdge server coming out of his datacenter. A deal that was so low it could not be missed so after checking the bank account I sprang on it. I walked away with 3 PE 2950s for less than price of a point-and-shoot camera. So after a week of figuring out how to get 3 30kg servers home I finally got them into my apartment and wired up.

I manage our datacenter ESX servers at the office but I wanted to see what else is out there and compare to the VMWare solution. I’ve had limited exposure to Microsoft’s Hyper-V. Some with KVM. And little of Xen. The Hyper-V experience in the past has never been all that great. Disk space, memory, CPU, when you are virtualizing these things are important. KVM didn’t have good Windows server performance. Xen I hear is better and some of our clients use it.

After getting the servers plugged in and powered up the first thing that you notice is the noise. These beasts are built for the datacenter and the fans spin fast through a very small, metallic enclosure. Lots of noise. Also, they put out a good amount of heat. The passwords were unknown to me so I couldn’t login to ESX. I created a Linux boot CD and booted each server from a USB key.

mount /mnt/Hypervisor1
cp state.tgz /tmp/
tar -xzf state.tgz
tar -xzf local.tgz
vi etc/shadow

In vi completely delete the root password. Don’t change any other setting and don’t try and enter a new password since it would need to be encrypted. Save the file and copy it back to the mounted Hypervisor partition

:wq
rm local.tgz
tar -czf local.tgz etc
rm state.tgz
tar -czf state.tgz local.tgz
mv /mnt/Hypervisor1/state.tgz /mnt/Hypervisor1/state.tgz.bak
cp state.tgz /mnt/Hypervisor1/state.tgz

Reboot the server and remove the USB key. At the ESX console hit F2 to change the settings. Login as root and leave the password field blank. You’ll now be able to update the password and network settings. I downloaded the vSphere client using the browser and logged in. These machines have old ESX 4.1 Standard licenses. Fairly expensive but not what I planned to use anytime soon. So after recording the keys I formatted the top one.

The first server has 6x 146GB 10k SAS HDs with a Perc5i controller that has seen better days. The battery is kaput so I won’t be getting the performance I’d expect since Writeback caching is disabled. On this machine I decided to install Windows Hyper-V Server 2012 Core. This is the closest approximation that Microsoft has to ESXi. They do include a lot of functionality in the free version that VMware requires in their more expensive options.

The out of box experience was not nearly as nice as ESXi 5.0. ESXi only requires a 512MB USB key to install and then can install the hypervisor back onto that same VM. Hyper-V can technically do this but the instructions state that you need to download a lot of files and create a VHD. So I formatted the disk array to RAID 5 and installed the server to the local disk.

Hyper-V Server restarts a few times during the install. This was surprising to me since I had stepped out to grab some tea and come back to the USB menu. So, remember to remove the USB key after the initial file install. The first time you connect you’ll be asked to enter a password. Make it a good one because the local policy by default won’t accept “password” as the password. Once you’re in you’ll be presented with the text-based menu. Here you can configure networking, enable remote desktop, and run system updates. You should do all three in that order.
Screen Shot 2013-02-22 at 4.57.12 PM
Once the updates have completed and the server rebooted you can disconnect from the server and resume the session through Remote Desktop.

I’m trying to bring a server instance up using only Powershell. Hyper-V Management tools for Server 2012 require Windows 8 and I only have Windows 7. My plan is to install Server 2012 as a bootstrap server than try the different aspects of Hyper-V and see how it really stacks up. So far the experience hasn’t been enjoyable but I’m willing to slog through it and see this as minor stumbling blocks. Microsoft has invested a lot of time improving Hyper-V so I’ll do my part likewise.

The 49th Paris Airshow

The weekend of June 22nd brought the Paris Airshow to town an me and Mack had a great time there. Clear skies and a bright sun ruled the day and we made the most of it. Getting there was a bit of a headache but we were able to board the bus from Paris to le bourget and get a seat for the 30-60 minute ride. The second stop had a lot more people who had been waiting for the bus for an hour. Heated words were exchanged between those wanting to get on but couldn’t and those packed on board like sardines in a can. I really enjoyed seeing the new planes like the A380 and the boeing 787. We briefly saw the solar impulse. A lightweight powered glider powered by a PV array.

20110705-135625.jpg

20110705-141421.jpg

1 month later…progress.

20110528-122826.jpg

About a month ago I started a small backyard garden. Since I didn’t know what I was doing it’s been an experience. My tomatoes are coming up strong. I’ve got another plant I have no idea what it is. Having good documentation and labeling is really important I’m finding out. I’ve battled a slug infestation successfully as well as small weeds. Getting started has cost more than I assumed. But just like any new project once you have the materials it gets cheaper and easier.

I’ll keep you posted as the basil and mints come up. There is also some green peppers somewhere in there as well.

I bought a Nook Color

A week ago I went on vacation to Hawaii to visit family. While I was there I took advantage of the exchange rate to get a device that I’ve been hearing a lot about. That was the Nook Color; an ebook reader available from the store Barnes and Noble. For a color ebook reader that boasts a 7-inch screen the $250 price is hard to beat.

Already, hackers have gotten hold of the system and rooted the custom version of Android that runs on it. The underlying OS is Android 2.1(Eclair) which doesn’t support Flash. According to BN an update should be available shortly that updates the system to 2.2 (Froyo) and includes the Flash player. The included reader application from BN is quite nice. It supports DRMed epub, open epub, text, and pdf files. A simple file browser allows you to open videos, pictures, and music files as well as browse towards your ebook files. There is a bookcase like display that you can add your current books too. Or, replacing the android status bar, you can go directly to the last opened book but selecting it from the status bar.

One thing the BN reader has that their Android Marketplace app doesn’t is the ability to read books aloud. Since I have a 4 year old this was a very handy feature that isn’t available in all the ebook reader applications I’ve tried. Without rooting the device your choices for entertainment are limited. It includes chess, crossword puzzles, and soduku as the included games. Rooting the device is as simple as downloading a file to an SDCard and booting from it. Instructions are available here. Once done you’ll have acces to a different applauncher and the android market. I downloaded the game Angry Birds and had a great time playing it on a much larger screen than my iPhone.

Not completely satisfied with spending money on a device that only has one purpose I decided to install an alternate version of the OS. Using this guide I installed the current version of Android, 2.3(Gingerbread). But after being disappointed with the battery life, button placement, and available ereaders I went back to the original firmware, that I eventually rooted.

The current release isn’t fast by any means but it is the one that works the best all things considered. CM7 didn’t have proper power management and having to charge an ereader everyday was beginning to be annoying. With the original firmware the Nook Color can go a week between charges.

Friday night at Patrick’s Irish Bar

The other day I got an invitation from a member of Github hosting a party in the 11th at an Irish pub. Me being an introverted programmer means I don’t get out too often and thought this would be an interesting way to meet other programmers. And I was not disappointed.

The guys I met, and they were mostly guys, were very open to talking about industry, programming, and international politics. Topics ranged from the direction of mobile development, embedded development, and history of mobile development. When we weren’t talking about programming we covered areas such as international finance and being a quant. Occassionally someone would make fun of the Symfony attendees. Since they were familiar with each other and we were not it felt good to break the ice by going after the insiders.

All in all, a great night out with a great bunch of people. As a developer you rarely get to meet other developers except online and the occassional conference. The sharing of ideas is very rewarding and I hope to do it again.

PS: It also helps when someone else is paying for drinks, so a big thank you to Github.

Hello world!

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

And that’s it. It’s coming together and I’m going through the process of migrating my assets and images. This blog has been an ongoing process for over half a decade. There was an even older one thats probably floating around a CD-R in my storage locker. Over the years I’ve gained new friends and a family. But at the same time I’ve also lost contact with people, friends becomes loosely defined due to the fact that we are on opposite sides of the planet.

I’ve also managed to acquire a large amount of stuff on this server through the years. It sort of reflects my life where there are things I can’t use, yet can’t get rid of because of what they mean to me. Sometimes it seems convenient to dump everything into the garbage and start over again. Believe me, I have a few times and everytime there comes a point where something was¬†accidentally¬†disposed of.

So bear with me as I go through the adjustment of working with a different bloggin system try to fix broken links, and try to find a way to get this thing up running.

36 hours and 2 oceans later…

I’ve just arrived back from a very long trip. A trip to Hawaii to see my sister get married. It was a hell of a long trip and completely worth it. I’m still in the process of sorting through the insane amount of photos taken on my iPhone (my son broke the good camera months ago) and as soon as I’ve found some gems I’ll be sharing them with you.
The trip was incredibly long but the cabin crew on Air France and American Airlines were great. I did have some complications while waiting for the LAX-HNL leg. American cancelled one of the earlier flights and there was a standby list of 15. Me and a couple from Australia were the last on the standby list (both of us traveling on staff concessions) and just as they were about to close the gate they let us on. And put us in First Class to boot.
The return flight was even longer due to the fact that Hawaii is 12 hours behind my homebase, and I had a 8 hour layover in LA. With such a large amount of free time on my hands I decided to take a trip to the beach.
(To be continued…)

Ker-plink!

Ker-plink! That was the sound of the hard drive in my Hackintosh that died. While normally I’ve been a stand up guy about archiving (not to be confused with backups) of data. In this case the blood drained from my face as I walked into the study and heard the noise that everyone dreads to hear. So I jumped into action, tried to find which drive was making the racket (there were 3) verified that indeed it was no longer responding, console was spitting out streams of errors, and all the folders came up empty. I shutdown the Mac, took it out, weighted a few seconds for the platter to spindown (it makes a thunk when it comes to a complete stop). reconnected and restarted.
BIOS reports no drive 3. Hmm. Take it out again, slowly rotate the drive listening closely for grinding noises, nothing. Well this is slightly better than terrible. You see Watson, this disk has fallen for the Sector 0 error, commonly known as the “click of death”. The read head tries to find Sector 0, and slides right past it, slamming the head into the spindle. So technically my data is still on the drive, I just have no possible way of getting it off.
Using my years of expertise, I know I can hopefully calibrate Sector 0 by freezing the drive and reading the data before the platter gets too warm. Into the freezer she goes. This hard drive has been sitting there while I order another hard drive and a NAS. I’m taking this opportunity to do something I know I should have done from a long time and formulate a real backup plan.
You see, the Free people came by and installed fibre to the apartment building. They assured me the units would be lit after the new year. Well that was 3 months ago and no fibre yet. My backup scheme sort of hinges on it, because pushing 60GB of data over 800kbit/s (that’s 100KB/s) aDSL is mindfuckingly slow.
So the scheme is use the ReadyNAS Duo duo_reflect2.jpg as the local NAS. And have it push my backups to an offsite point. In my case, Amazon S3. That way I can use the NAS as a local storage pool, torrentclient, and DLNA server. My energy sucking workstation Q6600/HD3870 doesn’t have to be running continuously. And if the hard drives die in the next year or 2 (the dead drive lasted 3 years) I can restore the data using 100Mbit/s fibre.
If it sounds good to you, it sounds good to me.

Happy New Year

Image and video hosting by TinyPic
I know I’m a little late but I want to wish everyone a Happy New Year. I was celebrating with my son and didn’t have time to post on this blog. Because we bought a house and all the expenses of moving into it, there wasn’t a lot of available money to go crazy balla’ style so I kept it simple: a bit of champagne, a walk to the top of Sacre Coeur, and watching the countdown from home. My son, who didn’t take his afternoon nap, didn’t have the energy to stay up til midnight. I had to carry him down the mountain.
THe following next days were spent trying to get my computer to boot. The hackintosh is failing hard since the last chameleon update and the problem doesn’t seem solvable. The quickest option would be to backup our data and build a new install but I’m all out of capacity and need a new hard drive.