Streaming My Consciousness

Shatto Milk was having an open house today at their farm up in Osborn, MO, and we thought it would be a fun thing to do with the girls. While Andrea was up at Troost Elementary for COR’s Bless The School missions project, I took the kids up to the farm.

They had a bounce house, hot dogs, beans, samples of their milk (and requisite cookies to go with, provided by Midwest Airlines). The girls got to see the bottling plant, the bottle washer, and the milking parlor, and got to pet some cows and some of the new calves (a few weeks old!). A good time was had by all and the weather was absolutely perfect: low 80s, sunny, with a breeze.

It was nice enough to go up in my car instead of the minivan (My car has been referred to by some as my “Dave Ramsey Car”). My car’s air conditioner is currently out of commission, so summer trips in it with the kids are rare. Luckily, it’s highway most of the way, so it’s easy to get a good breeze going in the car even with the windows shut. I filled up before I left and when I got back. Total round trip: 158.7 miles. Total fuel consumed: 3.8 gallons. Yes, you saw that right. My $1700, 16-year-old Corolla got 41.7 mpg on that trip. Take that, Prius!

I’ll edit this post later to add linky goodness and pictures!

Take a look at this data coverage map from Sprint. Orange is EV-DO, Yellow is 1xRTT:

That spot with lots and lots of EV-DO, disproportionate to the rest of the country? That’s Alberta. The entire province is a mere 3.5 million people.

The isolated (but large) blob in the northeastern part of the province is centered on Fort McMurray, population 47,000. The EVDO coverage area is probably several thousand square miles.

Three guesses where all the oil is

I thought it rather ironic that as I was installing my HA firewall cluster that I hadn’t planned the whole hardware redundancy thing all the way through.

In order to install the new machine and the NICs, I had brought a screwdriver to mount rack rails and such. This particular screwdriver was one of the ratcheting kind, and it’s been a poorly functioning department fixture since before I arrived. Today, it decided to completely and catastrophically fail. One moment, I’m turning a screw, the next finds my hand holding about half a dozen pieces of the ratcheting mechanism, and the screwdriver shaft spinning freely and uselessly.

… and in my planning to build the HA cluster for the firewall, I’d neglected to brnig a spare screwdriver in case that hardware failed (which we’d expected it to do long ago). Luckily, one of our “neighbours” happened to have one with him and let me borrow it.

Moral of the story, make sure you have full hardware redundancy, including your screwdrivers.

I now have a profound appreciation for BSD.

Yesterday, our pfSense firewall at 1102 Grand suddenly went silent and the panel on WhatsUp for that site went all red. Not good. I went down to the cage after small group last night and found the machine to have just simply locked up cold. I suspect hardware, since pfSense/BSD didn’t log a thing about it going dark.

It became quite clear that this setup was… suboptimal. Clif’s shooting for 99.99% on this new setup. I can’t be racing off to the datacenter every time the firewall machine decides to take a holiday from reality. Brian and I quickly determined that we needed not only a remote power control unit, but some sort of high-availability solution that wasn’t going to empty our wallet like a pair of NSA 4500s would. (sorry Mark, we simply don’t have that kind of money) We already had a spare, identical machine at the datacenter doing duty as a hardware spare and development server, and another one just like it in inventory at the Central Campus. I grabbed the extra machine and went back 1102 Grand for the second time in 12 hours, with a quick stop at Micro Center for some cheap NICs and a red crossover cable.

Fortunately, pfSense has high availability capability built in, thanks to BSD’s CARP and pfSync. CARP allows me to set up virtual IPs on both firewalls and synchronize between them with pfSync. The extra NICs were for a dedicated sync/heartbeat link between the two boxes. I’m still a little fuzzy on the technical details of how this works, but it works… Convergence/Failover time is 3 seconds or less, and everything is synchronized between the two machines, including state and session information. I had it all set up, and hit the reset button on the primary firewall… and nary a ping was dropped.
The setup involves giving each machine its own LAN and WAN addresses (and a unique address on the other zones as well) and then creating a CARP virtual IP on that interface. The virtual IP is the one used as the gateway and as the NAT address. All rules and configs (including IPSEC Tunnels!) is mirrored on the second box.

Reference material:

Excellent documentation on the process from the folks at Countersiege.

Some background from the OpenBSD folks.

A few good tips here. These proved to be crucial.

Pretty straightforward, once I found the correct search terms to do it. Did it with the aid of this great tutorial, with a few changes.

The tutorial assumes you’re on some kind of hosting provider. I eventually figured out that when you’re running your own server without CPanel (because CPanel sucks bigtime), when he says “park a domain”, all you have to do is point the DNS for the host you want to use at your wordpress installation and add it as a SiteAlias in your Apache config.

The other thing that got left out was duplicating the metadata from the original site to your new site ID. Especially important are allowedthemes, menu_items, and upload_filetypes.These merely need to be copied from the original site, unmodified.

Sure, you can do this without phpMyAdmin with manual SQL statements, but PMA makes it a lot easier.