Adventures in Network Plumbing

The last few days have been rather stressful.. Our shiny new web infrastructure at COR has been throwing major temper tantrums, which means I’ve been rather busy of late. Today, it melted down half a dozen times after I thought I’d fixed it. Each time it did something different. (and if that wasn’t enough excitement, our upstream provider had a BGP issue this morning that knocked their entire customer base off the web for about 5 minutes)

All you folks that hit our website, thank you for your patience. These have been trying times.

This morning, I noticed something very odd. And again this evening as I’m migrating the data to a new server.

root@corweb1:/content/sites# uptime
11:35:56 up -24855 days, -3:-14,  4 users,  load average: 0.00, 0.00, 0.00

Negative uptime??? What?

root@corweb1:/content/sites# date
Wed Aug 31 11:50:17 CDT 1955

Perhaps running apt-get install flux-capacitor wasn’t such a good idea, as the machine seems to be performing on about the level of a computer from 1955.

It’s become pretty obvious that something is very ill on that box, and I think it’s time to ditch VMWare Server for ESXi. Until then, we’re moving the servers over to the bare metal on the other box (which includes the blog server, it’s already been moved).

Last login: Wed Aug 31 11:44:51 1955 from XXXX.kc.res.rr.com

No wonder I feel old.

Now that a lot of the back end of our Arena Check-in system is in place and ready to go, I’ve been focusing my efforts on the fun part… Themes!

This school year’s theme is “Go Fish”, so I set about seeing if I could come up with something fun, without hiring an external designer/illustrator to do something for us. Here’s what I’ve got so far:

Still a work in progress, haven’t done the buttons yet (as anyone familiar with the default Arena theme has already spotted), but it’s coming along.

I’ve got 4 different scenes to wotk with for the fish, and the original art is vector-based, so I can move fish about as needed to get out of the way of text - I think I can even make them swim from one illustration to the other via the miracle of the Windows clipboard.

The artwork is from iStockPhoto, from a designer who goes by the username totallyjamie. The font is Chaloops Medium from Chank!.

Jason Wilson pointed me to some new pages at MainStreet.net today, wherein yours truly features rather prominently, along with Jason Powell and Mark Rock (and a few others) with the ParrotCam at MinistryTech from last April. There is also a picture of Justin.

This past weekend had the family taking a quick trip out to Connecticut to visit my grandmother for her 90th birthday party, a surprise celebration my dad and aunts have benn putting together for a couple of years now.

We flew up to Hartford on Saturday and rented a car to drive down to New Haven/Orange where I scored a room at the Courtyard Marriott for about 35% of the regular rate, thanks to Moreno’s tips on how to make Priceline work for you.

We met my dad who was holding a spot in line at Frank Pepe’s while the rest of the gang showed up. After some waiting, my aunt and grandmother showed up. I walked across the street and asked my grandmother if I could help a little old lady across the street After about half a second, it registered who asked her, and I got a big hug for my troubles after we got to the sidewalk. She had no idea we were coming to visit. A little later on, my brother showed up as well, on his way to Dublin. We got a table and had some great apizza. After the meal, our waitress got up and announced to the whole place that it was Bev’s 90th birthday and had them all sing. My grandmother told them all that her brother was one of the early customers of the place, and that she and her cousin were sitting in nearly the exact same spot when the radio came on to announce that President Roosevelt had died. Turns out the waitress was Frank Pepe’s granddaughter and that it’s still very much a family business, and they were honored to still have some one of their early customers. My grandmother promised to be back for her 100th.

The following day, more extended family gathered for another surprise at the Norm Bloom & Sons quay. My grandmother had been told that we were going to visit a “maritime museum”. After a little resistance from her and some convincing on my aunt’s part that she REALLY needed to come with us, she saw the extended family all wearing matching T-shirts for the occasion. As she’s digesting this and more family are trickling in, we take her around the back of the building where she sees the Catherine M. Wedmore, her grandfather’s oyster boat, set up for a Sunday afternoon cruise and picnic. The last time she’d seen the boat was 1951. It was sold by the Wedmores to the Blooms in the 1960s. 84 years after her maiden voyage, she’s still working the bay dredging for clams and oysters.

We have family footage of a similar sunday afternoon picnic, from 1940.

A reporter from the Stamford Advocate came along and wrote up a story.

Once out in the bay, we dropped one of the dredges and scooped up a bunch of fresh oysters, which were shucked and eaten. In one shell we found a small crab about the size of a dime rooming with one of the oysters. The girls had lots of fun meeting their 2nd cousin Adam (my Cousin Jeff’s son), and my cousin Catherine and her hubby gave my grandmother one more surprise, that she was going to have a fourth great-grandchild in February.

Lots of pictures were taken. Pics to follow.

My eldest, Faith (who will be *gasp* FIVE next month) showed from a very early age (6 months?) that she was probably going to be one of these extreme-sports adrenaline junkies.

Today, she proved it. She begged us to go on the EuroBungy ride at the Stampede. 10 bucks later, she was harnessed up:

And not too long after, she was flying:

She absolutely loved it, and followed it up with a trip on one of the kiddie coasters. She’ll be moving up to the big stuff before too long. We’ve been saying for years that she’ll be a coaster junkie by the time she’s six.

I had a great chat this morning with Gregg Hatton-Fearnley, the IS/IT director at Centre Street Church here in Calgary.

The main purpose of my meeting with Gregg was to introduce myself and tell him about CITRT. CSC is the second largest church in Canada (the largest is either in Toronto or Winnipeg, he couldn’t recall which). Like most people who hear about CITRT, he was quite enthusiastic about the idea, and was bemoaning the lack of funds in his budget to go to the fall event at Seacoast. I have high hopes of seeing him at spring RT, though.

CSC has got a great facility which, as most of us have experienced, they practically had outgrown the day they opened the building. I was very impressed with Gregg’s openness and transparency about some of the big challenges he faces - he shared a lot for a first date!

Unsurprisingly, his biggest hurdles aren’t so much technological as they are human. (Has anyone else had problems getting the Worship Arts people to play nice on the network? Yeah, thought so. Wasn’t that the whole point of MinistryTECH? )

The other really big challenge CSC faces is personnel. The Calgary job market is tremendously competitive, with entry-level fast food workers fetching a starting wage of $15/hour, and a low-level network admin commands a 6-figure salary. It’s hard enough finding people to work for church wages in a sane job market, it’s got to be ridiculously hard to do in Calgary, especially when Big Oil is snapping up all the IT people it can, and for huge money.

CSC is also a recent addition to Fellowship One’s growing list of customers, and I got to experience some quirks of the check-in system on Saturday night when Andrea and I went to check out.

CITRT is truly an international bunch now… We’ve got Canadians and Australians on board too. I hope to see Gregg on IRC or the Talkcast soon

Naturally, no church visiti is complete without pictures from the facility:

This is the projector set for one of the side screens. This thing is a monster. Hard to tell from the picture, but each one is about the size of a coffin. These are mounted in the tech booth at the back of the 2400-seat sanctuary. The throw on these is a good 200 feet, possibly more.

The video control room:

Video Switcher:

A decorative waterfall in the narthex. The sculpture in the middle can be removed to turn it into a baptismal pool. The “chandeliers” nearby each have about 32 dimmer channels and are built to support stage lighting as well.

The other side of the narthex:

I’ve been on vacation up in the great frozen cool north, in Calgary to visit my mom. My brother cashed in some of his frequent flier points to come visit as well. The story so far:

Thursday, 6/26: Arrived in Canada. Finished a novel. Got eaten by mosquitoes.

Friday, 6/27 : Went to the University to get some books for the girls and see my mom’s office. Took the girls to the wading pool. Bought some new hiking/work shoes. Got eaten by mosquitoes.

Saturday, 6/28 : Went hiking in Kananaskis Country, up Ribbon Creek. about 9km round trip, Faith walked the whole way. Went with a group of seniors from my mom’s church. Got left in the dust. These people go hiking nearly every week. Finished another novel. Got eaten by mosquitoes.

Sunday, 6/29 : Went to church with my mom. Met some of the hikers again. Finished another novel. Picked up my brother at the airport. Got eaten by mosquitoes.

Monday, 6/30: Made pilgrimage to IKEA. Lunch with my mom’s friends. Music night at my mom’s house. Finished another novel. Got eaten by mosquitoes.

Tuesday, 7/1 : Canada Day! Went downtown to the festivities on Prince’s Island. Went to see fireworks afterwards. Got eaten by mosquitoes.

Wednesday, 7/2 : Exhausted. Down day. Went to the city library to get some videos for the girls. Got eaten by mosquitoes.

Thursday, 7/3 : Went to the Royal Tyrell Museum in Drumheller. Went out for dinner and drinks with Andrea and my brother. Got eaten by mosquitoes.

Friday, 7/4 : Rested at home. Took the girls to the wading pool. Got eaten by mosquitoes. Introduced Andrea to Canadian Tire.

Saturday, 7/5: Spent the day at the Calgary Zoo Introduced Andrea to poutine. Caught evening service at Centre Street Church. Introduced girls to Tim Horton’s. Got eaten by mosquitoes.

Sunday, 7/6 plans: Stampede Breakfast at my mom’s church. Maybe get together with my godmother, as I haven’t seen her in years. Probably will get eaten by mosquitoes.

Monday, 7/7 plans: Meeting with Gregg, the IT Director at CSC to extend the hospitable hand of CITRT. Going to stampede events. Probably will get eaten by mosquitoes.

Tuesday, 7/8: Fly home. No mosquitoes, I hope.

Wednesday, 7/9: Back to the daily grind to try and fix everything that broke the second I stepped on the plane to Calgary. Rejoice in the lack of mosquitoes.

Canadian things to do:

• Tim Horton’s
• Poutine
• Canadian Tire
• Canada Day
• IKEA (OK, not really Canadian, but we don’t have IKEA in KC)
• Hockey (wrong season )
• Mountains
• Curling (wrong season )

Shatto Milk was having an open house today at their farm up in Osborn, MO, and we thought it would be a fun thing to do with the girls. While Andrea was up at Troost Elementary for COR’s Bless The School missions project, I took the kids up to the farm.

They had a bounce house, hot dogs, beans, samples of their milk (and requisite cookies to go with, provided by Midwest Airlines). The girls got to see the bottling plant, the bottle washer, and the milking parlor, and got to pet some cows and some of the new calves (a few weeks old!). A good time was had by all and the weather was absolutely perfect: low 80s, sunny, with a breeze.

It was nice enough to go up in my car instead of the minivan (My car has been referred to by some as my “Dave Ramsey Car”). My car’s air conditioner is currently out of commission, so summer trips in it with the kids are rare. Luckily, it’s highway most of the way, so it’s easy to get a good breeze going in the car even with the windows shut. I filled up before I left and when I got back. Total round trip: 158.7 miles. Total fuel consumed: 3.8 gallons. Yes, you saw that right. My $1700, 16-year-old Corolla got 41.7 mpg on that trip. Take that, Prius!

I’ll edit this post later to add linky goodness and pictures!

Take a look at this data coverage map from Sprint. Orange is EV-DO, Yellow is 1xRTT:

That spot with lots and lots of EV-DO, disproportionate to the rest of the country? That’s Alberta. The entire province is a mere 3.5 million people.

The isolated (but large) blob in the northeastern part of the province is centered on Fort McMurray, population 47,000. The EVDO coverage area is probably several thousand square miles.

Three guesses where all the oil is

I thought it rather ironic that as I was installing my HA firewall cluster that I hadn’t planned the whole hardware redundancy thing all the way through.

In order to install the new machine and the NICs, I had brought a screwdriver to mount rack rails and such. This particular screwdriver was one of the ratcheting kind, and it’s been a poorly functioning department fixture since before I arrived. Today, it decided to completely and catastrophically fail. One moment, I’m turning a screw, the next finds my hand holding about half a dozen pieces of the ratcheting mechanism, and the screwdriver shaft spinning freely and uselessly.

… and in my planning to build the HA cluster for the firewall, I’d neglected to brnig a spare screwdriver in case that hardware failed (which we’d expected it to do long ago). Luckily, one of our “neighbours” happened to have one with him and let me borrow it.

Moral of the story, make sure you have full hardware redundancy, including your screwdrivers.