While I’m on the topic of Slashdotting, I had a conversation with Clif the other day about what it takes to scale up sites to handle the onslaught of traffic generated by people looking for web coverage of Election Night. It’s one of those nasty scalability problems where if you get it wrong, you’re utterly screwed and don’t get a do-over or a few hours to fix it. If you’re in the business of selling eyeballs and your site goes dark during the Big Game, you’re pretty much hosed. And broke.
Data Center Knowledge has a neat article about what goes on behind the scenes to ramp up for an event of this magnitude.
Well, OK, it wasn’t Slashdot that was the culprit this time, but rather the pro blog Consumerist (if we’re a megachurch, does Consumerist count as a megablog? It claims nearly 3 million unique visitors a month)
Last week, Clif posted about his experience at Best Buy. Seems the folks at Gawker Media got wind of the story (Best Buy is a perennial favourite target of theirs) and posted it at 10:21am Eastern, 9:21 in KC. Here’s what happened to our Wordpress server:
Apache Processes, November 5
Web Datacenter traffic, November 5 (% of 10Mbps link)
Wow. I noticed the odd traffic behaviour (that particular server gets very little traffic most of the time) when I got in the office, and called a few folks to see if they’d done anything that would cause this. When that came up empty, I started looking at the access logs on the server and noticed a lot of referrer traffic from Consumerist. I threw AWstats onto the server to grok the apache logs. At posting time, Clif’s blog post had seen around 7000 visitors. Apache peaks out at a point due to the MaxClients directive, in order to keep the CPU from saturating and killing the site.
It’s always fun to see new an interesting traffic patterns. It’s very helpful to have active monitoring to tell us when things leap outside the bounds of normalcy.
She just built a large structure out of Lego, and it has a bell.
She was explaining that it’s a school, called "Zoo" (it has a piece above the door that says so, presumably from a zoo set), and that it was a high school.
Sounds like she has a pretty good grasp on what high school really is.
Given Kansas City’s central location on the IP backbones that traverse the US, it should be a heck of a lot easier than it is to get some semblance of decent bandwidth at the church (and other locations). Sadly, the best we can do is 1.5Mbit DSL (even if the line has been tested from our demarc to support 7.5Mbit, AT&T refuses to sell us any more than 1.5Mbit because their computers say that’s the best they can do) or bonded T1s.Even then, there’s a finite limit to what what we can achieve with that, because we have a finite amount of copper feeding the building.
Anything else involves construction costs in the $50K and up range. Even amortized over a 5-year contract, that still tacks on nearly $1000 to the monthly bill. Point-to-point wireless to somewhere with fiber would cost almost as much.
It amazes me that in one of the more affluent suburbs in the country, the best we can do is no better than what you get in and around most small towns in rural western Kansas.
Recently, we made some changes to the DNS infrastructure on our public wireless networks which has had the unintended consequence of breaking things when our laptop users are plugged into the LAN and have their wireless active. Brian and I have wrangled with this in the office, but we simply turned off the wireless as a workaround.
What’s happening is that when connected to both networks, the wireless has a higher priority by default, and so it resolves DNS via that interface. This is problematic when trying to access an internal resource, because our DNS is set to have a default resolution to our website for *.cor.org. To complicate matters further, Arena behaves differently when you’re on the guest network (sends to a forms-based auth portal instead of using IE integrated authentication).
After much digging, I found out how to change interface priority. Here’s the process in XP screenshots (the process is similar in Vista):
1. Open your network connection properties (XP: Via control panel or right-click on Network Places, then select Properties. Vista: Go to Network and Sharing Center and select “Manage Network Connections in the links on the left)
XP Network Properties
2. On the menu bar (press Alt to show it in Vista), Select Advanced, then “Advanced Settings”
Advanced Network Properties Dialog (XP)
3. Move the Wired LAN Connection (By Default, “Local Area Connection”) to the top, followed by the wireless connection. Make sure that any VPN virtual adapters come after these, otherwise the VPN will only use the ones above it. This tends to be problematic if you’re using split tunneling, as it will kill any network connection you have.
4. Hit OK, and you’re good to go.
The last few days have been rather stressful.. Our shiny new web infrastructure at COR has been throwing major temper tantrums, which means I’ve been rather busy of late. Today, it melted down half a dozen times after I thought I’d fixed it. Each time it did something different. (and if that wasn’t enough excitement, our upstream provider had a BGP issue this morning that knocked their entire customer base off the web for about 5 minutes)
All you folks that hit our website, thank you for your patience. These have been trying times.
This morning, I noticed something very odd. And again this evening as I’m migrating the data to a new server.
root@corweb1:/content/sites# uptime
11:35:56 up -24855 days, -3:-14, 4 users, load average: 0.00, 0.00, 0.00
Negative uptime??? What?
root@corweb1:/content/sites# date
Wed Aug 31 11:50:17 CDT 1955
Perhaps running apt-get install flux-capacitor wasn’t such a good idea, as the machine seems to be performing on about the level of a computer from 1955.
It’s become pretty obvious that something is very ill on that box, and I think it’s time to ditch VMWare Server for ESXi. Until then, we’re moving the servers over to the bare metal on the other box (which includes the blog server, it’s already been moved).
Last login: Wed Aug 31 11:44:51 1955 from XXXX.kc.res.rr.com
No wonder I feel old.
Now that a lot of the back end of our Arena Check-in system is in place and ready to go, I’ve been focusing my efforts on the fun part… Themes!
This school year’s theme is “Go Fish”, so I set about seeing if I could come up with something fun, without hiring an external designer/illustrator to do something for us. Here’s what I’ve got so far:
Still a work in progress, haven’t done the buttons yet (as anyone familiar with the default Arena theme has already spotted), but it’s coming along.
I’ve got 4 different scenes to wotk with for the fish, and the original art is vector-based, so I can move fish about as needed to get out of the way of text - I think I can even make them swim from one illustration to the other via the miracle of the Windows clipboard.
The artwork is from iStockPhoto, from a designer who goes by the username totallyjamie. The font is Chaloops Medium from Chank!.
Jason Wilson pointed me to some new pages at MainStreet.net today, wherein yours truly features rather prominently, along with Jason Powell and Mark Rock (and a few others) with the ParrotCam at MinistryTech from last April. There is also a picture of Justin.
Entries (RSS)