Streaming My Consciousness

(or, How Amazon Cloudwatch helps manage Wowza server load)

This morning I woke up to two things: Beautiful Kansas City February weather (aka, an ice storm), and a voicemail from the Senior Pastor, asking if we had sufficient online capacity to support a larger-than-usual stream audience. Online worship streaming is a great option for these weather events that have been so common this winter (and not just in the KC area – we see increased online attendance when the weather gets foul elsewhere, like the DC storms of a few weeks ago).

My first indication that this was going to be a big event was Woopra showing 30 people on the web page half an hour before we start sending any kind of video (which is itself 75 minutes before we actually start the morning service). Usually there are two or three. Fifteen minutes after we started sending video, we were already cranking out 20-30 streams (again, we usually only have a small handful at this point).

AWS CPU Usage

Most weeks, we run two Wowza repeaters pulling from a single origin server, which gives us plenty of capacity. I had to spin up a third repeater by the beginning of the pre-service music, a fourth about 10 minutes later, and a fifth after five more minutes. I set my threshold for spinning a new server at 75% CPU on the repeaters, as indicated by the AWS CloudWatch monitors. In the case of a heavy influx of viewers, this gives the new instance enough time to get up and running before the other repeaters hit 100% CPU.  Wowza tells me this is at about 180Mbit/sec on a small instance, which for us means around 300 streams. The CPU threshold of 75% works out to about 260 streams.

Unfortunately for our online worshipers, our web server was bogging down pretty hard at

Web Server CPU usage

the beginning of the service, where the two CPU cores were maxed out for about 15-20 minutes, which translated into slower page loads. The database server wasn’t sweating too hard, so I suspect this could have been helped with better PHP caching. Fortunately for me, this had the effect of slowing down the rate of incoming streams, which allowed me to get new repeaters going before the existing ones started choking.

You can see in the graph where we added new repeaters, and how fast they ramped up. It also shows how incredibly well Wowza’s built-in load balancing works. We eventually leveled out at a little over 1100 streams, which meant our EC2 instances were cranking out 600-700 Mbps for nearly an hour:

AWS Network Usage

Meanwhile, this is what we were seeing on Woopra (note the fortunate souls escaping the ice storm in Aruba and the Cayman Islands!):

Next step is to define rules in Cloudwatch for automatically scaling. For that to work, I’m going to need to build my own Wowza AMI, since the current method of starting repeaters involves sending the instance a startup package from the client. I’ll need to build this configuration into the server for CloudWatch scaling to work properly.

I mentioned a few posts back that I was looking for a way to automate startup and shutdown of the servers. Thanks to some great sleuthing by Justin Moore at Granger Community Church, I got some scripts to start from. I had to make some modifications to suit our exact purposes, but that was relatively easy.

Ingredients:

Linux system with:

• Java Runtime (in Ubuntu, the package is sun-java6-jre)
• Amazon EC2 API Tools (installed in /opt/ec2/tools)
• Wowza Startup Packages (installed in /root/ec2)
• EC2 keys (installed in /root/ec2)

Note: Because these scripts are run from cron, you’ll need to put all your environment variables to run EC2 at the beginning of each one.

I have 6 separate versions of the startup and termination scripts, one for each server I need to start. I could roll it into one big script, but putting them in their own individual ones not only lets me do an individual machine manually, I can run them all in parallel from cron, which shortens the startup time.

The startup script functions as follows:

1. Assign environment variables for EC2 and for the machine parameters
2. Launch machine with ec2-run-instances, redirect output to a temporary file*
3. Parse temporary file and extract the instance ID, and put it into an environment variable
4. Write contents of instance ID environment variable to a file in /root/ec2 for use by the shutdown script
5. Wait 30 seconds
6. Start checking instance status every so we know when it’s running (wait 10 seconds to check again if it’s not)
7. Attach any EBS volumes (Optional – I don’t currently need this, so it’s commented out)
8. Assign Elastic IP
9. Execute any additional commands via ssh (Optional, I don’t have any that need to run)

* The original scripts use /tmp/a, which is fine, but I had to make each script do its own temporary file since all 6 were running simultaneously and I ran into problems with getting the right Instance IDs set.

The shutdown script works like this:

1. Query AWS for all running instances
2. Issue EC2 termination call
3. ???
4. PROFIT!

Lastly, put it in your crontab:

# m h  dom mon dow   command
15 8 * * 0 /root/start-windows.sh
25 8 * * 0 /root/start-origin.sh
25 8 * * 0 /root/start-iphone.sh
25 8 * * 0 /root/start-repeater1.sh
25 8 * * 0 /root/start-repeater2.sh
25 8 * * 0 /root/start-repeater3.sh

0 19 * * 0 /root/term-windows.sh
0 19 * * 0 /root/term-iphone.sh
0 19 * * 0 /root/term-origin.sh
0 19 * * 0 /root/term-repeater1.sh
0 19 * * 0 /root/term-repeater2.sh
0 19 * * 0 /root/term-repeater3.sh

This starts up all of them (except the Windows instance which needs more time) at 8:25 on Sunday morning and shuts them down at 7 on Sunday evening.  (be sure that if you’re using GMT on your Linux box to take that into account).

If you’re using an encoder that can be started and stopped on a schedule, synchronize your times with this, and you’ll be golden. The Wowza EC2 images take about 60-90 seconds to fully get up and running, and the Windows one takes about 10-15 min. Currently, the Windows server pulls from the encoder via a 1:1 NAT rule, so the WME instance can be running before the EC2 server is going. When EC2 is ready, it simply connects to the encoder and is off and running.

We started streaming to iPhones today. Huge success, way easier than it ought to be, now that the iPhone does HTTP streaming and Wowza’s V2 software supports everything needed to stream to the FruitFone. All we had to do was shell out $250 to MainConcept for their AAC encoder plugin for Flash Media Live Encoder. (Although we subsequently discovered that there is a bug with the MainConcept encoder that cause audio sync problems on iPhone, so we’ve moved iPhone encoding off to a separate machine)

There are a lot of layers to this onion, so I put together a block diagram that links everything from the cameras to client.

I promised everyone at MinistryTECH way back in April that I’d rehash my presentation on my blog. Here we are almost 6 weeks later, and I owe you all!

In the beginning…

In April of 2008, we were asked to stream one of our Easter services live on the Web as an experiment. It was a very simple concoction of a Firewire cable coming out of one of the Sony DVCPro decks in the video rack, running into a Dell Optiplex 745 with a cheap add-in FireWire card, and Windows Media Encoder.  We made some slide loops into videos that we could bumper the live stream with (WME is good at switching recorded sources into a live stream like that), and broadcast the IMAG feed. It was simple, and we streamed 320×200@15fps to an audience of about 200 using a traditional streaming CDN. Total cost was about $300.

When our senior pastor started casting his vision of an Internet Campus which would revolve around a live (or nearly-live) stream of worship, it became pretty apparent that this was not going to scale well or cheaply. Over the course of the summer of 2008, we started exploring creative ways to do the impossible with nearly no money. We even looked at what it would take to run our own streaming servers, and that was also ruled out quickly due to the sheer bulk of cash that would have to be thrown at it and our lack thereof (this would have been so much easier if we were Congress and able to print it at will!).

One of the key requirements for our streaming platform was that it be able to stream Flash video in the interest of cross-platform compatibility. We love our Mac viewers enough that we didn’t want to subject them to Flip4Mac week after week. If we had any Linux viewers, we’d love them too, not wanting to let the lack of a Windows Media player keep our penguin-loving friends from hearing the Gospel.

It was around this time that Andrew Mitry, one of the regulars in the CITRT chat, informed us of Amazon’s fledgling cloud computing offering called EC2, which allowed for on-demand computing at ridiculously low prices. We were exploring licensing a software platform and creating our own images when we discovered that Wowza, one of the software platforms was available as a paid image for EC2.

Not only were we excited at the prospect of a software company that was willing to embrace this new platform, the ability to pay for that software based entirely on usage with no up-front licensing was very attractive for an experimental project with virtually no money available to it.

We now had a platform, it was time to start looking at how we were going to encode the video and get it up there.

Continue on to Part 2: Encoding