Streaming My Consciousness

Yesterday was the first live test of our EC2 automation, and it worked beautifully. One thing I discovered was that when the Wowza servers shut down, Flash Media Encoder simply attempts to reconnect to the server until it succeeds. This is very helpful, since I can just leave the VT5 system on a loop or on the program feed, and as soon as the servers are ready, they’ll start broadcasting.

That way, if something happens and nobody shows up, we’ll at least have something going on the stream.

If you’re using an unattended encoder, you can also take advantage of this (and even schedule start/stop of the encoder software, but it doesn’t have to be timed exactly to the server start/stop.

It’s all well and good that we’re putting this stream out there… But if we’re not reaching anyone, it’s kind of a big waste of money and time. How do we go about finding out how many people we’re reaching?

Fortunately, Wowza has a built-in mechanism for reporting the number of active stream connections via an HTTP connection to the streaming server. It does this primarily for load balancing purposes but the data is easily parsed for other things as well.

I currently have a VB Script (the code is horrid, because I suck at VB)  that connects to each of the streaming servers, parses the response into a numerical value, adds them up to get a total stream count. The script runs on a 5-second delay loop, keeps track of the peak, and gives me the following output, where the red area is the origin server, green is the repeaters, and blue is the iPhone streams. Windows streams are gleaned directly from Windows Media Services.

Stream count output

That’s all well and good, but how many actual people are watching? We know there are several people who watch this alone, but others do it in groups or with their family. Initially, when we benchmarked other churches, we were told that a ratio of about 1.8 people per stream was a pretty reliable guess. We went with that for a while, as we gathered our own data.

To gather our data, we created a sign-in/feedback form for our web audience that functions very much like the friendship pad we pass around in our physical worship services. One key question that you find only online is “how many people are worshipping today?” Based on the aggregated data from these forms (we’ve collected nearly 8,000), we found that our ratio was closer to 1.7, so we started using that for the purposes of reporting. We typically see about 40% of our peak stream count send us a feedback form, so the per-stream count is probably a fairly representative sample. Periodically, we’ll see the ratio jump up to 2:1 in special circumstances such as inclement weather at our central campus, and we’ll adjust the numbers for that service accordingly.

I’m sure there’s a better way than my cheesy vb script, but it works for now.

I mentioned a few posts back that I was looking for a way to automate startup and shutdown of the servers. Thanks to some great sleuthing by Justin Moore at Granger Community Church, I got some scripts to start from. I had to make some modifications to suit our exact purposes, but that was relatively easy.

Ingredients:

Linux system with:

• Java Runtime (in Ubuntu, the package is sun-java6-jre)
• Amazon EC2 API Tools (installed in /opt/ec2/tools)
• Wowza Startup Packages (installed in /root/ec2)
• EC2 keys (installed in /root/ec2)

Note: Because these scripts are run from cron, you’ll need to put all your environment variables to run EC2 at the beginning of each one.

I have 6 separate versions of the startup and termination scripts, one for each server I need to start. I could roll it into one big script, but putting them in their own individual ones not only lets me do an individual machine manually, I can run them all in parallel from cron, which shortens the startup time.

The startup script functions as follows:

1. Assign environment variables for EC2 and for the machine parameters
2. Launch machine with ec2-run-instances, redirect output to a temporary file*
3. Parse temporary file and extract the instance ID, and put it into an environment variable
4. Write contents of instance ID environment variable to a file in /root/ec2 for use by the shutdown script
5. Wait 30 seconds
6. Start checking instance status every so we know when it’s running (wait 10 seconds to check again if it’s not)
7. Attach any EBS volumes (Optional – I don’t currently need this, so it’s commented out)
8. Assign Elastic IP
9. Execute any additional commands via ssh (Optional, I don’t have any that need to run)

* The original scripts use /tmp/a, which is fine, but I had to make each script do its own temporary file since all 6 were running simultaneously and I ran into problems with getting the right Instance IDs set.

The shutdown script works like this:

1. Query AWS for all running instances
2. Issue EC2 termination call
3. ???
4. PROFIT!

Lastly, put it in your crontab:

# m h  dom mon dow   command
15 8 * * 0 /root/start-windows.sh
25 8 * * 0 /root/start-origin.sh
25 8 * * 0 /root/start-iphone.sh
25 8 * * 0 /root/start-repeater1.sh
25 8 * * 0 /root/start-repeater2.sh
25 8 * * 0 /root/start-repeater3.sh

0 19 * * 0 /root/term-windows.sh
0 19 * * 0 /root/term-iphone.sh
0 19 * * 0 /root/term-origin.sh
0 19 * * 0 /root/term-repeater1.sh
0 19 * * 0 /root/term-repeater2.sh
0 19 * * 0 /root/term-repeater3.sh

This starts up all of them (except the Windows instance which needs more time) at 8:25 on Sunday morning and shuts them down at 7 on Sunday evening.  (be sure that if you’re using GMT on your Linux box to take that into account).

If you’re using an encoder that can be started and stopped on a schedule, synchronize your times with this, and you’ll be golden. The Wowza EC2 images take about 60-90 seconds to fully get up and running, and the Windows one takes about 10-15 min. Currently, the Windows server pulls from the encoder via a 1:1 NAT rule, so the WME instance can be running before the EC2 server is going. When EC2 is ready, it simply connects to the encoder and is off and running.

I promised everyone at MinistryTECH way back in April that I’d rehash my presentation on my blog. Here we are almost 6 weeks later, and I owe you all!

In the beginning…

In April of 2008, we were asked to stream one of our Easter services live on the Web as an experiment. It was a very simple concoction of a Firewire cable coming out of one of the Sony DVCPro decks in the video rack, running into a Dell Optiplex 745 with a cheap add-in FireWire card, and Windows Media Encoder.  We made some slide loops into videos that we could bumper the live stream with (WME is good at switching recorded sources into a live stream like that), and broadcast the IMAG feed. It was simple, and we streamed 320×200@15fps to an audience of about 200 using a traditional streaming CDN. Total cost was about $300.

When our senior pastor started casting his vision of an Internet Campus which would revolve around a live (or nearly-live) stream of worship, it became pretty apparent that this was not going to scale well or cheaply. Over the course of the summer of 2008, we started exploring creative ways to do the impossible with nearly no money. We even looked at what it would take to run our own streaming servers, and that was also ruled out quickly due to the sheer bulk of cash that would have to be thrown at it and our lack thereof (this would have been so much easier if we were Congress and able to print it at will!).

One of the key requirements for our streaming platform was that it be able to stream Flash video in the interest of cross-platform compatibility. We love our Mac viewers enough that we didn’t want to subject them to Flip4Mac week after week. If we had any Linux viewers, we’d love them too, not wanting to let the lack of a Windows Media player keep our penguin-loving friends from hearing the Gospel.

It was around this time that Andrew Mitry, one of the regulars in the CITRT chat, informed us of Amazon’s fledgling cloud computing offering called EC2, which allowed for on-demand computing at ridiculously low prices. We were exploring licensing a software platform and creating our own images when we discovered that Wowza, one of the software platforms was available as a paid image for EC2.

Not only were we excited at the prospect of a software company that was willing to embrace this new platform, the ability to pay for that software based entirely on usage with no up-front licensing was very attractive for an experimental project with virtually no money available to it.

We now had a platform, it was time to start looking at how we were going to encode the video and get it up there.

Continue on to Part 2: Encoding