Streaming My Consciousness

Digital distribution is the future of media. Physical media is dead. Yeah, I know, you’re heard it but don’t believe it.

Today, Penny preached at Resurrection and showed a clip from The Blind Side. Andrea and I had been meaning to see the movie for a while. Since the kids actually went to bed quietly and early, we figured we’d RedBox it and have a nice movie night at home.

One problem though – when your pastor preaches to a couple thousand people and include a clip, there’s a pretty good chance that you’re going to have a hard time finding said movie anywhere near the church. As a backup plan, I fired up google and searched for a torrent version of it. Within about 30 seconds, it was downloading. I then went to Redbox.com, searched for the movie (got lucky and the local box actually had one!), reserved it online, hopped into the car, drove over to the price chopper and picked up the movie. (in retrospect, it would have been faster to take my bike, but it was warm and VERY humid) . I took less than thirty seconds at the kiosk, and drove home. As I sat down in front of my computer, the download had just completed.

Total time elapsed: 17 minutes. In that time, 700MB had downloaded, and hadn’t even uploaded a complete single block (so don’t worry, MPAA, I didn’t actually share any of it). Since I had the DVD, I watched that instead, and had to confront issues such as cleaning the last renter’s fingerprints off the disc and sitting through commercials on the DVD that I paid to rent. I’ll go on faith that the file I downloaded contained the video, and it was kinda nice having a backup plan in case the disc was unusable. Either way, the content owners did get paid.

The process of reserving and picking up a movie on redbox is insanely easy and quick. And downloading a torrent was even easier. If you’re in the business of physical entertainment media, I hope that you’re trying to figure out your exit from that strategy. Browsing and renting at Blockbuster is a painful and expensive process, and that’s why their days are numbered. Redbox has a good thing going, but looking five to ten years down the line, they should be seeing a world of digitally distributed content, not physical media. Netflix has the right idea, but their streaming catalog could use much improvement.

How can content producers leverage the ease and efficiency of peer-to-peer technologies like BitTorrent? The distributed distribution model is incredibly efficient as several companies have discovered where software distribution is concerned. They need to stop fearing peer-to-peer digital distribution and instead leverage its power.

One of the big challenges of streaming to the web is the sheer diversity of devices out there.

This past week, I pushed out some modifications to the player code on our live page that switches the player code based on what the user is connecting with. The genesis of this change was a problem with our change to JW Player Version 5 causing our PlayStation users to no longer be able to watch our video since JW v5 requires Flash 10 and Sony apparently doesn’t care about its customers. After a successful test with the Playstation, I extended the code to provide an HTML5 <VIDEO> tag for our iPhone users (allowing us to clear up some the clutter on the sidebar), as well as MMS and RTSP links around a graphic mimicking the Flash-based player in order to provide a consistent user experience for our Android/WebOS/BlackBerry/WinMo users.

EDIT: The main reason I’m not doing straight HTML5 with Flash fallback (a much more elegant solution) is that we’re sending out VP6 for our flash users and a lower-bandwidth h.264 stream for our mobile users. We’re not currently using h.264 for our flash users because of the poor quality of the h.264 encoder in Flash Media Live Encoder. Once we get a “real encoder“, we’ll send out a single set of h.264 streams and use HTML5 with fallback.

The code is here.

I’ve added two pages where I’m keeping a list of IP cameras and mobile devices known to work with Wowza.

(or, How Amazon Cloudwatch helps manage Wowza server load)

This morning I woke up to two things: Beautiful Kansas City February weather (aka, an ice storm), and a voicemail from the Senior Pastor, asking if we had sufficient online capacity to support a larger-than-usual stream audience. Online worship streaming is a great option for these weather events that have been so common this winter (and not just in the KC area – we see increased online attendance when the weather gets foul elsewhere, like the DC storms of a few weeks ago).

My first indication that this was going to be a big event was Woopra showing 30 people on the web page half an hour before we start sending any kind of video (which is itself 75 minutes before we actually start the morning service). Usually there are two or three. Fifteen minutes after we started sending video, we were already cranking out 20-30 streams (again, we usually only have a small handful at this point).

AWS CPU Usage

Most weeks, we run two Wowza repeaters pulling from a single origin server, which gives us plenty of capacity. I had to spin up a third repeater by the beginning of the pre-service music, a fourth about 10 minutes later, and a fifth after five more minutes. I set my threshold for spinning a new server at 75% CPU on the repeaters, as indicated by the AWS CloudWatch monitors. In the case of a heavy influx of viewers, this gives the new instance enough time to get up and running before the other repeaters hit 100% CPU.  Wowza tells me this is at about 180Mbit/sec on a small instance, which for us means around 300 streams. The CPU threshold of 75% works out to about 260 streams.

Unfortunately for our online worshipers, our web server was bogging down pretty hard at

Web Server CPU usage

the beginning of the service, where the two CPU cores were maxed out for about 15-20 minutes, which translated into slower page loads. The database server wasn’t sweating too hard, so I suspect this could have been helped with better PHP caching. Fortunately for me, this had the effect of slowing down the rate of incoming streams, which allowed me to get new repeaters going before the existing ones started choking.

You can see in the graph where we added new repeaters, and how fast they ramped up. It also shows how incredibly well Wowza’s built-in load balancing works. We eventually leveled out at a little over 1100 streams, which meant our EC2 instances were cranking out 600-700 Mbps for nearly an hour:

AWS Network Usage

Meanwhile, this is what we were seeing on Woopra (note the fortunate souls escaping the ice storm in Aruba and the Cayman Islands!):

Next step is to define rules in Cloudwatch for automatically scaling. For that to work, I’m going to need to build my own Wowza AMI, since the current method of starting repeaters involves sending the instance a startup package from the client. I’ll need to build this configuration into the server for CloudWatch scaling to work properly.

Now that we’ve gotten streaming to computers down pat, I’ve set my sights on delivering a good experience for mobile users. Unfortunately, with the wide variety of mobile platforms out there, this is not an especially easy task. The Mac/PC/Linux issues are complicated enough, and it gets really tricky when the platform ecosystem has half a dozen major players (and a truckload of minor ones)

Since July or so, we’ve been using a preview version of the recently released Wowza V2 server software to deliver our video content to iPhone/iPod devices that support Apple’s new HTTP Streaming format. With minimal changes, Wowza V2 can also rebroadcast the same H.264/AAC stream over RTSP, which reaches a lot more devices. But this is where it gets complicated. BlackBerry has been supporting RTSP for some time, but it’s only recently that they’ve supported h.264/AAC media. According to their KB article on the subject, you can do H.264 on the following:

• Bold 9000/9700
• Tour 9630
• Storm 9500/9520/9530/9550
• Curve 8900/8520

Most HTC phones have a streaming media app that supports RTSP, but only recent versions seem support H.264. For example, my Mogul has the app, but I can only hear the audio. Brian‘s Touch Pro 2 gets both (and on the TP2′s WVGA screen, it looks amazing!).

Windows Media Player supports RTSP, but doesn’t come with an H.264 codec (even in Windows 7!!!! BOOO!!!!). I have yet to get the RTSP stream to work on Windows Media Player. The mobile player doesn’t support RTSP at all, just MMS and HTTP (but not the same HTTP as Apple! Grr!), and with the 9.5 generation of Windows Media Services (2008), MMS has gone away in favor of HTTP (which Microsoft calls Smooth Streaming, also not supported on WiMo).

The Palm Pre is supposedly able to do RTSP and H.264, but I’m waiting to hear back from one of our pre-wielding pastors to see if this is actually the case.

Thanks to Daryl Hunter at lifechurch.tv for letting me know that it works on his HTC Hero (Android 1.5). It seems that on Android you can’t manually enter an RTSP URL into the browser bar, but a web link or tinyurl redirect that goes to an RTSP URL does work.

Meanwhile, VLC player will play just about anything you throw at it, including the RTMP flash stream. Pity it’s not available in a mobile version.

So, as it stands now, in order to deliver a mobile experience to as many people as possible, I’m still going to need to run a separate Windows Media server for our Windows Mobile clients, But everyone else should be able to pull from the “iPhone” stream (which I’m probably going to need to rename), as long as the device supports H.264/AAC and RTSP.

If you have a mobile device and want to see if yours can handle an H.264/AAC RTSP stream, Try the feed here. Feed is generally available on Sundays from 9:30 am to 1 pm, and 4 pm to 7 pm Central time. It’s also live right now for testing (I’ll remove this message when I shut it down).

Longtail recently released JW Player 5.0, but it had a bug that prevented it from being used with a Wowza load balance setup. It would catch the redirect and show the first few frames and then start buffering without end.

I just got the new 764 build and am happy to report that it works quite nicely now.

Earlier this week, I got an e-mail from Amazon Web Services, with some new goodies being announced.

The first was new pricing for Wowza. While the cost of each instance-hour is going up slightly, the cost of bandwidth is dropping about 15%. This makes me happy.

The second was that Wowza has released version 2. I’ve been working with preview versions of this since last summer with our iPhone streaming. They’ve made some very cool improvements to the product.

Lastly was an item that intrigued me. Amazon has, via Flash Media Server, added RTMP streaming capability to Cloudfront, Amazon’s cloud answer to CDN. Now instead of being able to widely distribute files, you have the capability of setting up a distribution that provides RTMP streaming of any supported video file in an S3 bucket. If you use S3 for on-demand video content, this is big for you. No longer do your viewers have to download the entire file (and run up the bandwidth meter in the process). They can now skip directly to the points they need and only use the bandwidth for stuff they actually watch.

This is potentially very good news for services that serve on-demand content from S3 (such as blip.tv)

It’s not so good news for the folks at Wowza, because I no longer have to spin up a Wowza instance to serve content stored in S3. Luckily for Wowza, the Cloudfront streaming doesn’t do live video.

It’ll be interesting to see how this plays out. As it currently stands, we could replace blip.tv with this functionality, but for the small cost, we get a whole lot of value from Blip.

For the last couple of years, we’ve used LED Christmas lights in our sanctuary. Considering how many we have (hundreds), the electricity savings are probably non-trivial.

All our LED strings in the sanctuary are plugged into either a stage dimmer or, where a dimmer port was unavailable, an Elation UniBar hooked into an RC4 Magic wireless DMX receiver (with the transmitter wired into DMX up in the catwalks). This allows us to control the Christmas lights along with the rest of the theatrical lighting via the Hog. It’s a very nice setup.

The other day, when Frank was running the stream, he saw the Christmas lights were fading in and out in sequence, and called up to the Penalty Box (the plexiglass-wrapped area at the back of house where the lighting operator and worship producer sit) and asked them to quit playing with the lights. As it turns out, they weren’t and the lights were all on. Mysteriously, they were fading in and out in sequence on the wide shot camera. When we looked at them on one of the other remote cameras, everything looked normal.

Then it hit me. I went to the remote control on the wide camera and cranked down the shutter speed, and lo, the lights gradually came together until they were all on. This is what it looked like:

Most stage dimmers operate by switching the AC cycle on and off via pulse width modulation. LEDs then only show one half of the AC sine wave, making them strobe, effectively reproducing the square-wave pulses that are modulating the dimmers. What We were seeing on the cameras was a beat frequency of the camera’s shutter speed and the strobing of the LEDs. You don’t see this on incandescent lighting because of the thermal persistence of the filaments. But why were the lights cycling at different times? Each one was connected to a different dimmer circuit, and those circuits are spread among the three AC phases coming into the dimmer room (which has a monster 2000-amp breaker).

So, if you’re shooting video of anything that has LED lights in it, make sure your shutter speed is at 1/60, or the lights are going to start acting strangely.

Since there’s been a flurry of interest lately on my series of posts on live church streaming, here’s an index to the entire set of posts:

• Live Streaming On a Budget (Part 1) – Genesis
• Live Streaming On a Budget (Part 2) – Encoding
• Live Streaming On a Budget (Part 3) – The Process
• Live Streaming on a Budget (Part 4) – How it works
• Live Streaming on a Budget (Part 5) – Automating EC2
• Live Streaming on a Budget (Intermission) Speedlinks!
• Live Streaming on a Budget (Part eleventy) – Metrics
• Live Streaming on a budget: Wowza Load Balancing (Part … um.. how many is it now?)
• Live Streaming on a Budget (Part eleventy point one) – Metrics Revisited

• MIDI Control Surfaces
• MIDI Surfaces: Korg NanoKONTROL
• MIDI Surfaces: Behringer BCF2000

For the benefit of the less technically inclined, I’ve moved scripts and code to their own section and updated posts that contained the code with links to the appropriate page.

Back in July, I made a post about metrics and a cheesy VB Script that got the job done, but wasn’t particularly elegant. I’ve since improved on this due to load balancing (I posted about that in September).

I’ve since then learned a bunch about RRDTool, and have put together a script that pulls the XML data, groks it, and then populates an RRD. The net result is that I get a graph like this:

This graph gives me the following information: The iPhone stream count (with a 10:1 vertical exaggeration), the Flash stream count, and the total viewer count, which is the sum of Flash and iPhone streams, multiplied by a factor of 1.7 (which we’ve found reasonably reflects how many actual people are watching, versus streams. Then the vertical red line shows the time the peak occurred, and the horizontal line shows the level of that peak. The actual peak numbers are listed on the bottom. The RRD and the graph are set up to take into account Windows streams, just as soon as I find a good way to pull that data from WMI via perl.

The general operation is as follows: There’s a script that’s started in a cron job 10 minutes after the servers are spun up, and it polls the origin server every 10 seconds for its counts and populates the RRD. There’s another cron job that runs every minute to generate a current graph, which is then displayed on a page with some javascript that refreshes the image in realtime. Then, at 12:15 and 6:30, there’s another cron job that takes a snapshot of the previous two hours, puts it into a web-accessible directory, and appends an HTML file with a link for easy access later.

All of the metrics scripts, automation scripts, and graphing tools live on a linux virtual machine that runs on our central campus.

This is big improvement over dumping a vbscript into a CSV and then graphing manually with Excel. This happens automatically, in real time, without me being there.