Streaming My Consciousness

Photo: bdtyre

Working in an IT shop, sometimes you have the luxury of forgetting just how bad consumer-grade hardware support has gotten. I’ve been absolutely spoiled by our Dell team, so the past few weeks have been a serious reality check.

About a year and a half ago, I bought a Toshiba laptop for my wife after her desktop computer’s power supply had unleashed its magic smoke. She was needing an upgrade, and we opted for the laptop since she was going to be starting school. After shopping arond, we found a good deal at Costco which included a color laser printer. Added bonus of buying at Costco is that on computers (and TVs) they extend the standard 1-year manufacturer warranty out to two years.

Somewhere around the middle of the fall semester, Andrea noticed the power connector inside the laptop was getting loose. Having encountered this sort of problem before (and I understand it’s actually quite common on consumer laptops), I suspected the plastic shell of the connector had come loose. They typically have three or four plastic pegs that secure the body to the circuit board, taking the mechanical strain off the soldered electrical leads. It’s not uncommon after repeated plug/unplug cycles for one or more of these pegs to break off, causing the connector to come loose. The long-term risk is that metal fatique will then break the electrical connectors. If you’re lucky, it won’t short out in the process. Business-grade laptops usually use much better connectors and methods of securing them. It’s just one of those corners you have to cut to sell $500 laptops to the masses.

Along the way, her touchpad started acting flaky as well, so we figured we’d call Costco and get this fixed while we still had some time left in the warranty. After trying to explain to someone in India for 45 minutes that the problem was not in the external power adapter, I got tired of him sending me on hold to talk to his supervisor, and asked him to transfer me. I spoke with Wade, who was quite helpful, and got the dispatch sent out to their service provider, an outfit in the Orlando area called “Encompass Service Solutions”. They sent me an empty box with packing material via 2-day air. That was two weeks ago.

Since I was sending the computer offsite to some company I’d never heard of, there was not only a high likelihood that they were gonna wipe the system and do a factory reload (crapware and all), there was absolutely no way I was trusting them with the security of our data (even with backups, I’d rather they not have access to my banking data, thankyouverymuch). So I picked up a blank hard drive at Micro Center and shipped it off with the blank.

A week later, it comes back, with the technician notes that the system was cleaned and the CPU “repacked”, and the OS reloaded. No mention of either of the original problems on the service ticket. Sure enough, the power connector is in even worse shape than it was when we sent it off. I call Costco and gripe. They’re not amused. A little later, I flip it over to put our hard drive back in and discover that someone went crazy with stickers. There are 4 new stickers with barcodes, and “Inspected by” sticker, and half a dozen “Warranty Void if broken or removed” stickers over the chassis screws, as well as those to the hard drive and memory bays. I’m a geek, so I’m genetically predisposed to ignore such stickers, and I carefully removed the ones on the drive and memory bays, and put our hard drive back in.

Then things got interesting. I power the machine on, and the display remains dark. No backlight, no image, nothing. I check das blinkenlights, and they’re behaving like a normal boot process. After some fumbling and troubleshooting, I hang an external monitor from it and fire it up again. That part works. It seems that the people in QA at Encompass were sleeping that day, because I’d imagine that a non-functional display normally would cause QA to kick it back to the tech who forgot to plug in the ribbin cable feeding the display. Apparently at Encompass, this is not the case.

So now we have a laptop that is effectively a compact desktop. A couple calls back and forth to Costco, and Encompass wants me to send it back to them for repair. Only here’s the problem, guys, I don’t trust you to not break it worse this time, and Andrea needs her laptop this week, that’s why we sent it to you when we did. Even better, you can’t e-mail me a UPS label for me to use the existing box, you insist on sending me another empty box via UPS. Not only is that insanely wasteful of diesel, jet fuel, and cardboard, and it adds 3 days to the process. Andrea can’t be without the laptop any longer than she already has. She has interviews this week and school starts up next week. I don’t care how fast you expedite it, you’ve proven already that you half-ass the job when it’s a rush. If I wanted that kind of shoddy work done on my laptop, I would have handed it off the the Geek Squad at a local Best Buy.

Wade, the guy at Costco, is trying to come up with alternatives. At this point, acceptable outcomes are that we get an onsite tech to do a motherboard replacement, find me a local depot in the Kansas City area, or you give us at least partial credit toward a replacement system at the warehouse.

When you buy a laptop, spring for the onsite support. If anything happens, it will save your sanity. Next time, I’m buying a Dell.

I’d love to hear your depot repair horror stories.

Now that we’ve gotten streaming to computers down pat, I’ve set my sights on delivering a good experience for mobile users. Unfortunately, with the wide variety of mobile platforms out there, this is not an especially easy task. The Mac/PC/Linux issues are complicated enough, and it gets really tricky when the platform ecosystem has half a dozen major players (and a truckload of minor ones)

Since July or so, we’ve been using a preview version of the recently released Wowza V2 server software to deliver our video content to iPhone/iPod devices that support Apple’s new HTTP Streaming format. With minimal changes, Wowza V2 can also rebroadcast the same H.264/AAC stream over RTSP, which reaches a lot more devices. But this is where it gets complicated. BlackBerry has been supporting RTSP for some time, but it’s only recently that they’ve supported h.264/AAC media. According to their KB article on the subject, you can do H.264 on the following:

• Bold 9000/9700
• Tour 9630
• Storm 9500/9520/9530/9550
• Curve 8900/8520

Most HTC phones have a streaming media app that supports RTSP, but only recent versions seem support H.264. For example, my Mogul has the app, but I can only hear the audio. Brian’s Touch Pro 2 gets both (and on the TP2’s WVGA screen, it looks amazing!).

Windows Media Player supports RTSP, but doesn’t come with an H.264 codec (even in Windows 7!!!! BOOO!!!!). I have yet to get the RTSP stream to work on Windows Media Player. The mobile player doesn’t support RTSP at all, just MMS and HTTP (but not the same HTTP as Apple! Grr!), and with the 9.5 generation of Windows Media Services (2008), MMS has gone away in favor of HTTP (which Microsoft calls Smooth Streaming, also not supported on WiMo).

The Palm Pre is supposedly able to do RTSP and H.264, but I’m waiting to hear back from one of our pre-wielding pastors to see if this is actually the case.

Thanks to Daryl Hunter at lifechurch.tv for letting me know that it works on his HTC Hero (Android 1.5). It seems that on Android you can’t manually enter an RTSP URL into the browser bar, but a web link or tinyurl redirect that goes to an RTSP URL does work.

Meanwhile, VLC player will play just about anything you throw at it, including the RTMP flash stream. Pity it’s not available in a mobile version.

So, as it stands now, in order to deliver a mobile experience to as many people as possible, I’m still going to need to run a separate Windows Media server for our Windows Mobile clients, But everyone else should be able to pull from the “iPhone” stream (which I’m probably going to need to rename), as long as the device supports H.264/AAC and RTSP.

If you have a mobile device and want to see if yours can handle an H.264/AAC RTSP stream, Try the feed here. Feed is generally available on Sundays from 9:30 am to 1 pm, and 4 pm to 7 pm Central time. It’s also live right now for testing (I’ll remove this message when I shut it down).

Longtail recently released JW Player 5.0, but it had a bug that prevented it from being used with a Wowza load balance setup. It would catch the redirect and show the first few frames and then start buffering without end.

I just got the new 764 build and am happy to report that it works quite nicely now.

On our live stream page, we have a nifty little javascript counter that lets you know when the next service is.

Leo noticed today that in Internet Explorer, it’s counting down properly, while in Firefox, it’s saying the event is already happening. On a hunch, we changed the target date to 12/31, and it started working properly again.

So, IE’s Javascript is smart enough to figure out that on December 29, the target date of January 3 is likely to be the one next week. Firefox is clinging to the past and assuming that I really meant the January 3 that happened 51 weeks ago.

How is it that the same script can be interpreted so differently within the same language on two different browser platforms? This stuff is supposed to be standard!

Earlier this week, I got an e-mail from Amazon Web Services, with some new goodies being announced.

The first was new pricing for Wowza. While the cost of each instance-hour is going up slightly, the cost of bandwidth is dropping about 15%. This makes me happy.

The second was that Wowza has released version 2. I’ve been working with preview versions of this since last summer with our iPhone streaming. They’ve made some very cool improvements to the product.

Lastly was an item that intrigued me. Amazon has, via Flash Media Server, added RTMP streaming capability to Cloudfront, Amazon’s cloud answer to CDN. Now instead of being able to widely distribute files, you have the capability of setting up a distribution that provides RTMP streaming of any supported video file in an S3 bucket. If you use S3 for on-demand video content, this is big for you. No longer do your viewers have to download the entire file (and run up the bandwidth meter in the process). They can now skip directly to the points they need and only use the bandwidth for stuff they actually watch.

This is potentially very good news for services that serve on-demand content from S3 (such as blip.tv)

It’s not so good news for the folks at Wowza, because I no longer have to spin up a Wowza instance to serve content stored in S3. Luckily for Wowza, the Cloudfront streaming doesn’t do live video.

It’ll be interesting to see how this plays out. As it currently stands, we could replace blip.tv with this functionality, but for the small cost, we get a whole lot of value from Blip.

For the last couple of years, we’ve used LED Christmas lights in our sanctuary. Considering how many we have (hundreds), the electricity savings are probably non-trivial.

All our LED strings in the sanctuary are plugged into either a stage dimmer or, where a dimmer port was unavailable, an Elation UniBar hooked into an RC4 Magic wireless DMX receiver (with the transmitter wired into DMX up in the catwalks). This allows us to control the Christmas lights along with the rest of the theatrical lighting via the Hog. It’s a very nice setup.

The other day, when Frank was running the stream, he saw the Christmas lights were fading in and out in sequence, and called up to the Penalty Box (the plexiglass-wrapped area at the back of house where the lighting operator and worship producer sit) and asked them to quit playing with the lights. As it turns out, they weren’t and the lights were all on. Mysteriously, they were fading in and out in sequence on the wide shot camera. When we looked at them on one of the other remote cameras, everything looked normal.

Then it hit me. I went to the remote control on the wide camera and cranked down the shutter speed, and lo, the lights gradually came together until they were all on. This is what it looked like:

Get the Flash Player to see this player.

Most stage dimmers operate by switching the AC cycle on and off via pulse width modulation. LEDs then only show one half of the AC sine wave, making them strobe, effectively reproducing the square-wave pulses that are modulating the dimmers. What We were seeing on the cameras was a beat frequency of the camera’s shutter speed and the strobing of the LEDs. You don’t see this on incandescent lighting because of the thermal persistence of the filaments. But why were the lights cycling at different times? Each one was connected to a different dimmer circuit, and those circuits are spread among the three AC phases coming into the dimmer room (which has a monster 2000-amp breaker).

So, if you’re shooting video of anything that has LED lights in it, make sure your shutter speed is at 1/60, or the lights are going to start acting strangely.

Since there’s been a flurry of interest lately on my series of posts on live church streaming, here’s an index to the entire set of posts:

• Live Streaming On a Budget (Part 1) – Genesis
• Live Streaming On a Budget (Part 2) – Encoding
• Live Streaming On a Budget (Part 3) – The Process
• Live Streaming on a Budget (Part 4) – How it works
• Live Streaming on a Budget (Part 5) – Automating EC2
• Live Streaming on a Budget (Intermission) Speedlinks!
• Live Streaming on a Budget (Part eleventy) – Metrics
• Live Streaming on a budget: Wowza Load Balancing (Part … um.. how many is it now?)
• Live Streaming on a Budget (Part eleventy point one) – Metrics Revisited

• MIDI Control Surfaces
• MIDI Surfaces: Korg NanoKONTROL
• MIDI Surfaces: Behringer BCF2000

For the benefit of the less technically inclined, I’ve moved scripts and code to their own section and updated posts that contained the code with links to the appropriate page.

Back in July, I made a post about metrics and a cheesy VB Script that got the job done, but wasn’t particularly elegant. I’ve since improved on this due to load balancing (I posted about that in September).

I’ve since then learned a bunch about RRDTool, and have put together a script that pulls the XML data, groks it, and then populates an RRD. The net result is that I get a graph like this:

This graph gives me the following information: The iPhone stream count (with a 10:1 vertical exaggeration), the Flash stream count, and the total viewer count, which is the sum of Flash and iPhone streams, multiplied by a factor of 1.7 (which we’ve found reasonably reflects how many actual people are watching, versus streams. Then the vertical red line shows the time the peak occurred, and the horizontal line shows the level of that peak. The actual peak numbers are listed on the bottom. The RRD and the graph are set up to take into account Windows streams, just as soon as I find a good way to pull that data from WMI via perl.

The general operation is as follows: There’s a script that’s started in a cron job 10 minutes after the servers are spun up, and it polls the origin server every 10 seconds for its counts and populates the RRD. There’s another cron job that runs every minute to generate a current graph, which is then displayed on a page with some javascript that refreshes the image in realtime. Then, at 12:15 and 6:30, there’s another cron job that takes a snapshot of the previous two hours, puts it into a web-accessible directory, and appends an HTML file with a link for easy access later.

All of the metrics scripts, automation scripts, and graphing tools live on a linux virtual machine that runs on our central campus.

This is big improvement over dumping a vbscript into a CSV and then graphing manually with Excel. This happens automatically, in real time, without me being there.

I ran up to Musician’s Friend this afternoon with C so that my wife could have some peace and quiet to work on seminary and candidacy stuff, and picked up a Behringer BCF2000. It’s so very handy to have the MF outlet locally (their distribution warehouse is up near Liberty, MO), and it’s unfortunate that they’re closing the outlet center at the end of the year.

Unboxing

The BCF2000 is a substantial unit with some heft. The box contains the unit, a 6′ USB cable, 6′ power cord (no lumps or wall warts!) and a documentation pack containing a set of manuals in a number of languages, a catalog, and a sticker. The controller is roughly the size and weight of one of their 8-channel baby mixers. Unit is reasonably well  built and heavy enough that it’s not going to unintentionally wander off the desk

Using it

The documentation is pretty straightforward, considering the plethora of operating modes this device provides for routing MIDI signals. While this unit isn’t nearly as easy to program as the Korg Nano (which uses a GUI that writes programming changes to the unit), it doesn’t rely on any external software to do its thing. The 8 rotary encoders at the top are all you need, and once you get used to it, it’s pretty simple. ETA: Behringer does provide a Java app that lets you do visal programming from the desktop. Very cool.

The B-Control also has the technical yumminess of motorized faders and presets, which make scene changes easy.

Speaking of scenes, one of the big downsides to the unit is that 8 faders is all you get. The only controls that have scene capability are the rotary encoders at the top, which can have up to four scenes (helpful for EQ settings), but no such luck on the faders. The Korg Nano would do multiple fader scenes quite easily. On the other hand, you can gang a bunch of these together through standard MIDI connections.

ETA: I stand corrected. The presets on the BCF are not just for fader positions, but for programming as well. There are several of these.

Random cool tool: MIDI Sniffer – allows you to see what’s coming across the wire.

Tomorrow, we’ll see how well it plays with the VT5 machine.

I got the OK from Clif to get the VT5 MIDI interface from Dhomas, and a control surface. The first one to try, simply by virtue of its ready availability at the local Guitar Center was the Korg NanoKONTROL.

This is a USB MIDI device in a plastic shell that’s meant to look suspiciously like one of its parents was a white MacBook. The device offers 9 sets of a fader, a knob, and two lighted buttons, as well as a 6-button set of transport controls and a scene selection button that lets you cycle through 4 different scene presets. It definitely doesn’t have the build quality of the Mac. For sixty bucks, you can’t expect much, though. Faders, knobs and buttons feel cheap. No software is included with the device, with Korg directing customers to their website to download a driver (optional, it works with the standard Windows USB MIDI driver, but the Korg driver offers some additional functionality. For the people who actually use MIDI for, you know, MUSIC, you’ll be happy to know that this device has two siblings, one with a set of pads, and the other is a 2-octave keyboard – all three are available in black, if you don’t like the Apple Fanboy shade of white.

Programming the unit requires Korg’s software, the Korg Kontrol Editor. It presents a UI that is more than a little reminiscent of a mac (this is aimed at music people, after all) that lets you set the parameters for each control on the unit. As of this post, the software is in version 1.0, and is only able to send CC and MMC commands, and there’s no option for any PC commands. Given that several of the items I want to control on the VT5 require PC commands to change, I find this to be a major shortcoming. Buttons can be set to toggle on/off or be momentary, with attack/decay controls along with what values are represented by the on and off states of the buttons. Similarly, the fader settings allow you to define values for top and bottom, allowing you to reverse the operation of the faders.

On the VT5 side, I downloaded the demo version of MIDI-VT, which allows only the control of the output faders on the VT5’s software audio mixer, but I was able to configure the NanoKontrol unit with very little difficulty, and controls on-screen are very responsive to the fader inputs on the external device. It’s considerably easier than using the mouse. Unfortunately, MIDI-VT doesn’t currently support MMC commands for DDR transport operation.

I contacted Korg about the PC issue, and they responded “As a product that is only designed to be a MIDI controller, it wouldn’t be used nor is it intended for system control that would ordinarily be handled via mouse and keyboard”. Sorry, Korg, that’s not gonna cut it. I want to use this precisely to AVOID using mouse/keyboard controls. Guitar Center, you can have it back.

Overall, this is a great inexpensive solution for an audio mixer control surface in VT5, but Korg’s lack of support for PC commands on the unit severely limits its usefulness for anything beyond the audio mixer.