streamNerd

Go big, or go home!

Ian Beyer — Sun, 22 Jan 2012 04:03:19 +0000

I’m currently working on setting up Wowza on an EC2 “Cluster Compute Quadruple Extra Large” instance (or as I’ve heard it called, the “super-duper-quadruple”, which sounds like something I’d get at Five Guys). There’s no pre-built AMI for this one, so you have to use a stock Linux image (I use the standard Amazon one) and install Wowza with a subscription license, and do the tuning yourself. But the payoff is this: for $1.30 an hour, you get a streaming server capable of delivering 10Gbps of data. On a 750Kbps stream, that’s over 13,000 concurrent clients. This for about the same cost as nine or ten m1.small instances which can deliver an aggregate of about 1.5Gbps. On a reserved instance, you can get this down to just under 75 cents an hour.

In addition to Ludicrous Speed on the network I/O, this instance comes with 8 multithreaded Xeon 5570 cores (at 2.97GHz), 23 GB of RAM, and 1.7TB of local storage. (a quick speed test downloaded a half-gigabyte file in about four seconds, limited by the gigabit interface at the remote server). This is roughly equivalent to a moderately configured Dell R710. There’s also a GPU-enabled version of this that adds a pair of nVidia Tesla GPU cores.

If that’s not enough, you can go bigger, with 16 cores, 60GB of memory, and 3.5TB, Recently, someone clustered just over a thousand of these instances into the 42nd largest supercomputer in the world.

As of right now, these monster instances are only available in the us-east-1 zone.

PHP Direct Upload to S3

Ian Beyer — Thu, 19 Jan 2012 16:12:05 +0000

Here’s an easy way to upload directly to S3 from an HTML form. No Java, no Flash. Thanks to @RaamDev for his handy little bit of code for doing HMAC signatures. Based on the documentation from Amazon. Makes use of the AWS SDK for PHP to list the buckets.

Screen shots:

Choosing a bucket

File Selection

After selecting the file

Upload Confirmation

code:




Upload a file to S3



S3 File Upload

 array(
                'key' => $accesskey,
                'secret' => $secretkey,
                'certificate_authority' => false
        ),
        '@default' => $name
));

$s3 = new AmazonS3();

if(!isset($_GET['bucket'])) {
        echo "Please select a bucket:\n";
        $buckets=$s3->get_bucket_list();
        echo '';
        foreach ($buckets as $b) { echo ''.$b."
\n"; }
        echo '
'."\n";

        } else {
        $bucket = $_GET['bucket'];

        $policy='
{"expiration": "2012-01-31T00:00:00Z",
  "conditions": [
    {"bucket": "'.$bucket.'"},
    ["starts-with", "$key", ""],
    {"acl": "public-read"},
    {"success_action_redirect": "http://mt.cdnbuilder.com/s3uploader/success.php"},
    ["starts-with", "$Content-Type", ""]
  ]
}';

/*
 * Calculate HMAC-SHA1 according to RFC2104
 * See http://www.faqs.org/rfcs/rfc2104.html
 */
function hmacsha1($key,$data) {
    $blocksize=64;
    $hashfunc='sha1';
    if (strlen($key)>$blocksize)
        $key=pack('H*', $hashfunc($key));
    $key=str_pad($key,$blocksize,chr(0x00));
    $ipad=str_repeat(chr(0x36),$blocksize);
    $opad=str_repeat(chr(0x5c),$blocksize);
    $hmac = pack(
                'H*',$hashfunc(
                    ($key^$opad).pack(
                        'H*',$hashfunc(
                            ($key^$ipad).$data

                        )
                    )
                )
            );
    return bin2hex($hmac);
}

/*
 * Used to encode a field for Amazon Auth
 * (taken from the Amazon S3 PHP example library)
 */
function hex2b64($str)
{
    $raw = '';
    for ($i=0; $i < strlen($str); $i+=2)
    {
        $raw .= chr(hexdec(substr($str, $i, 2)));
    }
    return base64_encode($raw);
}

/* Create the Amazon S3 Policy that needs to be signed */

/*
 * Base64 encode the Policy Document and then
 * create HMAC SHA-1 signature of the base64 encoded policy
 * using the secret key. Finally, encode it for Amazon Authentication.
 */
$base64_policy = base64_encode($policy);
$signature = hex2b64(hmacsha1($secretkey, $base64_policy));

?>
    
      
      
      
      
      
      
      
      

      
      

      
    
        Use a different bucket

success.php:



Upload Success!



Upload Success!

Successfully uploaded ".$_GET['key']." to bucket ".$_GET['bucket'].".\n";
$url = "https://".$_GET['bucket'].".s3.amazonaws.com/".$_GET['key'];
echo 'Direct URL to the file is '.$url."\n";

include '../footer.php';
?>

New Wowza 3 Startup Packages

Ian Beyer — Mon, 16 Jan 2012 17:43:49 +0000

Wowza has just updated its repository of startup packages to include pacakges specific to version 3. Get them here:

http://wowzamediasystems.s3.amazonaws.com/packagelist.html

Wowza Stats – Single Server

Ian Beyer — Fri, 13 Jan 2012 19:24:53 +0000

Here’s the code for gathering Wowza Stats from a single server. This assumes that you’ve enabled the stats on Wowza (serverinfo, connectioncounts in VHost.xml HTTPProviders), with authentication turned off). You’ll also get better detailed stats if you use Andrew Kennedy’s CasterStats module for the serverinfo.

                                        
                                                com.wowza.wms.plugin.test.HTTPServerInfo
                                                serverinfo*
                                                none
                                        
                                        
                                                com.wowza.wms.http.HTTPConnectionInfo
                                                connectioninfo*
                                                none
                                        
                                        
                                                com.wowza.wms.http.HTTPConnectionCountsXML
                                                connectioncounts*
                                                none

PHP code:



 
  Single-Server Metrics
 
 
  Single-Server Stats Gathering
  
   Server Address:

   
   Admin Port

   
   Data to display: 

   Detailed Count

   Load Graph

   Load in Mbps

   Simple Count

   Geographical Count

   Stream Count

   

   
  
 


Server Load"; }

	$filter = array(
        	"Name" => 'ip-address',
        	"Value" => $repeater
        	);
	$opts = array (
        	"Filter" => $filter
        	);

	if($load) { // compute load per server

                $repeaterurl = "http://$repeater:$adminport/connectioncounts";
                $repeaterxml = simplexml_load_file($repeaterurl);

                foreach ($repeaterxml->VHost as $vhost) {
                        foreach ($vhost->Application as $app) { // parse bandwidth usage by app in case we want it.
				$appname = (string)$app->Name;
				if ($app->Status == "Loaded") {
					$inbytes = (string)$app->MessagesInBytesRate;
					$outbytes = (string)$app->MessagesOutBytesRate;
					$bwin[$appname] = $inbytes / 131072; // Convert Bps to Mbps
					$bwout[$appname] = $outbytes / 131027; // Convert Bps to Mbps
				}
			} // End App Loop
		} // End VHost Loop
		$server_bwin = (string)$repeaterxml->MessagesInBytesRate / 131072; // Convert Bps to Mbps
		$server_bwout = (string)$repeaterxml->MessagesOutBytesRate / 131072; // Convert Bps to Mbps
		$server_conns = $repeaterxml->ConnectionsCurrent;

		if ($graph_load) {
			$server_load = $server_bwout / 1024 * 100; // convert to percentage

			$load_string = sprintf("%.2f",$server_load);
			$bwout_string = sprintf("%.3f",$server_bwout);
			$load_round = round($server_load);

			switch ($server_load) {
				case ($server_load > 99): $color = "black"; $heavy = true; break;
				case ($server_load >= 85): $color = "red"; $heavy = true; break;
				case ($server_load >= 75): $color = "yellow"; break;
				default: $color = "green";
				}
				$info = ''.$repeater.': '. $load_string ."% with ". $server_conns . " connections (". $bwout_string ." Mbps out)";
				print "" . $info . ""; 

			?>
			
				
			
			VHost as $vhost) {
			foreach ($vhost->Application as $app) {
				foreach ($app->ApplicationInstance as $instance) {
					foreach ($instance->Client as $client) {
						// Skip repeaters and encoders - FMS is already filtered out by the HTTPProvider
						$skip = false; // Init variable
						if (preg_match('(Repeater)',$client->Referrer)) { $skip = true; }
						if (preg_match('(Kulabyte)',$client->Referrer)) { $skip = true; }

						if (!$skip) {
							$protocol = (string)$client->Protocol;
							$proto[$protocol]++;
							$streamnames = (string)$client->StreamNames;
							$streamarray=explode("/",$streamnames);
							$str_array_len=sizeof($streamarray);

							$str_end_index = $str_array_len -1;
							$strname = $streamarray[$str_end_index];

							if ($streamarray[0] == "amazons3") {
								$vod[$strname]++;
								} else {
								$live[$strname]++;
								}
							$client_list[] = (string)$client->IpAddress;

							if ($detailed) {

								if ($client->UserAgent) // HTML clients will have a useragent defined
									{
										echo $client->IpAddress.": ".$client->UserAgent ."
\n";
										if (preg_match('(Roku)',$client->UserAgent)) { $agent['Roku']++; }
										if (preg_match('(iPhone)',$client->UserAgent)) { $agent['iPhone']++; }
										if (preg_match('(iPad)',$client->UserAgent)) { $agent['iPad']++; }
										if (preg_match('(iPod)',$client->UserAgent)) { $agent['iPod']++; }
										if (preg_match('(stagefright)',$client->UserAgent)) { $agent['Android']++; }

									} 

								if ($client->FlashVersion) // Flash clients will have a different string.
									{
										echo $client->IpAddress.": Flash version " . $client->FlashVersion ."
\n";
										$agent['Flash']++;
										}
								}	//Client Details
							} // Detailed Loop
					} //Client Loop
				} //Instance Loop
			} // App Loop
		} //VHost Loop

	} // Count Loop

if ($geo_count) {

	print "Client Connections\n";
	echo "Last Refresh: ";
	echo $date = date("Y-m-d H:i:s");
	echo "\n";

	foreach($client_list as $client_ip) {
		$geo_record = geoip_record_by_name($client_ip);
		$cn = $geo_record['country_name'];
		$cc = strtolower($geo_record['country_code']);
		if ($cc == "us") {
			$sn = $geo_record['region'];
			$state_total[$sn]++;
			}

		$country_name[$cc]=$cn;
		$country_total[$cc]++;
	} //End Client iteration
	arsort($country_total);
	print ''."\n";
	print 'Countries'."\n";
	foreach (array_keys($country_total) as $ccode) {
		print ' ';
		print "$country_name[$ccode] : ".$country_total[$ccode]."
\n";
	} // End Country iteration
	print '';

	arsort($state_total);
	print ''."\n";
	print 'States'."\n";
	foreach (array_keys($state_total) as $state) {
		print "$state : ".$state_total[$state]."
\n";
	} // End State iteration
	print '';

} // End Geo Loop

if ($detailed) {
	$agent['iOS'] = $agent['iPhone'] + $agent ['iPad'] + $agent['iPod'];
	echo "
\n";
	echo "Total HLS Clients: ". $proto['HLS']."
\n";
	echo "Total Roku clients: ". $agent['Roku']."
\n";
	echo "Total iOS clients: ". $agent['iOS'] ." (".$agent['iPhone']." iPhone,". $agent['iPad']." iPad,". $agent['iPod']." iPod)
\n";
	echo "Total RTMP clients: ". $proto['RTMP']." (". $agent['Flash'] ." Flash)
\n";
} // Detailed Loop

if ($simple_load) { printf("%.3f", $server_bwout); }

if ($simple_count) { print $server_conns; }

if ($stream_count) {
	print "Stream Connections\n";
	arsort($live);
	arsort($vod);
	print "Live\n";
        foreach (array_keys($live) as $lstr) { print strtoupper($lstr)." : ".$live[$lstr]."
\n"; }
	print "
On-Demand\n";
        foreach (array_keys($vod) as $vstr) { print strtoupper($vstr)." : ".$vod[$vstr]."
\n"; }
	print "";
}

}

?>

Switching Live Streams

Ian Beyer — Tue, 10 Jan 2012 04:00:21 +0000

Continuing on the rack theme mentioned yesterday, I got to wondering about a stream monitor that could be switched to any of a number of live streams, without reloading the page. Fortunately, JW Player makes this easy. I threw together a player embedded inside some CSS and then added a button panel. Each button is a DIV with an onclick() action that calls jwplayer().load(). While it’s well known that you can use this to switch files simply by passing the filename to the file flashvar, we need to also pass the streamer value. Fortunately, the load() method has the ability to pass on not only files as a string value, but also objects, which are nothing more than an array of flashvars (it can also take playlist items, but that’s beyond the scope of this post). So, all you need to do in order to switch streams with JW Player is call the following JavaScript method:

jwplayer().load({file: 'streamname', streamer: 'rtmp://w.streampunk.tv/live'})

The result I got was this:Clicking on each of the buttons in the bar below the screen will switch the stream. A live example can be found here. If you want to see what else the JWPlayer API can do, head on over to the API docs. FlowPlayer appears to have something similar in the clip.update() method (flowplayer API documentation), but I haven’t tested it.

Page code:






	JW Player Switching

	



	
	
	
	 
	
	Player...
	
	
	
	
	
	
		
		RezOnline
		
		
		Commons
		
		
		StreamClass
		
		
		AirCam

Stupid CSS Tricks: Wowza Load Graphs

Ian Beyer — Mon, 09 Jan 2012 03:48:58 +0000

I’ve been experimenting with ways to generate load graphs for Wowza. The best way for doing bar graphs on a webpage is to go all crazy with CSS. It’s really well suited to doing this. Here’s a PHP script that will query a Wowza origin server for its repeaters, and then polls the repeaters for their load stats. This also provides buttons for launching/terminating repeaters, and visually representing them as a rack full of servers:

You can get really crazy with this. For example, here’s a farm of 14 servers:

How do I do this?

The basic structure of my virtual rack is not unlike that of a real rack: The Rack Unit. In this case, it’s a DIV 60 pixels high and 700 pixels wide. It then contains a rack space which is 600 pixels wide, leaving me some room on the end to put something if I want to. The rack space div then uses the server image as a background. Inside that I have the display, power button, Wowza button (all three of these are simple absolute position DIV elements – the Wowza button is simply an empty DIV with rounded corners set to make it an actual circle, and then a TITLE attribute containing the build info of that particular server), and graph, which is the larger container div (with opacity set to be slightly transparent) that then has the smaller graph div, with the width set to the calculated load percentage.

Now, the code. Note that this requires the AWS SDK for PHP to be installed on your box (and apparently a prerequisite for that is a bowl of alphabet soup) Bear in mind that querying each server is going to take 2-3 seconds, so this doesn’t scale well (the 14-server graph made use of a variant of this script that gets edge data via SQL, making it much faster). I’d love to know if it’s possible to fork each of those getLoad() processes off so they can run in parallel.

syntax is:

loadrack.php?origin=1.2.3.4

loadrack.php:

 99): $color = "#000000"; break;
	case ($total_load >= 85): $color = "#FF0000"; break;
	case ($total_load >= 75): $color = "#FFFF00"; break;
	default: $color = "#00FF00";
	}
		$label = 'Origin | '. $origin_ip . "";

	?>
		
		
			
				
			
			
				 Connections ( Edges)

				 Mbps  ( %)
			
			
			












			
			

		
	
 99): $color = "#000000"; $heavy = true; break;
			case ($server_load >= 85): $color = "#FF0000"; $heavy = true; break;
			case ($server_load >= 75): $color = "#FFFF00"; break;
			default: $color = "#00FF00";
			}
		$label = 'Edge | '. $repeater . "";
		$lcd = $conn[$repeater]." Connections 
$bwout_string Mbps ($load_string%)";
		?>
		
		
			
				
				
			
			
			
			
				';
					echo '';
					echo '';
					echo '';
					}
					else
					{
					echo '';
					echo '';
					echo '';
					}
				?>
			

			"
				style="position: absolute;
					left: 23px;
					top: 24px;
					width: 20px;
					height: 20px;
					border-radius: 10px;
					border: none;
			">
		
	
Last Refresh: ";
			echo $date = date("Y-m-d H:i:s");
			echo "\n";

?>

aws-init.php:

functions.php:

function getRepeaters($origin_ip) {

	$addr = array();
	$lb = "http://$origin_ip:1935/loadbalancer?serverInfoXML";
	$xml = simplexml_load_file($lb);

	if ($xml === FALSE) {
		return("");
	}

	$server = $xml->LoadBalancerServer;

	// Get IP addresses of active repeaters and put them in an array.
	foreach ($server as $rep) {
		$ip = (string)$rep->redirect;
		if ($rep->status == "RUNNING" or $rep->status =="PAUSED") {
			$addr[]=$ip;
		} // End Running If
	} // end Server foreach
	return($addr);
} // End Function

function getLoad($repeater) {

        $repurl = "http://$repeater:1935/connectioncounts.xml";
        $repxml = simplexml_load_file($repurl);
        $bwin = (string)$repxml->MessagesInBytesRate / 131072; // Convert Bps to Mbps
        $bwout = (string)$repxml->MessagesOutBytesRate / 131072; // Convert Bps to Mbps
        $conns = (string)$repxml->ConnectionsCurrent+0;

	$data = array ($bwin, $bwout, $conns);

	return($data);

} // End Function

function getVersion($server) {
        $repurl = "http://$server:1935/";
        $repxml = simplexml_load_file($repurl);
	$ver = (string)$repxml->body;

	return($ver);

} // End GetVersion Function
?>

capacity_vars.php:

Wowza Architecture Diagrams

Ian Beyer — Mon, 02 Jan 2012 01:35:20 +0000

I’ve posted a couple of diagrams of Wowza’s architecture relating to repeaters and load balancing.

There’s also some additional information in the Wowza Media Server 3 Overview.

Site Migration Complete

Site Administrator — Thu, 29 Dec 2011 18:55:05 +0000

Got all the pages moved over and the images all fixed, and took care of a number of broken links. If you see any problems, let me know.

Site Migration

Site Administrator — Thu, 29 Dec 2011 05:35:10 +0000

In the process of moving the site to a new hosting provider – bear with me. Still working on moving the images, they’ll start cropping up over the next few days.

Wowza Media Server V3 for Amazon EC2

Ian Beyer — Sun, 06 Nov 2011 23:56:14 +0000

Wowza V3 pre-built AMIs are now available – The devpay licensing remains, as does the pricing. The new AMI listing can be found at the Wowza V3 for EC2 page. Wowza has also added pre-built AMIs for subscription licenses, which are priced at standard instance rates. The caveat is that on devpay, the premium add-on modules won’t be available – if all you’re doing is what you were doing on V2, that won’t change anything for you.

The license key instances can also be used as a basis for your own custom images. License key can either be manually changed or included in your startup packages.