Thoughts on technology and media
Ian Beyer
(23 comments, 134 posts)
This user hasn't shared any profile information
Home page: http://ianbeyer.com
Posts by Ian Beyer
Browser-Aware Player Code: Episode V, IE Strikes Back
0Not so long ago, I updated my browser-aware player code to check for the presence of a stream. Recently, it’s come to light that Internet Explorer 9 doesn’t play nice with this particular snippet, because in IE9, the Javascript engine is rather brain-damaged when it comes to cross-site requests. In order to deal with this properly, we must alter the way we query the server for the presence of a stream:
console.log("Starting stream Check");
if (is_ie9) {
console.log ("Using XDR because IE9 is stupid");
streamcheckXDR();
setInterval(function(){streamcheckXDR()},10000);
}
else {
streamcheck();
setInterval(function(){streamcheck()},10000);
}
function streamcheckXDR() {
console.log("Starting XDR");
xdr = new XDomainRequest();
if(xdr) {
xdr.onerror = function(){ console.log("XDR Error"); };
xdr.onload = function(){ startPlayer(xdr.responseText,"XDR"); };
url = "http://"+streamer+":8086/streamcheck?stream="+stream;
xdr.open("get",url);
xdr.send();
} else {
console.log("Failed to create XDR object");
}
}
function startPlayer(result,mode){
// for some inexplicable reason, running "Boolean" on the XDR output doesn't work
// so we have to call the function and tell it if we're dealing with XDR data or AJAX data.
if(mode == "XDR") {
if (result === "true") { curstatus = true;}
if (result === "false") { curstatus = false;}
} else {
curstatus = Boolean(result);
}
//console.log("Result: "+result);
//console.log("Previous: "+prevstatus);
//console.log("Current: "+curstatus);
if (curstatus == prevstatus) {
//console.log("No Change");
} else {
if (curstatus) {
if (is_iphone || is_ipad || is_ipod) { iOSPlayer("videoframe",plwd,plht,server,stream);}
else if (is_blackberry) { rtspPlayer("videoframe",plwd,plht,server,stream+'_+240p');}
else { flashPlayer("videoframe",plwd,plht,server,stream); }
console.log("Changed from false to true");
} else {
var vframe=document.getElementById("videoframe")
if (is_iphone || is_ipad || is_ipod || is_blackberry) {
} else {
jwplayer("videoframe").remove();
}
vframe.innerHTML = '<IMG SRC="image.png" WIDTH="'+plwd+'" HEIGHT="'+plht+'">';
console.log("Changed from true to false");
}
}
prevstatus = curstatus;
}
function streamcheck() {
console.log("Starting AJAX");
$.ajax({
dataType: "json",
contentType: "text/plain",
type: "GET",
url: "http://"+streamer+":8086/streamcheck?stream="+stream,
error: function(XMLHttpRequest, textStatus, errorThrown)
{
console.log('AJAX Failure:'+ textStatus+':'+errorThrown);
//some stuff on failure
},
success: function(result){startPlayer(result)}
});
}
Dun & Bradstreet = Extortion Scam
0I don’t often get on a soapbox on the blog, but these guys really ticked me off today.
“D&B Credibility Corporation” is the self-appointed credit bureau of the business world. If you want to get a report on a company, you pay these guys $180+ for the report. Like any other credit bureau, they’re in the data business. That’s fine. But how they get that data is highly questionable.
Today, after sending me numerous unsolicited letters urging me to update my company file, they called me and asked to update the info, saying that there had been inquiries lately (a little googling tells me that’s usually utter BS, and they wouldn’t tell me who, so color me dubious).
Fine, I say, what do you need to know? I get a vague response and it’s starting to sound like a sales pitch. So I cut to the chase and ask these guys if they plan on charging me money for this. The guy on the other end of the line mumbles something about it being $299 but that they would “provide a tax-deductible receipt” (duh, it’s a business expense, of course it’s tax-deductible). I put the brakes on his script.
“Let me get this straight. You sell data about my company to other companies who pay you a couple hundred bucks for that information.”
“Yes, that’s correct”
“So you’re asking me to provide you with information that will improve the quality of the data you sell to someone else”
“Yes.”
“And you have the gall to charge ME for the privilege of improving your product? What’s in it for me? Do I get a cut of this?”
“If you don’t update your information, we’ll report your file as incomplete to the requesting companies, and your risk score will indicate that as well”
“Wait, you’re telling me that if I *don’t* pay up, you’re going to give me a lower score?”
*mumbles and tries to get back on his sales script, it’s not going well for him*
“Who else in the industry provides this sort of data?”
“We’re the only ones, 175 years in business, blah, blah”
“So you’re a monopoly, a self-appointed one at that, and you want to charge *me* money to improve *your* product? Sounds like extortion to me. How is this different from a mafia protection racket?”
*mumbles more sales script, implying that my business will fail if I don’t cough up the money*
“You’re making my point for me. If you want to go out and improve the quality and value of your data on your own, knock yourself out, but I sure as hell won’t be paying you for the privilege. Good day.”
If the consumer credit agencies ran this kind of scam, there would be utter hell to pay from Washington regulators. D&B is not subject to the FCRA. They may be a legitimate company, but their data collection tactics stink to high heaven.
If your company is requesting a D&B report on someone else, be aware that the data in it is not impartial. You’re probably getting taken for a ride.
Wowza EC2 Capacity Update
32 months ago
It’s been a while since Wowza has updated their EC2 performance numbers (they date back to about 2009), and both Amazon and Wowza have made great improvements to their products. Since I have access to a high-capacity system outside of Amazon’s cloud, I am able to use Wowza’s load test tool on a variety of instance sizes to see how they perform.
The test methodology was as follows:
- Start up a Wowza instance on EC2 with no startup packages (us-east)
- Install the server-side piece of Willow (from Solid Thinking Interactive)
- Configure a 1Mbps stream in Wirecast
- Monitor the stream in JWPlayer 5 with the Quality Monitor Plugin
- Configure the Wowza Load Test Tool on one of my Wowza Hotrods located at Softlayer’s Washington DC datacenter
- Server is 14 hops/2ms from us-east-1
- Increase the load until:
- the measured bandwidth on JW player drops below stream bandwidth
- frame drops get frequent
- Bandwidth levels out on the Willow Graphs while connection count increases
- Let it run in that condition for a few minutes
In Willow, it basically looked like this (this was from the m1.small test). You can see ramping up to 100, 150, 200, 250, 275, and 300 streams. The last 3 look very similar because the server maxed out at 250 Mbps. (Yes, the graph says MBytes, that was a bug in Willow which Graeme fixed as soon as I told him about it)
Meanwhile, this is what happens on the server.. the CPU has maxed out.
So that’s the basic methodology. Here are the results:
Size | Capacity (Mb/sec) | Capacity (GB/Hr) | Instance Cost | Cents/Mb |
|---|---|---|---|---|
| m1.small | 250 | 110 | 0.15 | .060 |
| m1.medium | 500 | 220 | 0.26 | .052 |
| c1.medium | 950 | 417 | 0.26 | .027 |
| m1.large | 700 | 308 | 0.46 | .066 |
| m1.xlarge | 950 | 417 | 0.89 | .094 |
| m2.xlarge | 300 | 132 | 0.62 | .207 |
| c1.xlarge | 950 | 417 | 0.89 | .094 |
| m2.2xlarge | 950 | 417 | 1.21 | .127 |
There are a couple of things to note here. Naturally, if you’re not expecting a huge audience, stick to m1.small. But the best bang for the buck is the c1.medium (High-CPU Medium), which is a relatively new instance type, which gives you 4x the performance of a m1.small at less than 2x the price. The big surprise here was the m2.xlarge. It performs only marginally better than an m1.small at 4x the price.
All the instances that show 950 are effectively giving you the full benefit of the gigabit connection on that server and maxed out the interface long before the CPU maxes out. In the case of the c1.xlarge, there’s lots of CPU to spare for things like transcoding and such if you’re using a BYOL image. If you want to go faster, you’ll need to roll your own Cluster Quad or do a load-balanced set.
Disclaimers: Your mileage may vary, these are just guidelines, although I think they’re pretty close. I have not tested this anywhere but us-east-1, so if you’re using one of amazon’s other datacenters, you may get different results. I hope to test the other zones soon and see how the results compare.
Wowza/EC2 Q&A Session
2I’ll admit, the nerd meter on here has been running past redline for a while, so I’m going to step back and answer some questions that have come in via e-mail or twitter from those of you still trying to get started with Wowza streaming.
Q: How do I find a good Wowza EC2 server to use, what should I be using?
A: I understand your confusion. There are a lot of Wowza AMIs out there on the EC2 stack, and that’s just the official ones! When starting an instance from the AWS console, simply searching for “wowza” under the community AMIs is a bit of a daunting process. Just in us-east-1, you get 68 results:
As a general rule, the most recent version available is the one you want to use. Wowza keeps a list of the most recent AMIs on their support site, and unless you have bought a license (Plug: I’m a reseller, support your streamnerd!), you’ll want to use the DevPay instances which require a $5 monthly subscription from Amazon.
If you need to run an older version, Wowza’s support team can give you the AMI ID you need, as they continue to make older versions available (which is why you get such a huge list), although they’re not maintained once a newer AMI has been released. With a few exceptions, startup packages designed for one version will generally work on another version.
Where it gets complicated is if you’re dealing with BYO Licenses and Marketplace instances. That’s probably a topic for another post.
Q: How do I know if I need to set up load balancing and how do I do that, in plain English?
A: Simply put, you’ll need to do load balancing any time the audience on your server exceeds the network throughput that your instance size will support. Wowza has published some numbers in the past for a few of the instance sizes, but those numbers are several years old, and changes to both Amazon’s infrastructure and the Wowza code base have rendered them obsolete. Since I now manage some very large Wowza servers outside the Amazon stack, I will be running some new benchmarking tests in the next few weeks. When I get the results, I’ll be posting those here as well as sharing them with the Wowza team.
The best way to make your Wowza stack scalable is to configure your single server as both an origin and an edge, and make it a load balancing army of one. That way, if you get an unexpected surge in traffic, adding more repeaters is a fairly simple task. All you need to do is keep a set of startup packages with the configuration for each type on hand, and you can start them up.
The key thing to remember about Wowza load balancing is that the load balancer and the origin/edge architecture are independent of each other. In an origin/edge configuration, you publish to one application, and another application (on either the same server or a different server) pull the stream from the origin application and then distribute it out to clients.
There are many ways to determine which edge application a client goes to, but the most common one is the Wowza load balancer module. This module consists of a “sender” that lives on edge servers, and a “listener” that lives on the load balancing server (which is usually the origin server, but doesn’t have to be). The sender keeps track of how many connections are on its server, and reports those back to the listener along with its address and a unique identifier. When queried, the load balancer will then provide the address of whichever server has the fewest connections (or an XML dump listing all the servers in the pool and their status).
Once you have the information of which server has the fewest connections, it’s simply a matter of directing an incoming connection to that edge server. The load balancer package provides an application that will redirect RTMP clients, but that doesn’t work with HTTP clients such as iOS and Android devices, so you’ll need some additional steps for them.
Q: I can check connection counts for stats, but all other posts I read about analytics is confusing. Is there a better, easier way?
One of the most challenging things about Wowza facing new users is that it lacks all the pretty graphs – but the flip side of that is that you get access to raw data. Out of the box, there’s not really anything other than the connectioncounts.xml and serverinfo.xml XML dumps. There are a number of third-party components that can massage this data into something a little more useful and “boss-friendly”. On the commercial side there’s CasterStats (Plug: I’m a reseller. Support your streamnerd!) which can do both live stats and post-analysis of log files, as well as Sawmill and AWStats which will analyze logs. There are probably a few others as well. And, naturaly, there are also a number of PHP scripts that I’ve posted on this blog before that can help with analyzing and processing the XML dumps.
One final plug: If you’re interested in a web-based control system that manages your startup packages and live stats, ask me about cdnBuilder.
That’s all for today, if you’ve got a Wowza question (EC2 or not!), drop me a line on twitter or leave me a comment below!
Supercharging a Wowza Hotrod
3
I’ve posted in the past about running Wowza on large server instances. When running Wowza on systems with a 10Gbps network interface, you run into a 5Gbps limitation inherent to the Java Virtual Machine. With Wowza tuning parameters maxing out at 12 virtual cores, most machines with a 10Gbps interface will have processing power to spare, but why let all that horsepower go to waste?
For relatively little money (under $1000/month), you can get server from 100TB.com with 12 or 16 physical cores (24 or 32 virtual) and park it in the Softlayer datacenter in Washington, DC with a 10Gbps interface, pretty much in the middle of the Internet. With 100TB/month of data transfer included in the price (and for an extra charge, they’ll take the meter off entirely), this is a great option for heavy or full-time streaming. But there’s that pesky JVM throughput limitation. It’s time to take the governor off this hotrod.
What you’ll need:
- A big honkin’ server
- A big honkin’ pipe
- Some extra IP addresses (the 100TB servers at Softlayer come with 8 usable addresses)
- Wowza with either a subscription license key or two perpetual license keys
Note that I’ve done this in Linux, I’m not quite sure how it would work in Windows with registering the second instance with the services.
Step 1
- Install Wowza and tune
- Configure it with a license
- Choose an IP address on the system
- In VHost.xml, and Server.xml replace all instances of <IpAddress>*</IpAddress> with the IP.
- Set up load balancing (both sender and listener) with redirect
- Add an origin application (type live).
- Add an edge application (type liverepeater-edge) pulling from the origin app
Step 2
- Make a duplicate of the following (I appended a -2):
- /usr/local/WowzaMediaServer-3.5.0
- /etc/init.d/WowzaMediaServer *
- /usr/bin/WowzaMediaServerd *
- Edit all references to WowzaMediaServer in the files indicated by a *
- in the cloned installation:
- change bin/setenv.sh and bin/startup.sh to point to the cloned install
- delete conf/Server.guid
- Edit conf/VHost.xml to point to a second IP address
- Edit conf/Server.xml to point to a second IP address
- Edit conf/Server.xml to bind jmx to ports 8081/8082
- Set up load balancing as a Sender (edge) pointing to load balancer IP in primary install
- Create edge application
Step 3
- Start the primary instance
- Check http://primary.ip:1935/loadbalancer?serverInfoXML to verify the primary edge is connected
- Start the secondary instance
- Check load balancer again and make sure secondary instance is connected
- Publish a stream
- Pull a stream directly from each edge to make sure the repeaters are working
- use http://primary.ip:1935/loadbalancer to get least loaded server for load-balancing non-RTMP clients
- use rtmp://primary.ip/redirect/stream to get load balanced RTMP.
Step 4
- Sit back, relax, and enjoy the fact that you now have a server that can handle 10,000 connections.
- If you need more, build up a second server just like this one, and just make both instances edge instances
Browser-aware player code, revisited again
13It’s the code snippet that just won’t go away. I’ve updated the code for some additional functionality. This version takes server, port, and stream parameters via the URL, parses them in javascript, and then queries a streamcheck HTTPProvider on the server to see if a stream by that name is currently published. If it is, it will load the player, otherwise load a message, and check periodically to see if the stream is published, and load the player if the state changes to true, and unload it if it changes to false, returning to the message. The player is designed to scale to fit whatever window it’s in, so make an IFRAME of whatever size you want the player, and you’re off and running
<IFRAME SRC="player.html?streamer=wowza.nhicdn.net&port=1935&app=live&stream=livestream" WIDTH="640" HEIGHT="360" SCROLLING="NO" />
Without further ado, here’s the code:
<body style="margin: 0px; background: black; color: white; font-family: sans-serif;">
<div id="videoframe" style="text-align: center;font-size: 14;">The video stream is currently offline. Playback will resume as soon as a stream is available.</div>
<script type="text/javascript" src="/assets/jw5/jwplayer.js"></script>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" type="text/javascript"></script>
<script type='text/javascript' src='https://www.google.com/jsapi'></script>
<SCRIPT type="text/javascript">
// Browser-aware video player for JW Player and Wowza
plwd=self.innerWidth;
plht=self.innerHeight;
// var debugwindow=document.getElementById("debug")
// debugwindow.innerHTML='<P>Dimensions: '+plwd+'x'+plht+'</P>';
var streamer=getUrlVars()["streamer"];
var app=getUrlVars()["app"];
var port=getUrlVars()["port"];
var stream=getUrlVars()["stream"];
var server=streamer+':'+port+'/'+app;
var agent=navigator.userAgent.toLowerCase();
var is_iphone = (agent.indexOf('iphone')!=-1);
var is_ipad = (agent.indexOf('ipad')!=-1);
var is_ipod = (agent.indexOf('ipod')!=-1);
var is_playstation = (agent.indexOf('playstation')!=-1);
var is_safari = (agent.indexOf('safari')!=-1);
var is_blackberry = (agent.indexOf('blackberry')!=-1);
var is_android = (agent.indexOf('android')!=-1);
var streamstatus = false;
var prevstatus = false;
var curstatus = false;
streamcheck();
setInterval(function(){streamcheck()},10000);
function streamcheck() {
$.ajax({
type: "GET",
url: "http://"+streamer+":8086/streamcheck?stream="+stream,
dataType: "json",
success: function(result){
curstatus = Boolean(result);
//if (result === "true") { curstatus = true;}
//if (result === "false") { curstatus = false;}
if (curstatus == prevstatus) {
} else {
if (curstatus) {
if (is_iphone || is_ipad || is_ipod) { iOSPlayer("videoframe",plwd,plht,server,stream);}
else if (is_blackberry) { rtspPlayer("videoframe",plwd,plht,server,stream);}
else { flashPlayer("videoframe",plwd,plht,server,stream); }
console.log("Changed from false to true");
} else {
var vframe=document.getElementById("videoframe")
if (is_iphone || is_ipad || is_ipod || is_blackberry) {
} else {
jwplayer("videoframe").remove();
}
vframe.innerHTML = 'The video stream is currently offline. Playback will resume as soon as a stream is available.';
console.log("Changed from true to false");
}
}
prevstatus = curstatus;
}
});
}
function getUrlVars() {
var vars = {};
var parts = window.location.href.replace(/[?&]+([^=&]+)=([^&]*)/gi, function(m,key,value) {
vars[key] = value;
});
return vars;
}
function iOSPlayer(container,width,height,server,stream)
{
var player=document.getElementById(container)
player.innerHTML='<VIDEO controls '+
'HEIGHT="'+height+'" '+
'WIDTH="'+width+'" '+
'title="Live Stream">'+
'<SOURCE SRC="http://'+server+'/'+stream+'/playlist.m3u8"> '+
'</video>';
}
function rtspPlayer(container,width,height,server,stream)
{
var player=document.getElementById(container)
player.innerHTML='<A HREF="rtsp://'+server+'/'+stream+'">'+
'<IMG SRC="poster-play.png" '+
'ALT="Start Mobile Video" '+
'BORDER="0" '+
'HEIGHT="'+height+'" '+
'WIDTH="'+width+'">'+
'</A>';
}
function flashPlayer(container,wide,high,server,stream)
{
jwplayer(container).setup({
height: high,
width: wide,
streamer: 'rtmp://'+server,
file: stream,
autostart: true,
stretching: 'uniform'
});
}
</SCRIPT>
</BODY>
The code for the streamcheck module is as follows:
package net.nerdherd.wms.http;
import java.io.*;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import com.wowza.wms.application.IApplicationInstance;
import com.wowza.wms.http.*;
import com.wowza.wms.logging.*;
import com.wowza.wms.vhost.*;
public class StreamCheck extends HTTProvider2Base {
public void onHTTPRequest(IVHost vhost, IHTTPRequest req, IHTTPResponse resp) {
StringBuffer report = new StringBuffer();
StringBuffer streamlist = new StringBuffer();
if (!doHTTPAuthentication(vhost, req, resp))
return;
Map<String, List<String>> params = req.getParameterMap();
String stream = "";
boolean status = false;
boolean listing = false;
if (req.getMethod().equalsIgnoreCase("post")) {
req.parseBodyForParams(true);
}
if (params.containsKey("stream"))
stream = params.get("stream").get(0);
try
{
if (vhost != null)
{
List<String> appNames = vhost.getApplicationNames();
Iterator<String> appNameIterator = appNames.iterator();
while (appNameIterator.hasNext())
{
try {
String Name = appNameIterator.next();
IApplicationInstance NowApp = vhost.getApplication(Name).getAppInstance("_definst_");
List<String> PublishedNames = NowApp.getPublishStreamNames();
Iterator<String> ThisPublished = PublishedNames.iterator();
if ( PublishedNames.size()>0 )
{
while ( ThisPublished.hasNext() )
{
try {
String NowPublished = ThisPublished.next();
if (NowPublished.equals(stream)){
status = true;
}
} catch (Exception e) {}
}
}
} catch (Exception e) {report.append(e.toString()); }
}
}
}
catch (Exception e)
{
WMSLoggerFactory.getLogger(HTTPServerVersion.class).error("StreamCheck: " + e.toString());
e.printStackTrace();
}
if (!listing) {
if (status){
report.append("true");
} else {
report.append("false");
}
}
try {
resp.setHeader("Content-Type", "text/plain");
resp.setHeader("Access-Control-Allow-Origin","*");
OutputStream out = resp.getOutputStream();
byte[] outBytes = report.toString().getBytes();
out.write(outBytes);
} catch (Exception e) {
WMSLoggerFactory.getLogger(null).error(
"MediaCasterHTTP: " + e.toString());
}
}
}
Creating a global streaming CDN with Wowza
07 months ago
I’ve posted in the past about the various components involved in doing edge/origin setups with Wowza, as well as startup packages and Route 53 DNS magic. In this post, I’ll tie the various pieces together to put together a geographically-distributed CDN using Wowza.
For the purposes of this hypothetical CDN, we’ll do it all within EC2 (although there’s no reason it has to be), with three edge servers in each availability zone.
There are two ways we can do load balancing within each zone. One is to use the Wowza load balancer, and the other is to use round-robin DNS. The latter is not as evenly balanced as the former, but should work reasonably well. If you’re using set-top devices like Roku, you’ll likely want to use DNS as redirects aren’t well supported in the Roku code.
For each zone, create the following DNS entries in Route 53. If you use your registrar’s DNS, you’ll probably want to create a domain name as many registrar DNS servers don’t deal well with third-level domains (sub.domain.tld). For the purposes of this example, I’ll use nhicdn.net, the domain I use for Nerd Herd streaming services. None of these hostnames actually exist.
We’ll start with Wowza’s load balancer module. Since the load balancer is not in any way tied to the origin server (although they frequently run on the same system), you can create a load balancer on one of the edge servers in that zone. Point the loadbalancertargets.txt on each edge server in that zone to that server, and point to the origin stream in the Application.xml configuration file. Once the load balancer is created, create a simple CNAME entry in Route 53 called lb-zone.nhicdn.net, pointing to the hostname of the load balancer server:
lb-us-east.nhicdn.net CNAME ec2-10-20-30-40.compute-1.amazonaws.com
If you opt to use DNS for load balancing, again point the Application.xml to the origin stream, and then create a weighted CNAME entry to each server in that zone, all with the same host record, each with equal weight (1 works):
lb-us-east.nhicdn.net CNAME ec2-10-20-30-40.compute-1.amazonaws.com
lb-us-east.nhicdn.net CNAME ec2-10-20-30-41.compute-1.amazonaws.com
lb-us-east.nhicdn.net CNAME ec2-10-20-30-42.compute-1.amazonaws.com
If you wish, you can also create a simple CNAME alias for each server for easier tracking – this is strictly optional:
edge-us-east1.nhicdn.net CNAME ec2-10-20-30-40.compute-1.amazonaws.com
edge-us-east2.nhicdn.net CNAME ec2-10-20-30-41.compute-1.amazonaws.com
edge-us-east3.nhicdn.net CNAME ec2-10-20-30-42.compute-1.amazonaws.com
Once you’ve done this in each zone where you want servers, Create a set of latency-based CNAME entries for each lb entry:
lb.nhicdn.net CNAME lb-us-east.nhicdn.net
lb.nhicdn.net CNAME lb-us-west.nhicdn.net
lb.nhicdn.net CNAME lb-europe.nhicdn.net
lb.nhicdn.net CNAME lb-brazil.nhicdn.net
lb.nhicdn.net CNAME lb-singapore.nhicdn.net
lb.nhicdn.net CNAME lb-japan.nhicdn.net
Once your DNS entries are in place, point your players to lb.nhicdn.net and use either the name of the edge application (if using DNS) or the redirect application (if using Wowza load balancer), and clients will go to either the edge server or load balancer for the closest region.
One other thing to consider is to set up a load balancer on your origin, and use that for gathering stats from the various servers. You can put multiple load balancers in the loadbalancertargets.txt file, so you can have each server report in to both its regional load balancer and the global one, except the global one is not being used to redirect clients, but rather only for statistics gathering. You can also put multiple load balancers in each region for redundancy and use a weighted CNAME entry.
If you would like assistance setting up a system like this for your organization, I am available for hire for consulting and/or training.
Adding EC2 instances to Route53
108 months ago
by Ian Beyer
in Cloud Computing
Today’s nifty bit of code is a startup/shutdown script for linux that allows you to add an EC2 instance to Amazon’s Route53 DNS automatically when you start up the instance, and remove it when the instance is knocked down. This script also allows you to add the instance to a weighted round-robin group.
This makes use of the very useful Python-based boto tool which is available in both Yum and Debian repositories under the package name python-boto.
Create this script in /etc/init.d and make it executable, and then add it to the requisite rcX.d directory for startup/shutdown.
#!/bin/bash
#
# /etc/rc.d/init.d/<servicename>
#
# Registers DNS with Route 53
#
# chkconfig 345 20 80
#
# Source function library.
. /etc/init.d/functions
# Best practice here is to create an IAM user/group with access to only
# Route53 and use this keypair.
export AWS_ACCESS_KEY_ID=<Access Key ID Here>
export AWS_SECRET_ACCESS_KEY=<Secret Key ID here>
# Use the domain you have configured in Route 53
export DOMAIN=ianbeyer.com
# This is the hostname of the Weighted Round Robin Group.
export RR=wrr
# Gets the current public hostname - you don't want to use an A record with
# the IP because the CNAME works in conjunction with Amazon's internal DNS
# to correctly resolve instances to their internal address
export PUB_HOSTNAME=`curl -s --fail http://169.254.169.254/latest/meta-data/public-hostname`
# Gets the Route 53 Zone ID
export ZONEID=`route53 ls | awk '($2 == "ID:"){printf "%s ",$3;getline;printf "%s\n",$3}' | grep $DOMAIN | awk '{print $1}'`
start() {
echo -n "Registering host with Route 53 : "
# This is the base name of the host you want to use. It will increase the index
# number until it finds one that's not in use, and then use that.
export HOST=amz
export HOSTINDEX=1
INUSE=`route53 get $ZONEID | grep ${HOST}${HOSTINDEX}\.${DOMAIN} | wc -l`
while [[ $INUSE > 0 ]]
do
HOSTINDEX=$((HOSTINDEX + 1))
INUSE=`route53 get $ZONEID | grep ${HOST}${HOSTINDEX}\.${DOMAIN} | wc -l`
done
if [[ "$HOSTINDEX" == "1" ]]; then
FQDN="${HOST}${HOSTINDEX}.${DOMAIN}"
else
# set the new fqdn hostname and shortname
FQDN="${HOST}${HOSTINDEX}.${DOMAIN}"
SHORTNAME="${HOST}${HOSTINDEX}"
fi
# Set the instance hostname -- If you want to make sure that bash
# updates the prompt, run "exec bash" after the script. If you do so
# in the script, bad stuff happens on startup.
hostname $FQDN
echo -n $FQDN
# Add the DNS record
RESULT=`route53 add_record $ZONEID $FQDN CNAME $PUB_HOSTNAME | grep "PENDING"`
if [[ "$RESULT" == "" ]]; then
echo "... failed.";
else
echo "... success.";
fi
# Add the CNAME record
echo -n "Adding host to round-robin group...";
# Checking to make sure it's not already there.
CNAME=`route53 get $ZONEID | grep $RR | grep $PUB_HOSTNAME | wc -l`
if [[ $CNAME = 0 ]]; then
RESULT=`route53 add_record $ZONEID $RR.$DOMAIN CNAME $PUB_HOSTNAME 60 ${HOST}${HOSTINDEX} 1 | grep "PENDING"`
if [[ "$RESULT" == "" ]]; then
echo "... failed.";
else
echo "... success.";
fi
else
echo "already exists, ignoring";
fi
}
stop() {
echo -n "Deregistering host with Route 53 : "
HOST=`hostname | cut -f1 -d.`
FQDN=`hostname`
# check to make sure it exists in Route53
CNAME=`route53 get $ZONEID | grep $FQDN | grep $PUB_HOSTNAME | wc -l`
if [[ $CNAME > 0 ]]; then
RESULT=`route53 del_record $ZONEID $FQDN CNAME $PUB_HOSTNAME | grep "PENDING"`
if [[ "$RESULT" == "" ]]; then
echo "... failed.";
else
echo "... success.";
fi
else
echo "... not found, ignoring";
fi
echo -n "Deregistering host from RR CNAME..."
# Checking to make sure it exists
CNAME=`route53 get $ZONEID | grep $RR | grep $PUB_HOSTNAME | wc -l`
if [[ $CNAME > 0 ]]; then
RESULT=`route53 del_record $ZONEID $RR.$DOMAIN CNAME $PUB_HOSTNAME 60 ${HOST} 1 | grep "PENDING"`
if [[ "$RESULT" == "" ]]; then
echo "... failed.";
else
echo "... success.";
fi
else
echo "... not found, ignoring";
fi
# resets the hostname to default
hostname `curl -s --fail http://169.254.169.254/latest/meta-data/hostname`
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo "Usage: <servicename> {start|stop|restart}"
exit 1
;;
esac
exit $?
Many thanks to Dave McCormick for his blog post on the subject from which I borrowed heavily.
Balloonar Eclipse!
0Here in Kansas City, we’re on the very edge of the eclipse path, and it was a beautiful night for a balloon flight too. (KOJC 210053Z 34006KT 10SM CLR 22/09 A3013 RMK AO2 SLP196 T02170094). As I was out looking at the eclipse, along comes Jason Jones of Old World Balloonery with a load of passengers, and I got a balloonar eclipse as a bonus.
My crappy phone camera gave me neat lens flare showing the eclipse in progress next to the balloon:
Converting EC2 S3/instance-store image to EBS
61 year ago
Amazon’s instance-store images are convenient, but ephemeral in nature. Once you shut them down, they’re history. If you want persistence of data, you want to use an EBS instance that can be stopped and started at will without losing your info. Here’s the process I went through to convert a Wowza image to EBS for a client to use with a reserved instance. I’m going to assume no configuration changes for Wowza Media Server, as the default startup package is fairly full-featured. This process works for any other instance-store AMI, just ignore the Wowza bits if that’s your situation.
Boot up a 64-bit Wowza lickey instance. I was working in us-east-1, so I used ami-e6e4418f, which was the latest as of this blog post.
Once it’s booted up, log in.
Elevate yourself to root. You deserve it:
sudo su -
Stop the Wowza service:
service WowzaMediaServer stop
delete the Server.guid file. This will cause new instances to regenerate their GUID.
rm /usr/local/WowzaMediaServer/conf/Server.guid
Go into the AWS management console and create a blank EBS volume in the same zone as your instance.
Attach that volume to your instance (I’m going to assume /dev/sdf here)
Create a filesystem on it (note: while the console refers to it as /dev/sdf, Amazon Linux uses the Xen virtual disk notation /dev/xvdf):
mkfs.ext4 /dev/xvdf
Create a mount point for it, and mount the volume:
mkdir /mnt/ebs mount /dev/xvdf /mnt/ebs
Sync the root and dev filesystems to the EBS disk:
rsync -avHx / /mnt/ebs rsync -avHx /dev /mnt/ebs
Label the disk:
tune2fs -L '/' /dev/xvdf
Flush all writes and unmount the disk:
sync;sync;sync;sync && umount /mnt/ebs
Using the web console, create a snapshot of the EBS volume. Make a note of the snapshot ID.
Still in the web console, go to the instances and make a note of the kernel ID your instance is using. This will be aki-something. In this case, it was aki-88aa75e1.
For the next step, you’ll need an EC2 X.509 certificate and private key. You get these through the web console’s “Security Credentials” area. This is NOT the private key you use to SSH into an instance. You can have as many as you want, just keep track of the private key because Amazon doesn’t keep it for you. If you lose it, it’s gone for good. Download both the private key and certificate. You can either upload them to the instance or open them in a text editor and copy the text, and paste it into a file. Best place to do this is in /root. Once you have the files, set some environment variables to make it easy:
export EC2_CERT=`pwd`/cert-*.pem export EC2_PRIVATE_KEY=`pwd`/pk-*.pem
once this is done, you’ll need to register the snapshot as an AMI. It’s important here to specify the root device name as well as map out the ephemeral storage as Wowza uses those for content and logs. Ephemeral storage will persist through a reboot, but not a termination. If you have data that needs to persist through termination, use an additional EBS volume.
ec2-register --snapshot [snapshot ID] --description "Descriptive Text" --name "Unique-Name" --kernel [kernel ID] --block-device-mapping /dev/sdb=ephemeral0 --block-device-mapping /dev/sdc=ephemeral1 --block-device-mapping /dev/sdd=ephemeral2 --block-device-mapping /dev/sde=ephemeral3 --architecture x86_64 --root-device-name /dev/sda1
Once it’s registered, you should be able to boot it up and customize to your heart’s content. Once you have a configuration you like, right-click on the instance in the AWS web console and select “Create Image (EBS AMI)” to save to a new AMI.
Note: As of right now, I don’t think startup packages are working with the EBS AMI. I don’t know if they’re supposed to or not.
