Mobile Internet In Haiti, Part 2

A while back, I posted about getting mobile Internet in Haiti. As technology changes rapidly, especially when it comes to Haitian internet access, I figured I’d post an update, having just returned from there in late February.

If you have a GSM-capable US phone (most Samsung Galaxy devices use software-defined radios and can speak CDMA or GSM fluently, simply by switching an option in the software), you’ll need to unlock it for international use:

Sprint: Contact Sprint Customer Service while still in the US and ask them for an international unlock. As long as your account has been active for more than 60 days, this should be no problem. They’ll walk you through the UICC unlock process. It helps to be on the Sprint network while this unlock happens, but it can also happen over Wi-Fi if you’re already out of the country.

Verizon: Verizon generally does not lock their phones. You may want to check with Verizon to make sure yours is unlocked. See item #18 in their Global Roaming FAQ.

AT&T: If your phone is under contract with AT&T or is an iPhone, you’re pretty much out of luck. AT&T is so terrified of losing their customers that they will only unlock the phone if you buy out your installment contract or pay an ETF. The good news is that most cell phone repair shops know the unlock codes and will unlock them for you for a small fee. (This is a tip I got from the manager of a local AT&T store who thinks corporate policy on unlocking for international use is dumb). If your phone is out of contract, simply go to and fill out the form. There is nobody at AT&T you can talk to about this, nor can the store personnel help you. If the process fails, then you’re simply out of luck, and should consider choosing a more customer-friendly carrier next time.

T-Mobile: No idea. I don’t know anyone who has a T-Mobile device. I expect their policy is probably very similar to AT&T.

Once you get to Haiti, you can stop at either the Digicel or Natcom shops just outside customs at the airport in Port-Au-Prince. (I would expect that there’s a similar setup at Cap-Haitien.) Natcom will load you up with 5GB of data and some voice minutes for 1000 Gdes ($25 US). I don’t know what Digicel’s current pricing is, but I expect it’s comparable. If you’re going to be out in the provinces, Natcom seems to have a better network than Digicel. If you’re staying in and around Port-Au-Prince, either network should work fine for you as both carriers have HSPA+ networks. I don’t know what the Natcom coverage situation is like on La Gonâve, but Digicel has EDGE coverage on most of the island, and HSPA/+ around Anse-a-Galets.

The staff at the Natcom shop had no trouble setting up my Galaxy S4, and in 15 minutes I walked out of there on the Haitian network. Using it as a hotspot was merely a matter of turning it on, and didn’t require any further configuration. Internet speeds in PAP average in the 2-3Mbps range.

It should be noted here that with both carriers, all Facebook traffic is free and doesn’t count toward your data plan usage. This is a pretty cool deal. My understanding is that Facebook located an edge node within Haiti to reduce transit off-island, and free access to the growing smartphone population in Haiti was part of the deal.

On a similar vein, Google also seems to be getting better presence in Haiti, and I’m told they too have edge nodes located in-country. Their maps product actually has pretty good data in PAP, although directions are still iffy as the addressing system there is a little tricky, and there aren’t necessarily names attached to many of the minor streets. It’s pretty good at figuring out where you are though. I wonder how soon they’ll get a Street View rig down there.

When you leave, your SIM will still be usable for 90 days, after which it will expire and no longer function on the network. There is currently excellent public wifi at the PAP airport, so handing your SIM off to one of your Haitian hosts is probably your best bet, as they can get some additional usage out of whatever unused data/minutes are left on it.

(I also discovered that on my Galaxy S4, GPS didn’t work unless there was a SIM in the slot)


Multi-Tenant/IP Hosting for Wowza Streaming Engine

For most users, running Wowza Media Server/Wowza Streaming Engine (which I’ll refer to as just “Wowza”) are perfectly content running it out of the box as is on a dedicated server. Where it gets a little more interesting is when you have to co-exist with other server applications that want some of the same ports (I’m lookin’ at YOU, web servers!).

By default, Wowza binds itself to all available IP addresses on ports 1935 (RTMP, but will also take HTTP requests on that port), 8086 (for some basic management functions), 8083 and 8084 (for JMX), and with WSE4, 8087 (REST) and 8088 (WSE Manager). It won’t bind itself to port 80 specifically because of the common problem of co-existing with web servers. If you have Wowza enabled for IPv6, it will also bind to all available IPv6 addresses on the same ports.

This technique is also good for reducing surface area, where you can have remote management such as SSH or RDP listening on one address, and someone scanning your streaming server IP won’t find any open management ports to attempt to exploit.

Read more

Mobile Voice in Haiti

As a follow-on to my previous post about getting mobile internet, here’s one about getting voice service on your US phone (at least if you have a Sprint phone).

I have a Samsung Galaxy S4 on Sprint. Sprint’s CDMA voice network is incompatible with the GSM networks in most of the rest of the world, but recent Samsung Galaxy devices (at least the S3 and S4, and other devices of the same generation/platform) use a software-defined radio that can be made to speak GSM or CDMA at will, with a simple settings change. CDMA doesn’t require a SIM but LTE and GSM do, so the Galaxy is a de facto international phone.

Sprint lets you do international roaming calls for $2/min, which is absurdly high. It’s much better to get a SIM from a local carrier and use that. Making it do this is relatively simple. If your account is in good standing, a simple phone call to Sprint will unlock your phone for using other SIMs (and before you try to do this for a GSM carrier in the US, it explicitly does NOT work on AT&T or T-Mobile). This unlock process does require a data connection (mobile or Wi-Fi) for the phone to receive the unlock signal. After doing that, there’s a simple process that the Sprint rep will give you over the phone to complete the process.

Once that’s done (took me about 5 minutes on the phone – which I did via Skype from Haiti!), all you have to do is go find a local SIM (and in the case of the Galaxy, trim it down to size), pop it in the phone, switch it over to GSM in the Mobile Networks settings, pick your carrier, and off you go.

I’ll add screenshots just as soon as I can make the phone do them. The normal S4 tricks aren’t working.


Mobile Internet in Haiti

Note: Be sure to read my March 2015 update about this…

I’m back down in Haiti, as some of you already know, working on some of the wireless networks linking the different sites of the Église Méthodiste d’Haïti (EMH), which is the Haitian Methodist Church. Knowing that I was coming into an environment where the internet connection was not functioning properly, and that I was likely going to need internet access for troubleshooting, I armed myself with a 3G GSM hotspot that I picked up on eBay.

After parting with about 50 bucks (plus another 15 for a charger and 2 spare batteries), the Huawei E583C unit showed up via USPS on my doorstep 4 days later bearing a postmark from Hong Kong (color me impressed, I can’t even get postcards from Toronto that quickly!)

20131125_150332I opened it up and inside was a “T-Mobile Wireless Pointer” from the UK division of T-Mobile. I popped on down to the local T-Mobile store and get a SIM for testing, and fired it up. After much futzing around trying to get it to speak 3G to the network without any success, I go back to T-Mobile and pick a tech’s brains. Turns out this one operates on the 800/1800/1900 band, which T-Mobile has phased out 3G on to make room for more LTE. Meanwhile, Jay was in Haiti, so I asked him to pick up a NatCom SIM and bring it home with him.

I’ll pause briefly here to talk a bit about mobile in Haiti. There are two major players, Digicel (which has a thing for island nations all over the world) and NatCom, which is formed out of what was left of the national telephone company (Teleco) and the Vietnamese national telecom (VietTel) that bought up a 70% interest in Teleco not long after the earthquake. What little copper telecom infrastructure existed in the country has long since been destroyed by a number of different Screen Shot 2013-11-25 at 3.20.19 PMmeans, both natural and human. Since the earthquake, NatCom has been building out a LOT of fiber. Digicel operates the only direct fiber link out of the country to Columbus Networks‘ Fibralink fiber network that links the Caribbean up to the rest of the world. The other way out of Haiti to the internet is via microwave backhaul to the Dominican Republic which has 2 landings of the ARCOS fiber ring.

In the nearly 4 years since the quake, mobile internet in Haiti has gone nuts. It’s now quite reliable, and surprisingly cheap if you know how to do it. Monthly postpaid plans for data cost about a quarter what they do in the US – a 10GB plan on digicel will set you back 1000 HTG (about 25 bucks). The same plan on Verizon in the US by comparison is about $100! Digicel offers current-generation Android phones like the S4 (but be prepared to part with full unsubsidized price for it), and Apple recently started making unlocked SIM-less iPhones available on its own store. The smartphone revolution is coming to Haiti, and it’s going to be interesting to watch. There was someone at church on sunday using an iPad, and it wasn’t someone from our team.

When I got down to Haiti and put the SIM Jay obtained for me into the hotspot (erm, “Pointer”… can any Brits enlighten me as to the origin of that term?), and getting no joy. Realizing that the zillion config changes I’d made to try and get it to work on T-Mobile’s network were probably interfering, I hit the factory reset button, and as soon as it rebooted, it was speaking 3G on Natcom’s network. It was that easy.

Next step was to load up some funds on the card, since it was a basic card that came empty of funds. Normally you can do this from the phone, but since this was a hotspot, I didn’t have the ability to dial numbers (although the Huawei firmware does allow you to SMS, which turned out to be a critical component). Natcom partners with a third party called EzeTop which allows you to reload phone cards online (yours or anyone else’s). So I dropped 10 bucks onto it (which translates to 392 HTG, a fairly lousy exchange rate) plus a penny per 10 Goudes as a transaction fee, and off I go. No sign anywhere of what the per-MB cost is. NatCom’s website isn’t particularly helpful in that regard (I later find out that it’s 1.9HTG/MB, about 4 cents.)

Now that I had mobile internet, I fired up the iPad and did some testing on the drive to Petit-Goave, and was getting quite reasonable speeds around 1.5-2Mbps in both directions, very much capable of posting pictures to facebook and whatnot.

Once we got to the guest house where we were staying, we discovered that the wifi there was indeed out of service. I put the hotspot to good use downloading information I was going to need to fix it. In very short order, net access ceases, and I get a screen from NatCom saying that my card is empty, and provides a helpful list of plans and how to activate them. I then go find our hostess and borrow her laptop and internet access to load up some more funds on the card, and then try to activate one of the listed plans. It tells me I can’t do that because I have the wrong type of card.

Then, disaster. Within a matter of little more than an hour, 20 bucks worth of data on the card had vanished. After some digging, I discovered that my good buddy CrashPlan had stabbed me in the back and decided to start a big backup. I killed CrashPlan and reloaded the card (this is getting expensive, and I’m still not entirely sure how much data I’m burning through, especially now that the team is sharing in the internet joy — and the cost!)

Now that I’m back online, I start digging around the NatCom site again to figure out what plans I can access through the SIM I already have. Turns out that they have slightly different SIMs and plans for laptop/USB modems and for mobile phones. I had the latter, a “Nat-Mango” card, which can be had from any street vendor for 25 HTG. I finally found the list of mobile internet plans for the phones, and the correct number to SMS the plan change to. So I send off the text, only to get back “You don’t

Screen Shot 2013-11-05 at 8.03.55 AM

have enough funds for this plan”. I keep moving down the list until even the cheapest one kicks back the message… Uh-oh, I’m running on fumes again. Just as I go to top it up again, it shuts off. Fortunately, one of our Haitian team members had data on his Digicel phone, and I was able to get the account charged up, and switched over to the “Unlimited” plan. Unlimited in this case means 3.5GB at max HSPA+ speeds, then you’re rate-limited to 3.5 Mbps after that. Given that I never saw 3Mbps anywhere, this isn’t really a huge hindrance (that may be a factor of the device more than the network, too). By the time the week was out, our team had gobbled up nearly 25 gigabytes of data through the device.

So, in short, mobile internet from local carriers in Haiti is reliable and cheap (if you know the trick to not paying out the nose per MB), and can be done on a fairly inexpensive piece of hardware. If you’re so inclined, you can also get USB sticks from NatCom for about 1500 HTG. My next step is going to be to see if a device from Cradlepoint can handle the Natcom USB sticks, since they don’t have such a tight limitation on clients.

Fixing network Priority in Windows : Win7 Update

A long time ago, I made a post about fixing network priority in Windows, and I found myself having to do the same task again on my new Windows 7 system. The process isn’t quite as easy to find under Windows 7/Vista. Here’s the updated version:

Right-click on your network icon and go to the “Network and Sharing center” (if the “Network” icon is on your desktop, you can also get there by right-clicking and going to properties)

Click on “Change Adapter Settings”

Network Advanced

Press the “Alt” Key to show the menu, and click on “Advanced”, then “Advanced Settings”.

(from here, the process is unchanged)

Move the Wired LAN Connection (By Default, “Local Area Connection”) to the top, followed by the wireless connection. Make sure that any VPN virtual adapters come after these, otherwise the VPN will only use the ones above it. This tends to be problematic if you’re using split tunneling, as it will kill any network connection you have.

Once you’ve applied the settings, open a command prompt and run “nslookup” – it should default to the DNS server for your wired network.

March Madness: The Network Plumber’s Perspective

Web video is clearly here to stay. Heck, I currently have 40% of my time dedicated to producing and delivering web video of our weekend worship services. I think this is tremendously cool stuff, and traditional one-way RF-based video delivery (a.k.a. TV) is pretty lame. My kids have no concept of a broadcast schedule. Their content world is one that is immersive, interactive, and on-demand.

We’re now coming up on that season that we network admins have begun to dread over the last few years: March Madness. With networks advertising live web video of every. single. game., those of us charged with the care and feeding of our WAN pipes are blanched in abject terror. We know that 95% of our staff is going to want to watch them while they work. It doesn’t take much math skill to figure out that multiplied by a couple hundred people, even viewing one event means that the remaining 3 people in the organization that don’t really give a hoot about hoops aren’t going to be able to get any work done and pick up the slack the rest of us are leaving.

When you do internet video on the scale of the NCAA tournament, or a news network during a major news event, you’re relying on the performance of your CDN. Naturally, you want to accurately count eyeballs so that the advertisers pay you appropriately. A lot of time and effort goes into engineering thse things, and it’s quite remarkable how well this works.

CNN’s approach using Octoshape is a creative one, that really pushes P2P technology into the mainstream of legitimacy. I was present at the creation nearly ten years ago [+] [++]when Gnutella was leaked to the world, and changed the rules of the multimedia distribution game, and recall thinking how interesting things were going to become. Out of the Gnutella proof-of-concept came LimeWire and others, and then BitTorrent figured out how to dial the concept to a global scale. Now the same idea is being integrated into mainstream CDNs with Octoshape and other “cloud” applications.

It seems to me that the CDN operators should be able to find a way to engineer their networks such that a corporate network admin (such as myself) could download a piece of software onto a spare piece of gear and run a node of the CDN, internal to the corporate network (or, for that matter, run it as a VMWare virtual appliance). This not only softens the blow to my WAN pipes, but also lightens the load on the public parts of the CDN. The only thing then going across the WAN connection is a single instance of each stream being requested by clients internal to the company. Then it simply phones home with the proper client count for advertiser tracking, and bingo, people can get work done, as well as watch their favourite team make a run at the Final Four.

…Or we network admins can simply block the CDN in their content filters and tell their users that we’re mean party poopers, depriving them of their hoops and depriving the webcasters of their revenue.

What happens during a slashdotting?

Well, OK, it wasn’t Slashdot that was the culprit this time, but rather the pro blog Consumerist (if we’re a megachurch, does Consumerist count as a megablog? It claims nearly 3 million unique visitors a month)

Last week, Clif posted about his experience at Best Buy. Seems the folks at Gawker Media got wind of the story (Best Buy is a perennial favourite target of theirs) and posted it at 10:21am Eastern, 9:21 in KC. Here’s what happened to our WordPress server:

Apache Processes, November 5

Apache Processes, November 5

Web Datacenter traffic, November 5 (% of 10Mbps link)

Web Datacenter traffic, November 5 (% of 10Mbps link)

Wow. I noticed the odd traffic behaviour (that particular server gets very little traffic most of the time) when I got in the office, and called a few folks to see if they’d done anything that would cause this. When that came up empty, I started looking at the access logs on the server and noticed a lot of referrer traffic from Consumerist. I threw AWstats onto the server to grok the apache logs. At posting time, Clif’s blog post had seen around 7000 visitors. Apache peaks out at a point due to the MaxClients directive, in order to keep the CPU from saturating and killing the site.

It’s always fun to see new an interesting traffic patterns. It’s very helpful to have active monitoring to tell us when things leap outside the bounds of normalcy.

Bandwidth woes

Given Kansas City’s central location on the IP backbones that traverse the US, it should be a heck of a lot easier than it is to get some semblance of decent bandwidth at the church (and other locations). Sadly, the best we can do is 1.5Mbit DSL (even if the line has been tested from our demarc to support 7.5Mbit, AT&T refuses to sell us any more than 1.5Mbit because their computers say that’s the best they can do) or bonded T1s.Even then, there’s a finite limit to what what we can achieve with that, because we have a finite amount of copper feeding the building.

Anything else involves construction costs in the $50K and up range. Even amortized over a 5-year contract, that still tacks on nearly $1000 to the monthly bill. Point-to-point wireless to somewhere with fiber would cost almost as much.

It amazes me that in one of the more affluent suburbs in the country, the best we can do is no better than what you get in and around most small towns in rural western Kansas.

Fixing network priority in Windows

Recently, we made some changes to the DNS infrastructure on our public wireless networks which has had the unintended consequence of breaking things when our laptop users are plugged into the LAN and have their wireless active. Brian and I have wrangled with this in the office, but we simply turned off the wireless as a workaround.

What’s happening is that when connected to both networks, the wireless has a higher priority by default, and so it resolves DNS via that interface. This is problematic when trying to access an internal resource, because our DNS is set to have a default resolution to our website for * To complicate matters further, Arena behaves differently when you’re on the guest network (sends to a forms-based auth portal instead of using IE integrated authentication).

After much digging, I found out how to change interface priority. Here’s the process in XP screenshots (the process is similar in Vista):

1. Open your network connection properties (XP: Via control panel or right-click on Network Places, then select Properties. Vista: Go to Network and Sharing Center and select “Manage Network Connections in the links on the left)

XP Network Properties

XP Network Properties

2. On the menu bar (press Alt to show it in Vista), Select Advanced, then “Advanced Settings”

Advanced Network Properties Dialog (XP)

Advanced Network Properties Dialog (XP)

3. Move the Wired LAN Connection (By Default, “Local Area Connection”) to the top, followed by the wireless connection. Make sure that any VPN virtual adapters come after these, otherwise the VPN will only use the ones above it. This tends to be problematic if you’re using split tunneling, as it will kill any network connection you have.

4. Hit OK, and you’re good to go.

High-Availability firewall on the cheap!

I now have a profound appreciation for BSD.

Yesterday, our pfSense firewall at 1102 Grand suddenly went silent and the panel on WhatsUp for that site went all red. Not good. I went down to the cage after small group last night and found the machine to have just simply locked up cold. I suspect hardware, since pfSense/BSD didn’t log a thing about it going dark.

It became quite clear that this setup was… suboptimal. Clif‘s shooting for 99.99% on this new setup. I can’t be racing off to the datacenter every time the firewall machine decides to take a holiday from reality. Brian and I quickly determined that we needed not only a remote power control unit, but some sort of high-availability solution that wasn’t going to empty our wallet like a pair of NSA 4500s would. (sorry Mark, we simply don’t have that kind of money) We already had a spare, identical machine at the datacenter doing duty as a hardware spare and development server, and another one just like it in inventory at the Central Campus. I grabbed the extra machine and went back 1102 Grand for the second time in 12 hours, with a quick stop at Micro Center for some cheap NICs and a red crossover cable.

Fortunately, pfSense has high availability capability built in, thanks to BSD’s CARP and pfSync. CARP allows me to set up virtual IPs on both firewalls and synchronize between them with pfSync. The extra NICs were for a dedicated sync/heartbeat link between the two boxes. I’m still a little fuzzy on the technical details of how this works, but it works… Convergence/Failover time is 3 seconds or less, and everything is synchronized between the two machines, including state and session information. I had it all set up, and hit the reset button on the primary firewall… and nary a ping was dropped.
The setup involves giving each machine its own LAN and WAN addresses (and a unique address on the other zones as well) and then creating a CARP virtual IP on that interface. The virtual IP is the one used as the gateway and as the NAT address. All rules and configs (including IPSEC Tunnels!) is mirrored on the second box.

Reference material:

Excellent documentation on the process from the folks at Countersiege.

Some background from the OpenBSD folks.

A few good tips here. These proved to be crucial.