What’s In That Survey Kit? (Fall 2019 Edition)

The life and contents of a survey kit is a dynamic one. Here’s what’s in my kit these days. The Pelican 1510 is airline carry-on size, because there’s no way they’ll let you check that stuff with the batteries (which are all just under the airline limit of 100Wh), and the contents are valuable enough that you probably don’t want it out of sight, or trust it to the airline baggage handlers. If you’re carrying this stuff, it’s because you need it at your destination.

Links go mostly to Amazon where I get all this stuff

All this fits inside the case…
See? Whole thing weighs a little over 20 lbs, and there’s room to spare. Bright yellow inside makes it harder to lose stuff in there. Tripod is held in place with some velcro straps screwed to the lid.

  • Floater items that go between kits:
    • BatPower PDE2 96Wh USB Battery Pack (Amazon has removed this item. The upcoming Accelerator 2.0 battery pack from AccelTex will have full power type C PD ports on it)
    • EU/US power plug adapter
    • Swift Body Platform Harness (for carrying survey laptop and avoiding Survey Elbow, the nerd version of tennis elbow)

I used to carry a USB powered external monitor but DisplayLink doesn’t work so great on Macs anymore. I’m going to give this no-name 4K USB3 Type C display a go and see how it performs.

Also, if you’ve started adding this up in your head, you can see why I carry this on instead of leaving it to the baggage system. Make sure your business has insurance, especially if you’re self employed. If you’re traveling overseas, you’ll need special insurance coverage. I used to carry audiovisual insurance (the kind news crews carry) when doing streaming, and those policies will even cover against force majeure and acts of God. I don’t know if there’s an IT equivalent.

Hiding In Plain Sight

One of my favorite things to do when I’m at a Disney park is to play the wireless nerd’s version of Hidden Mickeys: Trying to spot the myriad creative ways in which Disney’s Imagineers have blended their excellent wireless network into the carefully contrived scenery. It truly is magical how they can make wireless everywhere while keeping it nearly invisible.

So naturally, when I’m wearing the wireless engineer hat and have a challenge where I get to flex some of that creativity, I’m all over it.

A few years back, I helped a church in Wichita overhaul their aging and underpowered WiFi by designing and installing a new Ruckus system. Last year, they embarked on a new project to add a chapel to their campus. Naturally they wanted to extend the wireless LAN to this new building.

But… It’s a chapel aimed at doing weddings and other sorts of events, so it was paramount that the wireless equipment not be visible, to maintain clean architectural lines with a minimum amount of obvious tech equipment. Some concessions had to be made for audiovisual, but visible access points were a (network) bridge too far.

After pondering the problem as well as observing drawings and renderings, I happened upon the architectural lighting elements in the plan that were mounted on each of the columns. I dug into the design of these and discovered that they were a pair of LED fixtures concealed inside some finish carpentry with a textured plastic surface. And most importantly, there was an empty space in the middle between the two light fixtures that measured about 20cm square by 40cm high, and centered approximately 8 feet off the floor. Not only was that low enough to keep the APs close to the clients, there was plenty of room to put in one of the Ruckus H510 Wall APs designed for the hospitality market (which I also currently have in my house running Unleashed, although they will soon make way for some of the Aruba AP303H units or their new Instant On AP11D counterpart). I’m a big fan of these in-wall units for many reasons.

I asked the electricians to give me a box and conduit to four of these columns, as well as a pair of data cables. I only planned to use two access points initially, but since running cable would be prohibitively difficult after the buildout, I wanted to keep my options open should capacity needs increase in the future.

After many months of construction (Summer of 2019 was an utterly awful weather summer if you were in the construction business), I finally got the green light to install these. I took a bit of personal time on my way down to another job in Oklahoma for my employer, and executed the plan. I’m pretty happy with the results.

The lighting fixture: two pieces of dark wood on either side floating 1″ off the wall with a textured face and tunable color temp LED fixtures facing up and down
The lighting fixture with the plastic face slid up (there’s a stop at the bottom). An electrical box was placed behind it and a 2″ hole drilled for cable access. The overall construction of this fixture is beautifully simple: a few pieces of solid oak and some stain. The overall look in this space is one of stone, wood, and glass, with 90° and 45° angles being dominant.
The Ruckus H510 bracket screwed directly to the finish carpentry. The mount could also have screwed to the electrical box but that was an unnecessary level of effort.
The Ruckus H510 access point mounted on the bracket.
The fixture with the AP mounted inside. The wood and textured face provide minimal attenuation, and in this environment, I’m using the attenuation constructively. The recess in white is where a large TV (in lieu of projection) will be mounted on a swing arm and can fold into the wall when not in use.
Side view – the gap was just enough to get a screwdriver in to secure the AP to the bracket using the provided T10 screws. I was concerned that this wouldn’t be possible.
Lighting fixture side view. Is this with or without the AP installed? If you can’t tell, that means I was successful.
The AP was mounted on the second column from the back of the room, near the sound booth. The corresponding column on the other side is wired for an AP if capacity requires one.
The AP was mounted on the third column from the front of the room, near the front of the stage. The corresponding column on the other side is also wired for an AP if additional client capacity requirements dictate it.
Morning Fog along the Flint Hills National Scenic Byway

Mist Deployment, Part Deux

Second in a series about our first deployment of a Mist Systems wireless network. 

In my last post, I gave you an overview of the various components of the Mist Wireless system. This post will go into some of the design considerations pertaining to this particular project.

Because we’re now designing for more than just Wi-Fi, there are a few additional things to factor in when planning the network.

Floor Plans

It’s not uncommon for your floor plans to have a “Plan North” that doesn’t always line up with “Geographic North”. Usually this isn’t a factor, but looking at it in hindsight, I would strongly encourage you to build your floor plans aimed at geographic north from the start, as the Mist AI will also use that floor plan for direction/wayfinding and the compass in mobile devices will be offset if you just go with straight plan north. You can also design on plan north, but then output a second floor plan file that is oriented to true north. Feature request to Mist: Be able to specify the angle offset of the plan from true north and correct that for user display in the SDK.

For this project, I had access to layered AutoCAD files for the entire facility, which (sort of) makes things easier in Ekahau Site Survey, but sort of doesn’t – the import can get a little overzealous with things like door frames. I had to go do a fair bit of cleanup afterwards, and might have been better off just drawing the walls in the first place. This was partly due to the general lack of any good CAD tools on MacOS that would have allowed me to look at the data in detail and massage it before attempting the import into Ekahau. The other challenge is that ESS imported the ENTIRE sheet as its view window, which made good reporting impossible as the images had wide swaths of white space. Having the ability to crop the CAD file would have been nice.

Density Considerations

View from the rear of the main sanctuary at College Park Church in Indianapolis.

Since one of the areas being covered is a large auditorium, we had to plan on multiple small cells within the space. We needed to put the APs in the catwalks, as we did not have the option of mounting the units on the floor because of the sanctuary being constructed onslab (and while the cloud controller allows you to specify AP height and rotation from plan north, there is no provision to tell it the AP is facing *up* and located on/near the floor). This posed a few challenges, the first being that we were well above the recommended 4-5m (the APs were at 10m from the floor), the other being that we needed to create smaller cells. For this, we used the AP41E with an AccelTex 60-degree patch antenna.

Acceltex 8/10 dBi 60° 4-element patch antenna

 We also needed to either run a whole lot of cables up to the theatrical catwalks, or place a couple of small managed PoE switches – we unsurprisingly opted for the latter, using two 8-port Meraki switches, and uplinked them using the existing data cabling that was feeding the two UniFi APs that were up there.

As an added bonus, the sanctuary area was built with tilt-up precast concrete panels, which allowed us to use that heavy attenuation to our benefit and flood the sanctuary space with APs and not worry about spilling out too much.

Capacity-wise, we used 10 APs in the space, which seats 1700. Over the course of several church designs, I’ve found that a ratio of one active user for every three seats usually works out pretty well – in most church sanctuaries, the space feels packed when 2/3 of the seats are occupied, which means that we’re actually planning for one client for every two seats. Now, we’re talking active clients here, not associated clients. An access point can handle a lot more associations than it can active clients. As a general rule, I try to keep it to about 40 or 50 active clients per AP, before airtime starts becoming a significant factor.

In an environment like this, you want as many client devices in the room to associate to your APs, even if they’re not actively using them – when they’re not associated, they’re sitting out there, banging away with probe requests (especially if you have any hidden SSIDs), chewing up airtime (kind of like that scene from Family Guy where Stewie is hounding Lois just to say “Hi.”). Once they associate, they quiet down a whole bunch.

In addition to the main sanctuary, there are also a couple of other smaller but dense spaces: the chapel (seats 300) and the East Room (large classroom that can seat up to 250). In these areas, design focused on capacity, rather than coverage.

Structural Considerations

As is often the case with church facilities, College Park Church is an amalgamation of several different buildings built over a span of many years, accommodating church growth. What this ends up meaning is that the original building is then surrounded on multiple sides with an addition, and you end up with a lot of exterior walls in the middle of the building, as well as many different types of construction. Some parts of the building were wood-frame, others were steel frame, and others were cast concrete. The initial planning on this building was done without an onsite visit, but the drawings made it pretty obvious where those exterior (brick!) walls were. Naturally, this also makes ancillary tasks like cabling a little interesting.

Fortunately, the church had a display wall that showed the growth of the church which included several construction pictures of the building, which was almost as good as having x-ray vision.

Aesthetic Considerations

Because this is a public space, the visual appearance of the APs is also a key factor – Sometimes putting an AP out of sight takes precendence over placing for optimal Wi-Fi or BLE performance.

Placement Considerations

Coverage Area

Mist specifies that the BLE array can cover about 2500 square feet. The wifi can cover a little more, but it doesn’t hurt to keep your wifi cells that size as well, since you’ll get more capacity out of it. In most public areas of the building, we’re planning for capacity, not coverage. With Mist, if you need to fill some BLE coverage holes where your wifi is sufficient, you can use the BT11 as a Bluetooth-only AP.

AP Height

Mist recommends placing the APs at a height of 4-5m above the floor, in order to provide optimal BLE coverage. The cloud controller has a field in the AP record where you can specify the actual height above the floor.

AP Orientation

Because the BLE array is directional, you can’t just mount the APs facing any direction you please. These APs are really designed to be mounted horizontally, the “front” of the AP should be consistently towards plan north, but the controller does have the ability to specify rotation from plan north in case mounting it that way isn’t practical. The area, orientation and height are critical to accurate calculation of location information.

AP Location

Several of the existing APs in older sections of the building were mounted to hard ceiling areas, and we had to not only reuse the data cable that was there, but also the location. Fortunately, the previous system (Ubiquiti UniFi) was reasonably well-placed to begin with, and we were able to keep good coverage and reuse those locations without any trouble.

There were also some co-existence issues in the sanctuary where we had to make sure we stayed out of the way of theatrical lighting and fixtures that would pose a problem with physical or RF interference. In the sanctuary, we also have to consider the safety factor of the APs and keeping them from falling onto congregants like an Australian Drop-Bear.

Planning for BLE

Since starting this project, I’ve begun working with Ekahau on testing BLE coverage modeling as part of the overall wifi coverage, and it’s looking very promising. I was able to go back to the CPC design and replan it with BLE radios, and it’s awesome. Those guys in Helsinki keep coming up with great ideas. As far as Ekahau is concerned, multi-radio APs are nothing too difficult – They’ve been doing this for Xirrus arrays for some time now, as well as the newer dual-5GHz APs.

Stay tuned for a post about BLE in Ekahau when Jussi says I’m allowed to talk about it.

Up Next: The Installation

 

Cover Image: Explore Kansas: The Flint Hills National Scenic Byway (Kansas Highway 177)

Misty valley landscape with a tree on an island

Mist Deployment (Part The First)

First in a series about our first deployment of a Mist Systems wireless network. Mist Systems Logo

Over the course of the past few months, I’ve been working with the IT staff at College Park Church in Indianapolis to overhaul their aging Ubiquiti UniFi wireless system. They initially were looking at a Ruckus system, owing to its widespread use among other churches involved with the Church IT Network and its national conference (where I gave a presentation on Wi-Fi last fall). We had recently signed on as a partner with industry newcomer Mist Systems, and had prepared a few designs of similar size and scope for other churches in the Indianapolis area using the Mist system. We proposed a design with Ruckus, and another with Mist, with the church selecting Mist for its magic sauce, which is its Bluetooth Low Energy (BLE) capability for location engagement and analytics.

Fundamentally, the AP count, coverage, and capacity were not significantly different with Ruckus vs. Mist, and Mist offered a few advantages over the Ruckus in terms of the ability to add external antennas for creating smaller cells in the sanctuary from the APs mounted on the catwalks, as floor mounting was not an option.

About Mist

Mist is a young company that’s been around for about two or three years, and they have developed a couple of cool things in their platform – The first is what they call their AI cloud, the second is their BLE subsystem, and the last is their API.

Their AI component is a cloud management dashboard (similar to what you would see with Ruckus Cloud or Meraki — many of the engineers that started with Mist came over from Meraki), where the APs are constantly analyzing AP and client performance through frame capture and analysis, and reporting it back to the cloud controller. The philosophy here is that a large majority of the issues that users have with Wi-Fi performance is actually related to performance on the wired side of the network (“It’s always DNS.” Not always, but DNS — and DHCP — are major sources of Wi-Fi pain). The machine learning AI backend is looking at the stream of frames to detect problems, and then using that to generate Wi-Fi SLA metrics that can help determine where problems lie within the infrastructure, and doing some analysis of root causes. An example of this is monitoring the entire Station/AP conversation during and shortly following the association process. It looks at how long association took. How long DHCP took (and if it was successful), whether 4-way handshakes completed, and so on. It will also keep a frame capture of that conversation for further manual troubleshooting. It also keeps a log of AP-level events such as reboots and code changes so that client errors can be correlated on a timeline to those events. There’s a lot more it can do, and I’m just giving a brief summary here. Mist has lots of informational material on their website (and admittedly, there’s a goodly amount of marketing fluff in it, but that’s what you’d expect on the vendor website).

Graphs of connection metrics from the Mist system

 

 

 

 

 

 

 

 

 

 

Next, we have their BLE array. This is what really sets Mist apart from the others, and is one of the more interesting pieces of tech to show up in wifi hardware since Ruckus came on the scene with their adaptive antenna technology. Each AP has not one, but *eight* BLE radios in it, coupled with a 16-element antenna array (8 TX, 8RX). Each antenna provides an approximately 45° beam covering a full circle. Mist is able to use this in two key ways. One is the ability to get ridiculously precise BLE location information from their mobile SDK, (and by extension, locate a BLE transponder for asset visibility/tracking) and the other is the ability to use multiple APs to place a virtual BLE beacon anywhere you want without having to go physically install a battery-powered beacon. There are myriad uses for this in retail environments, and the possibilities for engagement and asset tracking are very interesting in the church world as well.

Lastly, we have their API. According to Mist, their cloud controller’s web UI only exposes about 40% of what their system can do. The remainder is available via a REST API that will allow you do do all kinds of neat tricks with it. I haven’t had a chance to dig into this much yet, but there’s a tremendous amount of potential there. Jake Snyder has taught a 3-day boot camp on using Python in network administration to leverage the power of APIs like the one from Mist (Ruckus also has an API on their Cloud and SmartZone controllers)

Mist is also updating their feature set on a weekly basis – rather than one big update every 6 months that may or may not break stuff, small weekly releases allow them to deploy features in a more controlled manner, making it easy to track down any potential show-stopper bugs, preferably before they get released into the wild. You can select whether your APs get the early-release updates, or use a more extensively tested stable channel.

Much like Meraki, having all your AP data in the cloud is tremendously useful when contacting support, as they have access to your controller data without you having to ship it to them. They can also take database snapshots and develop/test new features based on real data from the field rather than simulated data. No actual upper-layer traffic is captured.

The Hardware

note: all prices are US list – specific pricing will be up to your partner and geography.

There are four APs in the Mist line. The flagship 4×4 AP41 ($1385), the lower-end AP21 ($845), the outdoor AP61 ($?) , and the BLE-only BT11 ($?). The AP41 also comes in a connectorized version called the AP41E, at the same price as the AP41 with the internal antenna.

The AP41/41E is built on a cast aluminum heat sink, making the AP noticeably heavy. It offers an Ethernet output port, a USB port, a console port, and what they call an “IoT port” that provides for some analog sensor inputs, Arduino-style. It requires 802.3at (PoE+) power, or can use an external 12V supply with a standard 5.5×2.5mm coaxial connector. In addition to the 4-chain Wifi radio and the BLE array, the AP41 also has a scanning radio for reading the RF environment. On the AP41E, the antenna connectors are located on the downward face of the AP.

The AP21 is an all-plastic unit that uses the same mounting spacing as the AP41, and has an Ethernet pass-through port with PoE (presumably to power downstream BT11 units or cameras). Like the AP41, it also has the external 12V supply option.

This install didn’t make use of BT11 or AP61 units, so I don’t have much hands-on info about them.

It’s also important to note that none of these APs ship with a mounting bracket, nor does the AP have any kind of integrated mounting like you would find on a Ruckus AP. Mist currently offers 3 mounting brackets: a T-Rail bracket ($25), a drywall bracket ($25) and a threaded rod bracket ($40). The AP attaches to these brackets via four T10 metric shoulder screws (Drywall, Rod), or four metric Phillips screws (T-Rail). More on these later.

The Software

Each AP must be licensed, and there are three possibilities: Wifi-only, BLE Engagement, and BLE Asset tracking. Each subscription is nominally $150/year per AP, although there are bundles available with either two services or all three. Again, your pricing will depend on your location and your specific partner. Mist recently did away with multi-year pricing, so there’s no longer a cost advantage in pre-buying multiple years of subscriptions.

When the subscription expires, Mist won’t shut off the AP the way Meraki does, however, the APs will no longer have warranty coverage. After a subscription has been expired for two months, Mist will not reactivate an AP. The APs will continue to operate with their last configuration, however, but there will no longer be access to the cloud dashboard for that AP.

Links:

Mist Systems

Jake Snyder on Clear To Send podcast #114: Automate or Die

Mist Product Information

Up Next: The Design

What’s In Your Go-Kit?

As I prepare for another trip to a customer site, I figured I’d post the contents of my wireless engineering go-kit for the benefit of others wanting to put one together. I’ve posted previously about my streaming go-kit, which has largely been retired as I’m not doing nearly as much streaming as before, having shifted over to Wi-Fi. Amazon links in this post are affiliate links, and it’s where I bought most of this stuff over the course the the last several years. Some of it was freebies from conferences like the Wireless LAN Professionals Conference.

What’s in the kit?

It will depend largely on the job I’m going to do, but I’ve got several sub-kits that go in it based on the needs of the job:

Frame Analysis Sub-Kit:

(this kit has largely been deprecated by my Macbook and Airtool)

  • 3 Netgear A6210 2SS 802.11ac adapters for use with Omnipeek – I don’t know if the 3SS version A7000 has requisite drivers for Omnipeek. Word on the street is that Metageek EyePA recently added support for these adapters. AirMagnet can also use these for surveys.
  • 1 AirPCAP Adapter for use with Omnipeek (pretty much obsolete at this point)

Site Survey Sub-Kit:

Spectrum Analysis Sub-Kit:

Pentest Sub-Kit:

Ethernet/Console Sub-Kit:

Test Tools:

Measurement/Installation Tools:

Miscellaneous:

Computing:

Software:

PPE/Safety

Depending on the combination of stuff, most of it goes in a Pelican 1510 carry-on case (yes, it all fits – other than the PPE – with some room to spare, especially if you add the lid organizer, which is great for keeping small things contained!) . Because some of the devices in there contain lithium batteries, I can’t check it – but in that case the scissors and the knife need to go in checked luggage – But if you do some mental calculations and add up what all this stuff costs, you’ll see that even without the computers and software, that’s not generally something I am willing to let out of my immediate control. I don’t bother with TSA locks, because those don’t provide any security.

If I only need some of the items, I put them in a smaller nylon case that used to be a carrying case for a projector, which does fit in a checked suitcase. The fiber kit has a dedicated Pelican 1490 case when not traveling in the 1510.

Wireless Engineering Kit in a Pelican 1510 case.

EC2 Monitoring with Raspberry Pi

I’ve been doing a little Raspberry Pi hacking lately, and put together a neat way to have physical status LEDs on your desk for things like EC2 instances.

The Hardware

In its most basic form, you can simply hook up an LED and a bias resistor between a ground line and a GPIO line on the Pi, but that doesn’t scale especially well – You can run out of GPIO lines pretty quickly, especially if you’re doing different colors for each status. Plus, it’s not overly elegant.

The solution? Unicorns!

No, really. The fine folks at Pimoroni in Sheffield, UK have made a lovely little HAT device for the Pi called a Unicorn. Its primary purpose is lots of blinky lights to make pretty rainbows and stuff, hence the name. However, this HAT is a 4×8 (or an 8×8) array of RGB LEDs, addressable via the I2C bus, which doesn’t eat up a line per LED (good thing, otherwise it would require 96 analog lines). The unicornhat library (python3-unicornhat) is available for Python 2 and Python 3 in the Raspbian repo. When installed onto the Pi, the Unicorn will fit within a standard Raspberry Pi case.

The Code

This is my first foray into Python, so there was a bit of a learning curve. If you’re familiar with object-oriented code concepts, this should be easy for you. Python is much more parsimonious with punctuation than PHP or perl are.

For accessing the EC2 data, we’ll need Amazon’s boto3 library, also available in the Raspbian repo (python3-boto). One area where boto3 is really nice is that the data is returned directly as a dict object (what users of other languages would call an array), so you don’t have to mess with converting JSON or XML into an object structure, and it can be manipulated as you would any other associative array (or a hash for you old-timers that use perl). AWS returns a fairly complex object, so you kind of have to dig into it via a few iterative loops to extract the data you’re after.

From there, it’s a matter of assigning different RGB values to the states. I chose these ones:

  • stopped: red
  • pending: green
  • running: blue
  • stopping: yellow(ish)

I also discovered that I needed to assign a specific pixel to each instance ID, otherwise they tended to move around a bit depending on what order AWS returned them on a particular request.

Here’s what the second iteration looks like in action:

import boto3 as aws
import unicornhat as unicorn
import time

# Initialize the Unicorn
unicorn.clear()
unicorn.show()
unicorn.brightness(0.5)

# Create an EC2 object 
ec2 = aws.client('ec2')

# Define colors and positions
color = {}
color['stopped']={'red':255,'green':0,'blue':0}
color['pending']={'red':64,'green':255,'blue':0}
color['running']={'red':32,'green':32,'blue':255}
color['stopping']={'red':192,'green':128,'blue':32}
	
pixel = {}
pixel['i-0fa4ea2560aa17ffd']={'x':0,'y':0}
pixel['i-06b95cd864acb1a8c']={'x':0,'y':1}
pixel['i-0661da0f50ffb604c']={'x':0,'y':2}
pixel['i-063ec151e0f44ef9b']={'x':0,'y':3}
pixel['i-02c514ca567d8a033']={'x':0,'y':4}

# Loop until forever
while True:

	response = ec2.describe_instances()
		
	
	statetable = {}
	resarray = response['Reservations']
	for res in resarray:
		instarray = res['Instances']
		for inst in instarray:
			iid = inst['InstanceId']
			state = inst['State']['Name']
			# print(iid)
			# print(state)
			statetable[iid] = state
	
	
	for ec2inst in statetable:
		x = pixel[ec2inst]['x']
		y = pixel[ec2inst]['y']
		r = color[statetable[ec2inst]]['red']
		g = color[statetable[ec2inst]]['green']
		b = color[statetable[ec2inst]]['blue']
		# print(x,y,r,g,b)
	
		unicorn.set_pixel(x,y,r,g,b)
		unicorn.show()


	time.sleep(1)

For the moment, this is just monitoring EC2 status, but I’m going to be adding checks in the near future to do things like ping tests, HTTP checks, etc. Stay tuned.

Enhancing the public Wi-Fi experience

Recently, there was an excellent blog post from WLAN Pros about “Rules for successful hotel wi-fi“. While it is aimed primarily at Wi-Fi in the hotel business (where there is an overabundance of Bad-Fi), many of the tips presented also apply to a wide variety of large-scale public venue wifi installations. Lots of great information in the post, and well worth a read.

At the 2016 WLPC there was an interesting TENTalk from Mike Liebovitz at Extreme Networks about the pop-up wifi at Super Bowl City in San Francisco, where analytics pointed to a significant portion of the traffic being headed to Apple.

Meanwhile, a few months later at the 2016 National Church IT Network conference, I heard a TENTalk about Apple’s MacOS Server, where I first heard about this incredibly useful feature (sadly, it wasn’t recorded, that I know of, so I can’t give credit…)

With most of the LPV installations I’ve worked on, I’ve found the typical client mix includes about 60% Apple devices (mostly iOS). For example, this is at a large church whose wireless network I installed. (Note that Windows machines make up less than 10% of the client mix on wifi!)

Client mix from Ruckus ZoneDirector

OK, So what?

This provides an opportunity to make the wifi experience even better for your (Apple-toting) guests. Whenever possible, as part of the “WiFi System” I will install an Apple Mac Mini loaded with MacOS Server. This allows me to turn on caching. This is not just plain old web caching like you would get with a proxy server such as Squid, but rather a cache for all things Apple. What does this do for your fruited guests? It speeds up the download of software distributed by Apple through the Internet. It caches all software and app updates, App Store purchases, iBook downloads, iTunes U downloads (apps and books purchases only), and Internet Recovery software that local Mac and iOS devices download.

Why is this of interest and importance? Let me give you an example: A few years ago, we were hosting a national Church IT Round Table conference at Resurrection on a day when Apple released major updates to MacOS, iOS, and their iWork suite. In addition to the 50 or so staff Mac machines on the network, there were another hundred or two Mac laptops and iThings among the conference attendees. The 200MB internet pipe melted almost instantly under the load of 250 devices each requesting 3-5GB of updates. That would have melted even a gigabit pipe, and probably given a 10Gbps pipe a solid run for its money (not to mention bogging down some of the uplinks on the internal network!. Having a caching server would have mitigated this. It didn’t do great things to the access points in the conference venue either, all of which were not only struggling for airtime, but also for backhaul.

Just by way of an example, Facebook updates their app every two weeks and its current incarnation (86.0, March 30, 2017) weighs in at 320MB (the previous one was about half that!), and its close pal Messenger clocked in at 261MB. Almost everyone has those to apps, so they’re going to find itself in your cache almost instantly, along with numerous other popular apps. Apple’s iWork suite apps and Microsoft Office apps all weigh in around 300-500MB apiece as well. This has potential to murder your network when you least expect it. (A few years back, the church where I was working hosted the national Church IT conference that happened to coincide with Apple’s release of OSX Mavericks, and a major iWork update for both iOS and MacOS. The conference Wi-Fi and the church’s 200Mbps WAN pipe melted under the onslaught of a couple hundred Apple devices belonging to the guest nerds and media staff dutifully downloading the updates.)

In any case, check out the network usage analytics from either your wireless controller or your firewall. If Apple.com is anywhere near the top of the list (or on it at all), you owe it to yourself and your guests to implement this type of solution.Network Statistics from Ubiquiti UniFi

The Technical Mumbo-Jumbo

Hardware

As mentioned previously, a Mac Mini will do the job nicely. If you’re looking to do this on the cheap, it will happily run on a 2011-vintage Mini (you can find used Mac Minis on Craigslist or eBay all day long for cheap), just make sure you add some extra RAM and a storage drive that doesn’t suck (the stock 5400rpm spinning disks on the pre-2012 era Mac Mini and iMacs were terrible.) Fortunately, 2.5″ SSDs are pretty cheap these days. Newer Minis will have SSD baked in already.

If you’re wanting to put the Mac Mini in the datacenter, you might want to consider using a Sonnet RackMac Mini (which is available on Amazon for about $139) and can hold one or two machines.

Sonnet RackMac Mini

You can also happily run this off of one of the 2008-era “cheese grater” Mac Pros that has beefier processing and storage (and also fits in a rack, albeit not in the svelte 1U space the Sonnet box uses). If you have money to burn, then by all means use the “trash can” Mac Pro (Sonnet also makes a rack chassis for that model!).

This is a great opportunity to re-purpose some of those Macs sitting on the shelf after your users have upgraded to something faster and shinier.

Naturally, if you’re running a REALLY big guest network, you’ll want to look at something beefy, or a small farm of them Minis with SSD storage (the MacOS Server caching system makes it quite easy to deploy multiple machines to support the caching.)

The Software

MacOS Server (Mac App Store, $19.99)

Since most of your iOS guests will have updates turned on, one of the first things an iOS device does when it sees a big fat internet pipe that isn’t from a cell tower is check for app updates. If you have lots of guests, you will need to fortify your network against the onslaught of app update requests that will inevitably hit whenever you have lots of guests in the building.

The way it works is this: When an Apple device makes a request to the CDN, Apple looks at the IP you’re coming from and says, “You have a local server on your LAN, get your content from there, here’s its IP.” The result being that your Apple users will get their updates and whatnot at LAN speeds without thrashing your WAN pipe every time anyone pushes out a fat update to an app or the OS, which is then consumed by several hundred people using your guest wifi over the course of a week. You’ve effectively just added an edge node to Apple’s CDN within your network.

Content will get cached the first time a client requests it, and it does not need to completely download to the cache before starting to send it to the client. For that first request, it will perform just as if they were downloading it directly from Apple’s servers. If your server starts running low on disk space, the cache server will purge older content that hasn’t been used recently in order to maintain at least 25GB of free disk space.

MacOS Caching Server Configuration

The configuration

If you have multiple subnets and multiple external IPs that you want to do this for, you can either do multiple caching servers (they can share cache between them), or you can configure the Mini to listen on multiple VLANs:

Mac OS network preferences panel

Once you have the machine listening on multiple VLANs, you can tell the caching server which ones to pay attention to, and which public IPs. The Mac itself only needs Internet access from one of those subnets.

MacOS Server Caching Preferences

The first dropdown will give you the option of “All Networks”, “Only Local Subnets”, and “Only Some Networks”. Choosing the last one opens an additional properties box that allows you to define those networks:

Mac OS Server Cache Network Settings

The second one gives you the options of “Matching this server’s network” or “On other networks”. As with the first options, an additional properties box is displayed.

In both cases, hit the plus sign to create a network object:

Mac OS Server Create a New Network

It should be noted here that this only tells the server about existing networks, but it won’t actually create them on the network interface. You’ll still need to do that through the system network preferences mentioned previously. If you don’t want to have the server listen on multiple VLANs, you can just make sure its address is routable from the subnets you wish to have the cache server available, define the external and internal networks it provides service to, and you should be off to the races. This will provide caching for subnet A that NATs to the internet via public IP A, and B to B, and so on. Defining a range of external IPs also has you covered if you use NAT pooling.

There’s also some DNS SRV trickery that may need to happen depending on your environment. There are some additional caveats if your DNS servers are Active Directory read-only domain controllers. This post elaborates on it.

 

Is it working?

Click the stats link near the top left of the server management window. At the bottom is a dropdown where you can see your cache stats. The red bar shows bytes served from the origin, and green shows from the cache. If you only have one server doing this, you won’t see any blue bars, which are for cache from peer servers. Downside is that you can only go back 7 days.

On this graph, 3/28 was when there were both a major MacOS and iOS update released, hence the huge spike from the origin servers on Apple’s CDN. Nobody has updated from the network yet… But guest traffic at this site is pretty light during the week. I’ll update the image early next week.

MacOS Server Cache Stats

Other useful features

A side benefit of this is that you can also use this to provide a network recovery boot image on the network, in case someone’s OS install ate itself – on the newer Macs with no optical drive, this boots a recovery image from the internet by default. This requires some additional configuration, and the instructions to set up NetInstall are readily available with a quick Google search.

If you want, you can also make this machine the DHCP and local DNS server for your guest network. With some third-party applications, you can also serve up AirPrint to your wireless guests if they need it.

Conclusion

From a guest experience perspective, your guests see their updates downloading really fast and think your WiFi is awesome, and it’s shockingly easy to set up (the longest and most difficult part is probably the actual acquisition of the Mac Mini) It will even cache iCloud data (and encrypts it in the cache storage so nobody’s data is exposed). Even if you have a fat internet pipe, you should really consider doing this, as the transfers at LAN speed will reduce the amount of airtime consumed on the wireless and the overall load on your wireless network. (Side note, if you’re a Wireless ISP, this sort of setup is just the sort of thing you ought to put between your customer edge network and your IP transit)

Of course, you could also firewall off Apple iCloud and Updates instead, but why would you do that to your guests? Are you punishing them for something?

Android/Windows users: So sad, Google and Microsoft don’t give you this option (Although Microsoft sort of does in a corporate environment with WSUS, but it’s not nearly as easy to pull off, nor is it set up for casual and transient users). I would love it if Google would set up something like this for play store, Chromebook, etc, as about half of the client mix that isn’t from Apple is running on Android. You can sort of do it by installing a transparent proxy like squid.

Now, if only we could do the same for Netflix’s CDN. The bandwidth savings would be immense.

Update

(Added November 16, 2017)

As of the release of MacOS High Sierra and MacOS Server 5.4 (release notes), the caching service is now integrated into the core of MacOS, so any Mac on the network can do it, without even needing to install Server. The new settings are under System Preferences > Sharing:

 

 

Controlling Audio With ProPresenter

Our church is a small one. So its not always especially easy to fully staff our tech booth, and sometimes, one must fly solo, which adds to the workload, and sometimes stuff gets forgotten, like unmuting microphones for the choir or the person reading the scripture.

Fortunately, there is some tech than can help us in this regard. We use ProPresenter for our graphics presentation, and an Allen & Heath QU-24 console for our audio. The Qu-24 is connected to the Mac that runs ProPresenter via a USB cable, which shows up in the Mac as a 32 in/32 out audio device, as well as a MIDI device. This is primarily to be able to use the console as a multitrack and DAW interface, but it also lets us play back audio from ProPresenter media cues without ever leaving the digital domain, and saving us a couple of inputs on the board (although there’s no shortage of those). But because it’s also a MIDI device, this gives us some options with ProPresenter’s $99 MIDI module add-on. The Qu series boards can also do MIDI over IP (in fact, the Qu-Pad remote control app for iPad uses MIDI over IP to work its magic). If you’re using MIDI over IP with a Mac, you’ll need a special driver for the Mac. No driver is needed for USB.

First, a few resources we’ll need:

In the Qu Series, mutes and mute groups are controlled by a sequence of a Note On/Off message. The specific note determines the channel or mute group being controlled, and a the velocity value determines if it’s being turned on (Muted) or off (Unmuted). Velocity values below 64 turn the mute off, and above turn it on.

Meanwhile, over in ProPresenter, since Version 6, we have the ability to add MIDI Note On/Off cues to a slide. See where this is going? Unfortunately, ProPresenter doesn’t have the ability to do anything other than MIDI notes in a slide at the moment, so we can’t get really crazy with starting recordings or anything else requiring non-note MIDI messages.

So how do we know what notes emulate button presses? The documentation provides this handy method:

OK, this requires thinking and math. Not so helpful. This is where the MIDI monitor comes in. Download it and run it, and it shows everything coming across the MIDI interface. Push the button you’re interested in, and lo, MIDI Monitor helpfully shows you what note you’re interested in:

In this case, G#4 is the mute group for our choir. A4 is the mute group for the speaking mics on the chancel. A1 is the lectern mic.

Screenshot 2016-11-20 13.51.30So now, to be able to add a cue at the beginning of a song the choir is singing, I simply have to add two cues to the first slide to turn on the choir microphones:

  • NOTE ON, G#4(80), 63
  • NOTE OFF, G#4(80)

Then I can add a slide at the end of the playlist entry that then turns them back off, or add these to the beginning of the next playlist entry:

  • NOTE ON, G#4(80), 127
  • NOTE OFF, G#4(80)

Likewise, when someone is at the lectern reading scripture, I can unmute that channel automatically using the corresponding note number, and mute them again when they’re done.

On the flip side, you can also use note on/off commands to control ProPresenter. So you *could* also use the Mute, SEL, and PAFL buttons on unused channels to trigger things in ProPresenter (you also want to make sure that you don’t overlap these with the mutes and mute groups that you are actively using so as not to inadvertently advance a slide when hurriedly muting a channel). ProPresenter also conveniently tells you what the last note sent was, so you can actively push the button you want to use, make a note of its number, and put it in the action you wish.

 

Another approach you can take is to create a presentation in ProPresenter containing blank slides with the various functions you wish to use. Then you can copy these slides into presentations and add a Go To Next timer to them to automatically advance to the next slide. I would also recommend using slide labels and colors to clearly identify what each slide is doing:

Screenshot 2016-11-20 13.47.55

 

If you have controllable lighting and your lighting console also has MIDI capability, This comes in handy as well. And if you’re really a one-man band, and like to do things like pads underneath certain worship elements, you can use this to trigger those as well. But if you get to that point, you may want to look into QLab to control all of them at the same time.

So there you have it: a quick and easy way to automate some of your workload with the Qu series boards. If you’ve got another board that you use, let me know in the comments if you do (or would like to do) something like this. Would also love to hear if anyone is using hardware MIDI controllers like the Novation LaunchPad and how you have it set up.

Additional Info:

Summary of MIDI Messages (midi.org)

In the wild: EGO cordless electric mower

It’s a clean, green, mowing machine!

mower_largeI’m going to veer off my usual topics here to give you my thoughts on a recently acquired tech toy of a different flavor: My lawnmower. This is NOT a sponsored post.

When we moved into our previous house almost 11 years ago, we went on a coupon-fueled shopping spree at Home Depot, and picked up your standard 4-stroke gas-powered mower. It worked well, but after a while, the tedium of dealing with oil and gas and all those moving parts makes you think “there’s got to be a better way”. We’d been eyeing electric mowers for a while, but either they were corded (obnoxious – chance of mowing the cord is high!), or had enough battery life to make you need a wagon full of spare batteries, and in neither case did they have enough oomph to cut grass that had gone more than about 12 hours since the last mowing.

When the Toro died this spring (I think it busted a rod, or something else pretty major in the engine), I started looking at battery-powered options again. At our new place, the lot is nearly half an acre. Anyone in their right mind would have bought a small riding mower. Consumer Reports narrowed me down to two options: The Black & Decker CM1936, with a 19″ deck and self-propelled version for $439 at Amazon (now $379), or the EGO LM2001, with a 20″ deck for $499 at Home Depot. I was initially leaning toward the B&D’s lower price, but was eventually won over by the EGO’s slightly wider cutting deck, its 5-year warranty, and the 30-minute charger (which would have been another $130 for the B&D). That the EGO is 50 pounds lighter was a big plus as well.

Since Home Depot had it in stock, I headed over there and picked it up. It comes in a large cardboard box that easily fit in the back seat of my full-size Toyota. Because the mower handle folds down and collapses, there was no assembly to speak of. It’s pretty much a matter of taking it out of the box, removing the requisite bits of tape and protective film, and a few plastic bags, and putting the box on the curb for the recyclers to pick up. Virtually all of the packaging is recyclable, which is a plus.

EGO Battery Pack

This is alien technology. Or at least from the future.

The battery pack and the charger look like something right out of a sci-fi movie. The battery is the heart of the system, and battery technology has made significant progress in the last few years. Power tool manufacturers love to tout the voltage of their battery system, under the idea that “More Volts = Better”. Being a geek with a background in electronics, I know that this is mostly bunk, but when it comes to battery systems, there’s some validity to it, because virtually all power tool batteries consist of a serial/parallel array of 1.2V rechargeable cells. More cells = more oomph. EGO says the motor in this mower is 600 watts, which works out to a little over 10 amps. The battery pack is 4Ah, so at full load, one should expect about 25 minutes of use. Since in the real world, the motor isn’t under full load the whole time, it gets pretty close to EGO’s claimed run time of about 45 minutes. I’ve found that this isn’t enough to do my whole yard, but after the battery runs out, I’m usually ready to go sit inside for half an hour, and cool off with a cold beverage and some air conditioning, while the battery charges back up (and yes, it DOES only take half an hour!).

Performance-wise, it does OK with normal mowing, but heavy grass is something best approached in phases at different cutting heights (which are adjustable to 5 levels with a single lever somewhat reminiscent of the shifter in my minivan). On my gas mower, I’d usually deal with heavy grass by putting the discharge chute on and letting it eject the cuttings rather than mulch them. On the EGO mower, this plan is no good. While it comes with a discharge chute that attaches in the same place as the bag, it clogs easily, and is generally useless. Bagging works pretty well, though.

So easy a kid can mow!

Where this mower really shines is how easy it is to use. Much of the body is molded polypropylene, so at 40-odd pounds, it weighs about half what my old gas mower did (the B&D unit is actually 15 pounds HEAVIER than my Toro), and is so quiet that it won’t bother the neighbors. When sitting inside while someone is mowing, it sounds like a gas mower several blocks away. A typical gas mower is usually over 90dB, where permissible exposure levels are not much more than an hour. I don’t know offhand what this unit is, but it’s a LOT quieter. There’s no fuel to mess with, or oil changes, or any of that. When you’re done, it folds up neatly and can be stood on end, taking up no more than 2 square feet of your garage. Between the light weight, the quiet, and the lack of fumes, my 10-year-old daughter is actually willing to mow the lawn. (Another major motivating factor is that mowing the grass pays for her cell phone service).

Other than keeping it clean, there’s really no long-term maintenance to worry about. It comes with a 5-year warranty, although the blade isn’t covered, as it’s considered a wear part. There are no moving parts other than the motor itself (compare to a 4-stroke gasoline engine, which even with a single cylinder is a very complex piece of machinery).

As for energy consumption, a full charge is 224 watts of electricity. A full charge will run you somewhere between 2 and 4 cents worth of electricity, depending on where you live. The EPA defines a gallon of gasoline as equivalent to about 33.4kW of electrical energy. My old Toro would go through about a quart of gas to mow the yard. At nearly 4 bucks a gallon for the non-ethanol stuff that won’t wreck the engine, that’s a buck a mow. With 2 charges, that’s also about 20x the amount of energy that the EGO uses to achieve the same job. Oddly enough, the “fuel” cost of the electric mower is also about 1/20 that of the gasoline.

EGO in storage mode

If carbon footprint is something you care about, 1 full mow is about half a pound of CO2 if your electricity is from natural gas, and about a full pound if it’s from coal. Zero if it’s from nuclear, solar, or wind. The gasoline mower belches out about 4.5 pounds per mow, along with a whole bunch of other nasty stuff that your car has the decency to clean up first.

As for quirks, there are a couple. The handle has a couple of different safety interlocks, and must be fully extended and locked for the mower to operate. There’s a dead-man switch on the push bar, much like virtually any other mower. There’s also a removable safety key that must be pushed in to start the mower. I’ve found that when mowing close to bushes that the slide lock tends to come undone, allowing the handle to slide in just enough to cut the mower out. The first few times, you find yourself standing there wondering why the mower won’t work, until you notice the bright green latch on the handle hanging open.

When going through heavy grass, if the current draw on the motor becomes too much, the green power light will start flashing yellow. If you don’t ease up on it, it will stop the mower. Once the battery gets low, the indicator light will turn red, at which point you’ve got about 3-5 minutes until break time. The battery is really good at delivering a fairly flat and constant amount of power, so there’s not really much decline in power until the battery decides to go completely dead.

EGO also has a blower, edger, and hedge trimmer that work with the same battery (they also make a smaller and lighter 2Ah battery for those devices, but all batteries in the EGO tool family are interchangeable, so if you find yourself about 10 minutes short on the mower, get the trimmer or blower with its battery, run the mower on the small battery for 20 minutes, and then switch to the big battery to finish the lawn while you recharge the small battery, which will be ready for trimming or blowing by the time you get done mowing. If you already have batteries and chargers, you can order just the tools without batteries directly from EGO for less money. Similarly, you can order extra batteries from them (the big one is $199, the little one is $129).

Bottom line, It may be a spendy piece of equipment, but not having to deal with gasoline, fumes, noise, maintenance, and being able to send the kid out to mow instead of doing it myself is well worth the price of admission. Oh, and it also has an LED headlight, for those midnight mowing escapades.

My God, It’s Full Of Pixels!

One of my desktop monitors gave it up last week, the backlight started flickering and randomly disconnecting from my Mac (which causes all attached displays to go dark for a second while it recomputes your workspace). Needless to say, it was sapping productivity even worse than Facebook. I’ve been eyeing 4K monitors for a while now, and both Dell and Lenovo have some decent low-cost options in 28″ sizes ($700ish), but that was still more than I was willing to cough up for a new monitor right now, even with the Lenovo dealer demo discount.

Then came last weekend, where TigerDirect ran a $70 rebate on a Seiki SE39UY04 39″ (yes, you read that right, thirty-nine inches) 4K TV with a whole mess of inputs (3 HDMI, 1 VGA, 1 Component, as well as a tuner). Base price before rebate: $400. Four hundred bucks. BEFORE the rebate. For 4K. That brings it down into the realm of even inexpensive 27″ monitors. Some googling found that the tech press actually has good things to say about this “off-brand” display (with the caveat that there are very few graphics chips currently out there that can drive this resolution, and that HDMI 1.4 is limited to 30Hz refresh at 4K). Since the rebate was a very limited-time offer, I jumped on it, figuring I was gonna have spend that much on a monitor anyway. For those who missed last weekend’s rebate from TD, It’s currently available on Amazon (with Prime!) for $299 (plus larger sizes too!).

Three days later, UPS shows up bearing goodies, and I hooked it up this morning. The unit is generally well built, comes with a solid but unobtrusive pedestal, and the bezel is not huge. It even comes with a decent HDMI cable. There’s been a lot of discussion online as to whether the first-generation Retina MacBook Pro can drive this monster at native resolutions at all. Let me put those to rest: it can, with Mavericks 10.9.3, at 30Hz. Because of the refresh rate, it’s not a great rig for gaming (but it’s still beautiful with X-Plane!!!), but it’s great for sheer pixel space to put my calendar, e-mail, a couple of network monitoring screens, multiple RDP sessions, and lots of other things that don’t require high refresh rates. Colors are quite good, as is brightness, and the built-in speakers are surprisingly loud (almost too loud to use with my computer even at the lowest volume setting) If you press “Menu” on the remote, followed by “0 0 0 0”, you can get into the factory menu which allows you to tweak the color temperature, and the “Warm” setting is shockingly close to my MacBook. The factory menu also lets you dial down the backlight (which I did – even so, this TV is already near the bottom of its category when it comes to energy consumption). I also dialed the default sharpness setting down to 0, as, like most TVs, the edge enhancement algorithms designed for making TV pictures look better really butcher computer signals.

Tweetdeck is 1440x900, RDP window is 1920x1080.  No scaling. Just pixels.

Tweetdeck is 1440×900, RDP window is 1920×1080. No scaling. Just pixels.

Did I mention that 4K is an awful lot of pixels? It is a LOT of pixels. 8,294,400 of them . Holy cow. I’m a lifelong pixel junkie, and I’m loving this. The 39″ display is big enough to use on your desktop at native resolution at a comfortable distance of about 4′ (rather than driving it at a pixel-doubled 1920×1080 workspace). Drawing network maps in Visio and Ekahau at 4K resolution is something out of a dream.

Visio at 3840x2160.

Visio at 3840×2160.

Ekahau Site Survey

Ekahau Site Survey

My biggest problem? Losing my mouse cursor. Gotta use a solid color background. That’s OK, my GPU is probably just as glad it doesn’t have to deal with an 8MP image.

X-Plane in its full 39" 4K Glory

X-Plane in its full 39″ 4K Glory

Bottom line, for under $400, this is a surprisingly good piece of hardware. Seiki may be considered an “off-brand” label, but don’t forget that Vizio was in the same position when they started selling good HD televisions for dirt cheap. The only real downside I see right now is using a VGA port that is limited to 1920×1080 instead of putting a DisplayPort interface there instead. OK, Roku, When can I expect a 4K version of your box so I can watch Breaking Bad and House of Cards in their full 4K glory? Et tu, Chromecast?

This video looks utterly spectacular on this screen (Downloaded with YTD and played with QuickTime)