Screenshot of Madison’s town-wide garage sale map mashup

I built the town’s extensible content management system starting in 2008, working closely with Jim Sanderson, Madison’s technology directory. The history of “RoseNet” (named for Madison’s “Rose City” moniker), one of the first community websites, dates back to 1997. It has gradually grown to include more and more content and interactive features. I haven’t seen any other municipal websites that come close in terms of quantity and quality of data. For example, Jim has invited all local businesses and non-profit organizations to have a self-maintained presence on the website, and this cool business map is a fun way to promote Madison’s downtown merchants. The local business listings provide a helpful service to residents and help boost revenue by promoting the business district in the face of competition from internet and big box retailers. There is, of course, room for improvement (we’re constantly debating the home page and navigation), but the site has proven to be a very effective communication medium for the town.

The garage sale registration process highlights a flexible workflow capability that will help the municipal government streamline many forms and processes. Plus, thanks to years of careful design and execution, Jim and I were able to deploy this feature with minimal cost and lead time.

I think we’ve really stumbled onto a great architecture for a civic website. I’d love to turn the design into an open source project. Recently, I’ve heard a lot about civic code volunteering and startups (like the Code for America project), and I think that’s really great. Even though Madison is a small town, I think we’re out in front in terms of designing an effective municipal content management system. Drop me a line [Twitter, Google Plus, or just leave a comment below] if you’re interested in collaborating.

I’ve heard older NiCd battery technology should be deep cycled to avoid the old “memory effect”, but our phones, tablets and laptops all have newer lithium-based batteries. The conventional knowledge seems to be that completely discharging a lithium battery permanently degrades its lifespan. But I haven’t found a definitive source on how to best care for my battery.

As an iPhone user, I thought I’d check with Apple first. For maximizing battery lifespan, their website suggests keeping your phone at the right temperature and adjusting your settings to use your battery less. They don’t offer any specific recommendation on deep vs. shallow charge cycling, but they do suggest going through at least one complete charge cycle (charging the battery to 100% and then completely running it down) per month. That seemed like an oddly specific recommendation with no data to back it up.

Checking the Literature

I decided to see what the scientific literature had to say about this. Here’s what I learned:

Measuring Battery Performance: When evaluating any kind of performance, engineers need a metric. The standard lifespan metric seems to be “number of charging cycles through which the battery can retain 80% of its original capacity”. For example, Apple promises 400 charging cycles for your iPhone battery.

Temperature: I found an interesting paper (behind a paywall, unfortunately) on the aging mechanisms of lithium ion batteries. Sure enough, there’s good science verifying that temperature plays a big role in battery lifespan. Most batteries are designed to operate at room temperature, and both hot and cold temperatures accelerate aging. They also note that battery capacity will fade over time regardless of your usage habits because the chemicals simply break down.

Shallow cycling: The best research I could find comparing deep and shallow cycling was Nicholas Williard’s master’s thesis from the University of Maryland. He experimented with a particular make and model of lithium ion batteries. He preconditioned a fraction of these batteries by shallow cycling them 900 times (draining just 7% of charge before recharging). Amazingly, he found the preconditioned (i.e., shallow-cycled) batteries were able to withstand about the same number of deep cycles as brand new batteries. In other words, shallow cycling didn’t seem to degrade battery capacity at all!

My conclusions:

• Keep your battery cool (but not cold).
• Shallow cycling is OK (go ahead and charge every night… or even more often).
• I couldn’t find any research backing up Apple’s “monthly deep cycle” recommendation.

Not surprisingly, iPhone chargers don’t handle 15-amp short circuits very well.

The metal shielding of the 30-pin connector completed shorted the 120 V A/C power prongs. My nephew reports seeing a large spark, hearing a pop, and then witnessing a bit of smoke and funny smell. Fortunately, the 15-amp circuit breaker tripped almost immediately, preventing him from getting shocked and probably stopping a house fire.

In the picture below, you can see the power prongs melted a bit, and there’s a coating of black soot all over the charger (the wall was dirty, too, I’m told).

Lesson learned? We sure hope so!

How to Parse Plays to CSV

If you don’t need the data in a SQL database, you can quickly parse the plays to a CSV file (comma-separated values). You could then open this file in a spreadsheet application or read it into the scripting environment of your choice (R, MATLAB, Python, etc.).

I created a specific branch of the repository that will parse the plays without any dependencies on database libraries (i.e. SQLAlchemy). Assuming you’re on Linux and have Git installed, just clone the “nodatabase” branch of the repository to get started like this:

$ git clone --branch=nodatabase https://github.com/10flow/playbyplay.git

Then you can run the parser script:

$ cd playbyplay/play-parser/
$ ./play_parser.py

After a minute or two of processing, you’ll have a “output.csv” ready to play with.

How to Ingest Plays into a Database

If you want to use a SQL database, I’ll assume you have one installed. If not, I’ll refer you here for Ubuntu’s guide to MySQL.

My ingest script requires SQLAlchemy, a popular database library for Python. This is easy to install with the “easy_install” package manager. If you don’t have easy install, you can install it on Ubuntu as follows: “sudo apt-get install python-setuptools”. Next, get SQLAlchemy like this:

$ sudo easy_install SQLAlchemy

Now you’re ready to download the code from GitHub. Grab the master branch if you’re planning on using a database:

$ git clone https://github.com/10flow/playbyplay.git

Before you run the parser script, there’s a few things to do. First, specify your database information in “play_parser.py” on line 28. And next, you’ll need to create the database before you can insert the data. For example, if your database connection string (line 28) looks like this:

database_engine = create_engine('mysql+mysqldb://root:password@localhost/playtest', echo=False)

Then create the database like this:

$ mysql -u root -p
mysql> create database playtest;
Query OK, 1 row affected (0.06 sec)
mysql> exit

Now you’re ready to run the script:

$ cd playbyplay/play-parser/
$ ./play_parser.py

Inserting the plays will take a long time. But once it’s done, you can issue queries this:

$ mysql -u root -p
mysql> use playtest;

Database changed
mysql> select result,count(result) from plays where season=2012 and result!='play' group by result;
+--------+---------------+
| result | count(result) |
+--------+---------------+
| downs  |           209 |
| fg     |           855 |
| fumble |           184 |
| half   |           185 |
| int    |           378 |
| loss   |            61 |
| miss   |           193 |
| ot     |            14 |
| other  |           154 |
| punt   |          2480 |
| td     |          1269 |
| tie    |             2 |
| win    |           148 |
+--------+---------------+
13 rows in set (0.02 sec)

And now you’re ready to analyze.

Let me know if you come with any interesting stats in time for the big game!

Here are some interesting stats I’ve found so far:

The stats show Andy Reid’s Eagles performed poorly when down a score in the final 2 minutes. The biggest problem seems to be the terrible 34% pass success rate and a below average plays-per-drive (4.4 compared to 4.7). Two-minute drill drives ended in 11 picks (21.2%), 12 turnovers on downs (23.1%), and 10 TDs (19.2%). They ran out of the clock 9 times. Their scoring percentage was 25% compared to a league average of 33.4%. Good luck Kansas City!

The 2012 Jets stand out as a weak offense. There was a big drop in touchdowns, pass play effectiveness, and interestingly, pass play selection. The Jets passed on 57.1% of plays in 2011, compared to 52.1% this year. The loss of CB Darrelle Revis also had an impact. Opposing team’s pass success rate went from 40.8% (with 15 picks) in 2011 to 42.8% (with 9 picks) in 2012.

Finally, the Pats offense was scary-effective this year. Their touchdown rate stands out the most — 32.4% compared to league average 21.0%.

Leave a reply below or Tweet at me if you find something interesting in the stats!

This seems like a gold mine for evaluating the overall effectiveness of offenses and defenses in different situations. The data is great, but you can’t do much with the data unless you can parse the text description. Here are some example descriptions:

• (Shotgun) E.Manning pass incomplete short middle to V.Cruz.
• (No Huddle) L.McCoy left tackle to CLV 43 for 7 yards (T.Ward).
• (Shotgun) T.Brady pass deep middle to W.Welker to DEN 1 for 19 yards (C.Harris). Touchdown was reviewed by Review Assistant. And overturned.

There’s a quite a bit you can do with this data as a spreadsheet. But when it comes to parsing these descriptions, a spreadsheet is simply not the right tool. After a few hours and a couple hundred lines of Python, I’ve parsed each description, filtered out non-offensive plays (i.e. kicks, penalties, and weird situations), and then determined the type of each play (pass or run) and the yardage gained or lost. I then rate each play as a success or failure. A successful play meets one of these criteria:

• Play results in a first down
• Play scores a touchdown
• On 1st or 2nd down, play gains at least 4 yards

And, of course, successful plays can’t result in turnovers or losing the game.

I also determined how each drive ended, and then I ingested the entire dataset in a MySQL database. SQL is a great tool for filtering the plays based on specific values or ranges for any of the columns.

I don’t claim 100% of the 350,000+ play descriptions were parsed accurately, but based on some spot checking, I’d guess accuracy is in the high 90s. Football is a game with strange rules, and it would probably take thousands of lines of code to nail down every weird situation involving penalties, official review, blocked kicks, and multiple fumbles.

Regardless, with this mostly accurate database available, it’s now possible to quickly generate stats and even make the data accessible as a web application. I hope to roll out a site in the next couple days.

Here’s a sneak preview of the run stats visualization:

Histograms show run play success

Stay tuned and let me know what great ideas you have for the web app!

Let’s try it out in a Rails console:

>> names = %w{Alice Bob Charlie David Eve}
=> ["Alice", "Bob", "Charlie", "David", "Eve"]
>> names.dup.shuffle.each_with_object({}) {|n, h| h[n] = names.find {|m| m != n}; names.delete(h[n])}
=> {"Bob"=>"David", "David"=>"Alice", "Alice"=>"Bob", "Charlie"=>"Eve", "Eve"=>"Charlie"}

Someone should really turn this into a webapp.

Happy Holidays!

The attendees were very engaged, and I got lots of great questions. When we got into wavelet-based JPEG2000 compression, there was one question in particular that I couldn’t quite answer at the time: if wavelet transforms are like digital FIR filters, how can they be reversible to provide lossless compression?

It’s a great question. The wavelet transform is based on tree-like cascading filter banks, where the signal is sent through a low-pass and high-pass filter at each stage, as shown here:

Wavelets are like a tree of cascading filter banks
Source: http://en.wikipedia.org/wiki/Discrete_wavelet_transform

In the diagram above, the filter denoted g[n] represents a low-pass filter, and h[n] a high-pass filter. In lossless image compression, the wavelet transform is carried on for a particular number of stages, and the resulting 2-D signal is compressed using entropy coding (which is lossless). In order for the original image to be recovered, the wavelet transform, and by association the filters g[n] and h[n], must be reversible.

In general, a digital Finite Impulse Response (FIR) filter is not reversible. Even filter banks, which pass a single input signal through a number of FIR filters, are generally not reversible. However, the Cohen-Daubechies-Feauveau (CDF) 5/3 wavelet employed by JPEG2000 in lossless mode has a couple interesting properties of the wavelet transform used in JPEG2000 lossless compression that make the filters reversible.

First, the CDF 5/3 wavelet is designed to be an integer-to-integer map, such that the output can be expressed with finite precision. This is achieved in part by approximating the filter coefficients with rational numbers—in fact, each coeffient is a dyadic fraction (i.e. fraction whose denominator is a power of two). Second, the low-pass and high-pass coefficients are designed such that the two filter outputs can be combined to recover the original input. In other words, the two filters are complementary. Actually, the two sub-band signals are generated using a lifting scheme, as described here.

In short, to call the wavelet filters high-pass and low-pass FIR filters is an over-simplification. The CDF 5/3 wavelet is a carefully designed reversible transform that makes it well-suited to lossless image compression.

(Wavelets are a dense, complex topic. If you’re interested in really understanding them, I would suggest Wavelets and Filter Banks by Strang and Nguyen. There are a host of other books available, but I’ve found their book to be the most readable.)

I like lectures that tell a story, so I’m approaching the topic chronologically. I got into computers as a kid in the 90s, and back then, web graphics were all about GIFs and JPEGs. These formats are based on very different compression techniques, and it was (and still is) important to choose the right one based on your image content. Beyond the web, many engineering applications require lossless photo compression, and JPEG2000 has long been the de facto standard. More recently, new vectorized formats (Scalable Vector Graphics, in particular) have taken off on the web, largely driven by devices with very high display resolution, such as Apple’s retina display.

With that narrative in mind, here’s my outline of the most important 60 minutes in Image Coding and Compression:

• How color works
• Simple image coding (e.g. run length encoding)
• LZW compression (GIF)
• Compression with the discrete cosine transform (JPEG)
• Compression with wavelet transforms (JPEG2000)
• New graphics formats (e.g. SVG)

Of course, there are a lot of interesting topics that didn’t make the cut (most notably, perhaps, predictive coding). What are your thoughts?

This map is certainly not complete, but it provides a good background on what’s out there. I hope to continue growing this map as I learn about new technologies. I’ve also noted a few topics that look like strong potential research areas.

Check it out here: Scott Sawyer’s mind map of Big Data platforms (or download the FreeMind file).

Leave a reply or tweet at me (@10flow) if you’ve got a technology to share!