Stephen Heise

Blogger’s Note: In response to reader demand, I have decided to make this animation a regular feature. The next installment will coincide with the May 22 data release.

Homes in The West from Washington state to California to Texas continue to sell at prices well above the long-term inflation-adjusted trend. There is the smallest of hints that New Hampshire, New York, and Pennsylvania are beginning to cool. Ohio and Indiana appear to be holding ground as their neighbor Michigan continues to have woes.

To show the trends I’ve produced an inflation adjusted heat map reflecting today’s OFHEO data release. The methodologies are the same as the original heatmap.

Red means that homes in the state are selling for prices above the historical price trend and blue indicates that homes are selling for prices below the area’s long-term trend based on same-home sales data. The movie starts in 1975 and runs through December 2007.

For more on how I did it, please refer to my detailed original post. The tab-delimited data set I used to construct this animation is here.

As a way of whetting your appetite — with just fifteen days to go until the next OFHEO House Price Index data release — I thought it would be neat to compile an Overprice Index using the same data set I used to generate the popular U.S. house price heat map animation.

Based on gov’t historical local growth rate data for homes in each metropolitan statistical area (MSA) going back to 1975, this table ranks each surveyed city from most overvalued to most undervalued. Assuming that the inflation-adjusted same-house sale price growth rate trend (rightmost number) continues, this table lets you see at a glance how hot (or not) a particular metro area is and how much of a home price premium you’d typically pay (or discount you’d get) if you bought a home there today.

Inflation adjusted analysis of mainland U.S. same home sales shows a cooling across Michigan and Massachusetts. For your enjoyment I have generated an animated heat map indicating the states that are burning above (red) and below (blue) the historical price trend for that area. The chart runs from 1975 through September 2007.

For every region I have done a regression analysis to determine the long-term historical growth rate. When local prices are way above sustainable long-term averages for that metropolitan area they contribute to a state being colored bright red. When home prices have fallen to be far below what they should be, they contribute to their state being colored deep blue.

Here’s how I did it. I built a mathematical model around the notion that a home price is dependent on volatility within a larger framework of inflation and regional factors such as population and income. To simplify the model I decided to reduce the regional factors into a single growth rate and set out to calculate the long term growth rate for every region.

Of course I needed data. I really liked that the U.S government’s Office of Federal Housing Enterprise Oversight (OFHEO) calculates a quarterly index based on same home sales. In order to calculate their trends they rely entirely on those homes that have sold two or more times. If there are two houses on the same street and one sold in 1980 for $100,000 and again in 1985 for the same price ($100,000) but a nearby house sold for $100,000 in 1980 and sold for $105,000 in 1986 then it can be inferred for that area that home prices did not change between 1980 and 1985 but increased 5% between 1985 and 1986. Clearly having access to lots of data helps establish trends, and that’s exactly what the OFHEO does.

Next I needed to remove inflation from the picture. To do that I chose to use the CPI data published by the Federal Reserve Bank of St. Louis to adjust prices into real dollars.

Then I logged the data and ran a least squared linear regression using the perl Regression.pm package off of CPAN using a constant weight for every data point. For each data point I then calculated the ratio of the prices relative to the trend line.

This was great. I had a massive spreadsheet and could look at any raw number I wanted. After a while of that I decided, “Hey, wouldn’t it be great to plot these trends?” Which I did. But it turned out that viewing a tangle of hundreds of metropolitan areas amounted to viewing a spaghetti of multicolored wiggles. Nonetheless I plowed ahead and graphed the discrete first derivative of every metropolitan area and noticed an unexpected smoothness in the data. So I analyzed the discrete second derivative and said to myself, “Wow, the time-variant trends are so predictable that this would make a great macroeconomic animation!!!”

But before I could make an animation I had to create even a single frame. I looked for charting software and found this amazingly helpful tutorial with sample code from IBM (Thank you IBM!!) that relies on the Cooperative Association for Internet Data Analysis (CAIDA) plot-latlong software (Thank you UCSD and NSF!!) to plot data onto maps.

I then annotated and animated the frames using ImageMagick convert. At this point I had an animated GIF, but I really wanted to let user move forward and back through the frames. So I converted the animation from GIF to FLA format using FFmpeg because that’s how all the cool cats like YouTube do it.

It was then I realized that I needed a bit more glue to make it all work. I researched for a couple hours and in the end decided to use the same javascript code as YouTube: Geoff Stearns’ SWFObject to play embedded shockwave files. To finish off what had started as a little project, I installed Jeroen Wijering’s FLV Media Player to provide the tidy controls and way cool full screen functionality.

All in all it was a nice “all nighter” project. I hope you find the animation educational and perhaps also a few coders out there can learn from some of the inside tricks on how I did it.