Lost Art of Computer Programming

Lifestyle Routing

Even with the best of maps and instruments, we can never fully chart our journeys.
– Gail Pool

The questions worth asking have not answers worth knowing, but rather they engender journeys worth taking.
– Curt Brune

This article describes the motivations behind and the inner workings of a "IP address to Google Map" mash-up application I wrote recently called Geo IP Route. Read on!.

What should I do?
Where should I go?
Am I sure this is the right direction?
What if I had chosen a different path?

Most of us have asked ourselves these types of questions at one time or another. It's only natural to ponder the meaning of our existence and wonder aloud at the complexity of the universe.

Like all good questions these questions apply to many different levels of our existance, from superficial daily decisions to subdermal compromises of family and career to the innermost vexing decisions regarding ethics and morality.

Off hand the questions arising from routine daily decision making are of little interest to me. "Don't sweat the small stuff," someone once said. Good advice as life is too short. Make decisions, take action and keep moving forward.

No, I find the subtle, difficult questions much more intriguing. As these decisions lie closer to our core being, the effects of minor course corrections dramatically influence the route we travel. While it's true destinations help focus your energy, upon arrival the journey gives more insight to where you are then the destination itself.

I would like to share with you one of my recent existential programming journeys. Along the way I ponder questions, fiddle with traceroute and geocoding. As I've written previously the best way to get going to is to have a project to work on. With the project in hand we can sharpen the saw and get to work.

Here's a snapshot of the final application:

Read on for more ...

Charting Your Route

I wanted to do something new, something new to me and to you. Also I wanted to do something different from my day job (embedded system design and programming). For the something different I wanted to learn how to make web pages with interactive maps using the Google Maps API.

If you have been following along you know I have a history of working with GIS and digital map applications. I had never played with online map applications though.

Now the something new to me and you was going to be difficult. Mapping APIs are great, but what are you going to put on the map? To put items on a map you first need their latitude and longitude, otherwise known as geocoding. Where could I get geocodes of something interesting? For free of course.

After much head scratching I decided it would be cool to map the Internet routers used for various transactions. If you are familiar with traceroute you know exactly what I was thinking.

I wanted to make sure what I was doing didn't already exist. I did a Google search for "traceroute geocoding" – much to my dismay several sites came up. Only one seemed relevant, in fact it was pretty much exactly what I was thinking of doing. Somewhat long faced I clicked on the link to check out the application. Much to my delight the application was non-existant, the link provided didn't work. I was still in business.

This project had several challenges that had to be surmounted in order to succeed.

The first trick was to convert the Internet Protocol (IP) addresses into latitude and longitude coordinates. I had a couple of ideas on how to do this.

The second trick was learning the Google Maps API and various bits of javascript to glue it all together. While this seemed straightforward at first plenty of surprises were lurking to insure an eventful journey.

With destination in hand, the navigation charts arrayed before me I was ready to embark on the journey.

IPv4 to Latitude and Longitude

How to do this? Companies sell databases that map IP addresses to latitude and longitude, along with a subscription service to keep the database up to date. That would likely be the mostly accurate way to go, but that would take all the fun out of it. Plus I don't have money to pay for it.

I harkened back to my experience which traditional geocoding, where the latitude/longitude (lat/long) is derived from a street address. Maybe a I could transform IP addresses to street addresses and then to lat/long. How could I get street addresses from IP addresses?

Given that this was a for-fun-not-for-profit project 100% accurracy was not required, so I decided to use the whois database to look up the administrative contact for given IP addresses. This was not the final solution I used for the finished application, but it was a start. Starting is key to any journey!

Clearly using whois was not going to tell me the exact location of the server in question, but it would tell me the location of the organization that ran the server. Good enough.

The following is the abbreviated whois output for LinuxDevices.com. First I looked up the IP address for www.linuxdevices.com and then fed that IP address into the whois command, using the whois.arin.net whois sever.

linux:$ whois -h whois.arin.net + 216.218.185.154
...
OrgName:    Hurricane Electric 
OrgID:      HURC
Address:    760 Mission Court
City:       Fremont
StateProv:  CA
PostalCode: 94539
Country:    US
...

Hurricane Electric is a co-lo in the San Jose, CA area. This is likely where the servers for LinuxDevices.com reside. Considering all the places the server could have been, this narrows it down considerably.

OK, great so I have the street address given an IP address. Now what?

I need a geo-coder. Free geocoders exist as web services, which are convienent, but also introduce latency as the query is serviced by a third-party. I didn't want to introduce more latency – I was already querying the whois database.

I wanted to do the geo-coding locally on my own server. Luckly the U.S. Census Bureau makes its Tiger Line data freely available for download. A nifty perl module, Geo::Coder::US, provides an interface to this database.

I was now in business – I could turn an IP address into a location on the map. While not optimal it got me going so I could move on to the next challenge, using the mapping APIs and making something worth looking at.

Google Maps and Javascript

Dear reader if you know next to nothing about Javascript then I knew less than you when I embarked on this adventure. Sure I had programmed a bunch in various language, but never javascript. I needed a good reference to get up to speed.

I found the W3 Schools Javascript Tutorial to be an excellent on-line reference. It is really, really helpful with nice working examples. Go there and read up.

The Google Maps API is well documented and pretty straight forward to use. My applicatin required two basic mapping features

• Draw a map, centered on a lat/long at a particular zoom level.
• Put icons on the map at particular locations.

Most mapping applications do little more than this. The Google APIs made this a relatively easy task.

In my application the user sends a series of IP addresses to my server, where I turn those IP address in to lat/long locations and send them back to the user's browser. The highlight for me was the using XML to send the data back to the browser.

The Google API provides a really nice XML parser making it simple to marshall arguments between web applications. What a time saver! Totally cool.

Another cool thing for me was using anonymous Javascript functions for asynchronous HTTP query/response handling. First an HTTP request is made and an anonymous function is registered as the response handling function. Later the response is handled asynchronously by the registered function. It was cool using an anonymous function for this – reminded me of the lambda functions of LISP.

Iteration and Closure

We are what we repeatedly do. Excellence then is not an act but a habit.
– Aristotle

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
– Antoine de Saint Exupery (1900 - 1944)

Just like writing an essay, creating good computer programs usually benefit from one or more development iterations – quickly putting down the first draft captures the fleeting flashes of insight bouncing around your ideas. Subsequent drafts smooth out the rough spots, refine the algorithms, shorten the code paths and delete the cruft.

Earlier I alluded that the final version of my application did not stick with the whois/geocoding method for turning IP addresses into lat/long coordinates. During development I found a more satisfying method.

The first enhancement was to realize that someone else was already offering a free service for U.S. and U.K IP addresses. Go check out hostip.info. This solved the problem for two large countries, but what about the rest of the world?

Well that was going to be tricky. In the end I knew the application did not need to be perfect, but only provide an estimate for the path that packets took across the Internet.

The hostip.info query fails whenever it does not have a location for an IP address – it can sometimes fail even for U.S. and U.K. addresses that it does not know about. When it fails, however, it does tell you the country that the IP address is in.

If the country is the U.S. then I fall back on the my original whois/geocode method, which works OK. For other countries I use the lat/long of the capital city of the country. So the resuls are not perfect, but the spirit of where the packets are going is preserved. I used the list of world capital lat/longs found here.

Until Next Time

Well I hope you have enjoyed this journey – good travels!

Oh, I almost forgot – please check out my Geo IP Route application.

Posted by curt at December 18, 2005 2:48 PM | TrackBack