Close

January 14, 2011

Rapid Prototyping with Google Earth

We recently started research work for a potential grant and some of the project focused on dynamic visualization of data sources. A few years ago we had done some work for a client in visualizing aircraft telemetry data using the stand-along Google Earth client. At that time, the Google Earth browser plug-in was not yet mature and still had a few issues.

With our current project, we needed a quick way to geographically visualize a large set of data. A years worth of Apache log files, a perl script, some PHP code, and we were able to create a fully automated data integration and visualization system with no cost.

1.) Our first step was to normalize our log-file data and create a basic back-end data integration system. Thankfully Apache2 log files are relatively easy to parse with Perl. We setup a script to crawl our local log directory, parse through the lines to collect unique visitors, run the IP address through a basic IP Geo database and output the cleaned data into a flat file.

2.) The next stage was to create the front-end to display the data. Google Earth is a relatively straightforward system to develop with as there is a ton of documentation on the Google Earth API and the KML language. The GE (Google Earth) Plugin installs easily into both Firefox and Chrome and is ready to go a minute later.

You’ll need to get your Google Earth API key (uses the same Google Maps API). The webpage setup of the Google earth map is a bit different, so pay attention to how you call the Google Javascript class with your key (sample below).

<!DOCTYPE html "-//W3C//DTD XHTML 1.0 Strict//EN" 
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>Google Maps JavaScript API Example</title>
<script src="https://www.google.com/jsapi?key=YOURKEYGOESHERE"></script>
    <script> 
google.load("earth", "1");
 
var ge = null;
 
function init() {
  google.earth.createInstance("map3d", initCallback, failureCallback);
}
var t=setTimeout("runLoggy()",3000); 
function initCallback(object) {
  ge = object;
  ge.getWindow().setVisibility(true);
	      ge.getNavigationControl().setVisibility(ge.VISIBILITY_SHOW);
}
 
 
function failureCallback(object) {
}
 
function runLoggy(){
	// fetch the KML
	var url = 'kml_data.php';
	currentKmlObject = null;
	google.earth.fetchKml(ge, url, finishFetchKml);
}
 
function finishFetchKml(kmlObject) {
  // check if the KML was fetched properly
  if (kmlObject) {
    // add the fetched KML to Earth
    currentKmlObject = kmlObject;
    ge.getFeatures().appendChild(currentKmlObject);
  } else {
    // wrap alerts in API callbacks and event handlers
    // in a setTimeout to prevent deadlock in some browsers
    setTimeout(function() {
      alert('Bad or null KML.');
    }, 0);
  }
}
 
    </script> 
  </head> 
  <body onload='init()' id='body' style='background-color:black'> 
      <center> 
		<div style='width:700px'>
		  <div> 
		   <A href="/"><img src='header.png' border=0></A>
		  </div> 
		   <div style='color: #C0C0C0;font-size:1.5em;font-weight:bold'> 
			Web Traffic Geographic Dispersion
		  </div> 
		   <div style='color: #C0C0C0;font-family:tahoma;font-size:1em;text-'> 
			<br/>A small Google Earth project that displays the geographic location of every unique visitor to our website over the past year.<br/>&nbsp;
			<a href='/'> Back to our website</a>
		  </div> 
    	   </div>
	</center> 
  </body> 
</html>

Notice the “kml_data.php” file that is used to pull in the placemark data into the model. This is a PHP file that crafts a valid Google KML file based on the data we had “mined” from our Apache web traffic logs and feeds it to the Google Earth plugin engine. The PHP code is pretty straightforward, but the header of the file needs to specify the content type of the data that we are serving. Without it, the Google Earth plugin will not render the data correctly. Listed below is the header section of the code.

<?
// set the output format to KML so that GE is the target 
header("content-type:application/vnd.google-earth.kml+xml"); 
// have the browser treat the output as a download and assign a name 
header('Content-Disposition: attachment; filename="kmldata.kml"'); 
?>

Warning: If you are changing the icon styles and/or colors of your placemarks be SURE to either clear your cache or use a different browser to test the changes. The old icons are cached and if you keep refreshing with your current browser you may think that the changes to your KML file are not working (not that we encountered this for around 15 minutes). Just alternate between two browsers like Chrome and Firefox to be safe.

3.) Add some “Shineyness” to the form and you are all set. For an hour or so of work, you have a quick prototype system that you can easily modify and add more bells to in the future. (Your prototype better have better comments and spacing).

Google Earth is a great tool for rapid prototyping, but beware that it is not completely free. You must purchase a commercial license if you are going to use it for a corporate product.

Click below for a live demo.

Click for a live demo.

UPDATE: 1/16/2011:

We were inspired of our relatively quick system to visuzalize data, so we built one using some of our weather data. The system is automated and should show the past hour of data from every site in the US. The weather model is still a work in progress, but it works for 99.9% of META site data.

Click for a live demo.

Recent Posts