Gravl: Referers, Geocoding, Charts

An article by Glen    3 Comments

Continuing my obsession with the power of Grails filters, it was time to add some referer tracking and ipaddress Geocoding to Gravl to see where people are coming from. So I created a new RefererFilters class in conf and I was up and running. I already had a little CacheService happening, so I created a new 24 hour cache, scooped up some interesting browser header data, and squirrelled it away for later:

 // Scoop up Referer and User-Agent and remote IP
def userAgent = request.getHeader("User-Agent")
def referer = request.getHeader("Referer")
def ip = request.getRemoteAddr()

// Grails adds request.forwardURI to keep the orig URI
def url = request.forwardURI 

// put in the cache, not use of applicationContext to lookup service bean
CacheService cacheService = applicationContext.getBean('cacheService')
cacheService.putToCache("referers", 60*60*24, Calendar.getInstance(),
				[ userAgent: userAgent, referer: referer, ip: ip, url: url ])

One thing to notice in the above, service injections don’t happen in the filter chain, but you are provided with an applicationContext that you can use to look up your services by name.

Anyways, now I’ve got some stats, it’s time to do some crunching… Referers was the first target to add to my stats page… though the css needs some work to fit the counters in

Now what to do with those ip addresses I was gathering? Picking up on the stuff Simon is doing in the latest Pebble, I thought it might be nice to add some geocoding to that referers list so I can see which country my traffic is coming from.

Turns out there’s a very simple little Java library available from MaxMind called GeoLite Countries. I mavenized the source into a jar, then dropped the resources file into my /src/java/resources. I already had the ip addresses I’d gathered from headers into my cache, so to convert them to a country name, it was time to create a CountryLookupService to wrap all the calls to the GeoLite library:

class CountryLookupService {

    LookupService lookupService

    private LookupService getLookupService() {
        if (lookupService == null) {
            // assume you put your data file in /src/java/resources
            def url = this.class.getResource("/resources/geoIp/GeoIP.dat")
            log.debug ("Looking for GeoIP database at: $url")
            lookupService = new LookupService(new File(url.toURI()),LookupService.GEOIP_MEMORY_CACHE);
        }
        return lookupService
    }

    def getCountryName(String ipAddress) {
        log.debug "Looking up address for ${ipAddress}"
        return getLookupService().getCountry(ipAddress).getName()
    }
}

And we’re off and running. The GeoCoding API requires you to supply a current GeoIP.dat file, and the sample above shows you how to load it from your Grails /src/java/resources folder to keep everything all relative. I’m not sure where the agreed convention is on resource placement in Grails, but there is as good as any for now… After that I just needed to iterate my cache, and we have ourselves a hashmap of countryname to hitcount:

Once you’ve crunched all that data, it’d be nice to produce some slick charts to make it all digestable. And what quicker way to chart it than with James Williams uber cool Google Charts Plugin.

If you haven’t used Google Charts before, the idea is just so good and so simple. Create an img tag with the src that points off to http://chart.apis.google.com/chart? then add url params to tell it your chart size, type, title, etc. and Google will render a png on the fly for you.

James plugin makes that process very simple and painless. I take my hashmap of countries (which carries countryName -> hitCount) and feed it straight into his taglib:

<g:pieChart type="3d" title='Hits By Country' size="${[700,300]}"
   labels="${countries.keySet()}" dataType='simple' data='${countries.values().asList()}' />

And the result?

So perhaps a pie chart might not be the best selection…

Wow.. that was all quite an adventure in Referers, Geocoding, and Charting! I’ll be talking about some of these adventures at G2X, so if you’re in North America next month, make sure you drop over and say hi!

Happy Grails Stats crunching!