GPSLog Labs Blog

GPSLog Labs Blog

Feb 20 / 9:24pm

Cleaning up a bad GPS log file

On a run the other day, my GPS logger (an AMOD AGL3080) recorded a really noisy signal and the resulting trace was inaccurate for the first 10 minutes. I have no idea why it did this, the second 10 minutes were fine.

The end result was a log which was almost unusable, the distance recorded was twice what it should be and the average speed was way off.

By experimenting with the Speed Filter I was able to find a set of filters that cleaned the log up remarkably, meaning I could include it in my training stats after all.  The following graph and map show the discarded points and the resulting "cleaned" data:


 

The red points are discarded by a Radius Filter as that section of the route goes down a lane-way where the signal is generally very bad.

The green points are discarded by the Speed Filter and show how bad the signal was for the first 10 minutes.

The speed is smoothed with a 10 second Speed Median filter too, but the discard filters do the bulk of the work.

I had to experiment with the Speed Filter parameters by varying the cut-off speed until the log's distance was correct (based on previous logs along the same route).  When the cut-off was too high, the resulting distance was still too long, and when it was too low, too many points were discarded and it was too short.  Once the distance was right, the resulting average speeds were reasonable enough for me to be able to use them.  As the table of before and after stats shows, the results were pretty dramatic:

This is the most successful filtering of a log I've seen so far, generally it's removing much smaller amounts of noise, and it's great to know it can be this powerful.  I hope some other GPSLog Labs users can get similar results too as it's very frustrating when a log of your activity doesn't record well and is unusable.

-- Tom

Comments (0)

Dec 6 / 10:15pm

Manual discard filters

In addition to the automatic discard filters, GPSLog Labs has 2 "manual" filters you can use to clean bad data out of GPS tracks that the automatic filters can't handle.

Manual Filter

This filter lets you discard logged data between two times. E.g. if you spent some time inside a building and got spurious signals as a result, you can "zero out" the data between when you entered the building and when you left.

Select the time range by highlighting the portion of the log on the graph, or adjusting the slider, or keying the times into the text boxes:

The data between those two times will be discarded and GPSLog Labs will interpolate the log between the remaining points:

You can use as many manual discard filters as necessary to clean up your log.

Radius Filter

This filter lets you discard logged data in a specified area. E.g. if you have travelled between buildings or through an underpass where you get bad signals you can zero all points within that area. 

Select the area to remove by moving the marker on the map and adjusting the radius:

The data between within the selected area will be discarded and GPSLog Labs will interpolate the log along a straight line between the remaining points.  This won't be perfect, but will be better than the original noisy data.  Like Places, the discard filter is "sticky" so that if the track leaves the radius briefly (less than 60 seconds) it will discard everything between the first point inside the radius and the last.  This should mean you don't have to make the radius too large and will still be able to capture all the bad points.

You can use as many radius discard filters as necessary to clean up your log.

-- Tom

Comments (0)

Dec 6 / 9:50pm

Automatic discard filters

GPSLog Labs currently has 2 "discard" filters that run automatically to clean bad data out of GPS tracks.

Speed Filter

This filter is the easiest to understand: It will discard any points where the speed is above a certain threshold.

It has two parameters, the cutoff speed (e.g. 20 km/h for running) and the number of points to discard either side of the "bad" point.

After the points have been discarded, the log will be interpolated between the remaining points, hopefully giving you a more accurate trace of your activity.

Static Navigation

The static navigation filter will only be useful if you have a GPS logger with a chipset that includes this "feature" and you can't disable it (e.g. a logger using the SiRF Star III chipset such as the AMOD AGL 3080.)  More details on static navigation can be found by reading http://www.etenblog.com/2007/11/20/static-navigation-explained/ and http://www.gpspassion.com/forumsen/topic.asp?TOPIC_ID=25575.

Basically, in this mode (which is optimized for driving, not walking or running), when your speed is below 5km/h the logger will "clamp" your position on the assumption it's getting some kind of signal "drift".  When it decides you really have moved the logged position will "jump" giving you a log that looks something like the following:

Applying a simple median filter to this data doesn't really help (see the detail below) as there are still lots of samples at the same location:

The static navigation filter will detect these points and remove not just the point, but as many of the "clamped" points as it can, giving a much better result:

(Note that this still needs a median filter to get the best results.)

The next post will describe the "manual" discard filters.

-- Tom

Comments (0)