GPSLog Labs Blog (filters)

Updates for November 10th

Sun, 10 Nov 2013 01:00:00 GMT

Some recent updates to GPSLog Labs:

When filtering a log file using a time or position filter, you can now copy nearby waypoints, rather than drawing the line each time.

Once you zoom into/select the area to be filtered, click the "Search for nearby waypoints" button.

This will list all the nearby waypoints and you can click on them to copy them into your filter.

This search uses not only waypoints you've created before, but also those created by all the other GPSLog Labs users, so you can take advantage of paths others have marked and save yourself some effort.
New forward vs reverse performance chart for routes.

This graph compares the forward speed with the reverse speed for any days where you go in both directions on the route. E.g. a commute to and from work.

The shading on the graph represents the 25th—75th percentile of the data, the red line indicates the median speed.

Points in the bottom-right represent fast forward direction with a slow log in the reverse direction, while the points in the top-left are slow forward and fast reverse. (Note, this description is flipped for pace formats.)
Download places as waypoints in a GPX file that you can load into your GPS to assist with navigation and route planning.

The Places page has a new "Download" menu that lets you generate a GPX file containing waypoints for each of the places you have created.

You can choose to get all your places, just the active places or just your favourites.

This menu also incorporates the CSV download option which will get all your place information into a spreadsheet.
Downloaded zip files now have modified dates of the files set to the current date/time.
The tabs on various pages around the site now load dynamically, so you'll notice a little "Loading..." message after the tab is clicked. You can then quickly switch between the tabs without waiting for the whole page to load again. This is very handy when comparing the heart rate graph with the altitude profile or speed graphs, for example.
And lastly, a little tip:

Set a place radius to 1m to prevent it being used to cut activity without having to mark it as inactive. This will still allow you to select the place as a start/end point manually.

This can be handy if you pass by a place on a regular commute and don't want to have to remove the stop from your processed activity.

Updates for August 25th

Sun, 25 Aug 2013 02:00:00 GMT

Some recent (and some not so recent) updates to GPSLog Labs:

New clock, altitude profile and map thumbnails.

The thumbnail images have a new updated look that is smoother and more detailed. They are also rendered in the browser, rather than being downloaded, which speeds up the display of the pages.

The clocks now "zoom in" when indicating short durations to make the most of the available space and show the duration in hours and minutes too. A duration of greater than 12 hours is indicated by shading the full circle.

An arrow indicates whether the time is a.m. or p.m. (the arrow points from the start or end time to noon).

Times for "junk" activity are shown in a small and simplified thumbnail.

Altitude thumbnails simply show the profile, with distance grid lines every 10km and altitude grid lines every 100m.

Map thumbnails have a 1km grid to indicate scale, and show the start location as a green dot and the end location as a red dot.
You can now generate a Google "static map image url" for an activity via the Activity detail "Share Log" page. I.e. a simple image file that can be downloaded or added to a page and it doesn't require any scripting to make it work.

This feature is only available for shared activity, but once the url has been generated or the image downloaded, the activity can be made private again.
The thumbnail for activity you share to your Facebook timeline is now a small "terrain" map. This looks nicer and has more detail than the old icon.
Activity tags are now sorted by overall frequency, not just alphabetically. This makes the titles of pages (particularly shared activity and posts to Facebook etc.) more sensible as the primary activity tag is likely to be first (e.g. "Bike" or "Run") followed by any minor tags (e.g. shoe type or bike being used).
I've added shading on activity heart rate graphs to indicate hear rate zones:

An earlier post has details on setting up heart rate zones for your tags.
Likewise, there is also shading on activity speed graphs to indicate speed zones if you have them configured.

There are links under these graphs to take you to the zone configuration if you haven't yet set up speed zones (or heart rate zones).
A bug that meant times weren't working properly for non-English dates on the manually create segments page has been fixed. Note, this page still won't work properly for very long files/segments. It's probably best that those are edited using gpsbabel before uploading to GPSLog Labs.

This page has also been updated with the new layout.

Updates for January 7th

Mon, 07 Jan 2013 09:37:56 GMT

I've made a few more changes to GPSLog Labs to better support heart rate logging:

Heart rate data can now be filtered!
These filters work in the same way as the other GPS data filters and can be added to individual logs or to tags so they are added automatically.

There are 3 heart rate filters, one takes the median heart rate, one the average (mean) heart rate and the last one is a special filter to clean up noisy heart rate data during your warm-up where the heart rate strap doesn't record properly until you've built up a bit of a sweat.

You put in a "warm up time" (7 minutes in the above image) and it will discard any readings that are too high during that period. It assumes your heart rate after you've warmed up is higher than when you're warming up which is pretty rough, but it cleans up a lot of messy data.
Additional heart rate log files can be uploaded in .zip files
The tag detail page summary tab now has "box whisker" plots for the heart rate distributions

Some other recent changes include:

Speed zones

These are configured per tag like heart rate training zones and will be displayed in the background of the activity tables to give you an indication of the intensity of the activity.
The Speed Average filter has been changed from an exponential average to a regular mean, so it no longer "lags" behind the signal.
The Speed Median and Speed Average filters have also had a cutoff added to preserve stops, any data below that threshold won't be filtered.
The GPSLog Labs home page has a new layout that highlights some key features a little more prominently.
Alert & confirm popups use the alertify.js library.
Support for the Garmin FIT file format has been added.

Updates for October 17th

Wed, 17 Oct 2012 09:58:05 GMT

The last few months have seen some minor updates to GPSLog Labs and a bunch of major heart rate logging related features that will be in a post of their own.

"Race results". This is a new visualisation available on the Best Efforts tab for Tags and Routes.
It takes the best efforts at a given distance from all of your activity and shows a "photo finish" of where each log would be as the "winner" crossed the line.
Choose the distance to be compared by clicking on the column headings of the best efforts table.
Some of the results may be collapsed so the chart doesn't take up too much space.
Together with the "Race results" there's also a histogram of Average Speed/Pace.
It's actually a few histograms overlayed, the darkest bars are the count from the last few months, the lightest the count for all time. This means you can see if there's a pattern to your recent activity that differs from the rest.
The Uploaded log file page has a new tab, Edit Activity (junk hidden) that lets you edit the places & tags without being distracted by the junk segments and merge points.
There's a new "Remove Stops" filter to clean up maps of hikes or walks where otherwise there would be too many stop markers.
This is a bit of a hack, but it allows you to filter out stops which are shorter than the threshold. Unfortunately it does that by substituting a speed of 1km/h during the stop, which will mess with the average speed stats, so this filter should only be used where those stats are not too important.
The Tag map now has an option to show log colours "By route" or "By direction". Choosing "By route" can help find logs that may have been mis-classified or to find logs that could be grouped together into a new route.

Using the position filter to clean tracks through tunnels

Tue, 27 Mar 2012 09:59:03 GMT

GPSLog Labs' Position Filter now has a new "tunnel mode" that interpolates along a custom path instead of a straight line so resulting distances and speeds are more accurate.

This is easiest to illustrate with an example: A route I regularly run goes under a raised freeway for about 500m. Throughout this section, the GPS signal is unreliable and the following map shows the kind of quality of track I can expect:

Picking one of these logs in particular shows that it's probably got the wrong distance and the speeds will be way off too, not very useful.

So, to fix this up, click on Filters in the right-hand side panel of the activity detail page:

Then add a filter:

And choose the Position filter:

First you'll need to position the "discard" circle over the area of the track you wish to delete.

By default, the discarded point from your track will be replaced with a straight line through the circle. This is likely to result in an incorrect overall distance being calculated for your track (i.e. it'll be too short and the average speed will be to low too.)

To add a line of waypoints to interpolate the track through, click the Add Waypoints button then click on the map to draw the new path through the area.

Double-click to stop drawing the line, and drag the points to edit them, clicking a point will delete it.

Once the filter is saved and the edit page reloaded, the discarded points will be shown as small markers.

The following graph shows how the new track is interpolated through the waypoints at a constant speed, so stopped time will be counted in your average speed.

And back on the activity detail page, there's a nice cleaned up track that shows where I actually went, much better!

Like other filters, the Position filter can be saved to a tag so it will automatically be applied for any appropriate activity passing through that noisy signal area, meaning you'll never have to deal with messed up GPS data again.

New "Splat" filter

Tue, 07 Dec 2010 01:34:37 GMT

A problem you'll encounter very quickly when using a GPS logger is "splats" in your log files caused by a bad signal while you're inside a building or other area with lots of signal reflections.

This can be very annoying as it will often create spikes in your speed that completely mess up any maximum speed or distance statistics for that log file.

For a while, GPSLog Labs has had "position" and "time" filters that let you manually remove these "splats" but it was tedious to get right, so now a simpler "splat filter" has been added that makes the process much less painful.

First, go to the activity you want to filter and click on the Filters link in the side panel:

Then click "Add a filter" and choose "Splat filter":

On the Splat filter page, you need to drag the marker so that it covers the "door" of the building. It doesn't have to cover all of the spikes of the splat, you just need to capture the last good point before you enter the building and the first good point when you get out and start getting a good signal again.

As you can see, the marker is only covering the actual building under the splat, not the whole splat.

The Splat filter will then discard all the points while you were inside the building and interpolate over the gap (which will give you a long period of very slow or 0 speed, restoring the stats of your log back to something sane):

The results are dramatically better, the distance and maximum speed are both now much more accurate and representative. This log also included the Static Navigation filter as the AMOD AGL 3080 logger was quite bad for logging at walking speeds.

Enjoy!

Updates for December 7th

Tue, 07 Dec 2010 01:05:45 GMT

There's another new button on the edit Log File tab exposing a feature that was previously a little too well hidden, this time to select which segments should be marked as "junk":

That brings up a popup where you can quickly choose which segments are junk (and therefore shouldn't be displayed in the rest of the site) and which have the activity you're interested in:
The colour of the junk segments on that page has been changed subtly from red/orange to a more neutral gray too.
The GPS Data Filters page has been completely overhauled too so it should hopefully be a bit easier to use and faster to add certain type of filters too.
Click on the Filters side panel of an activity you want to filter:

The GPS Data Filters page now has clearer menu options and a bigger preview maps and graphs to show the results of the filters you have applied:
And the individual filter pages now have more space to show their options and view the maps and graphs:
The Time filter now has a preview map to show you the segment that is being discarded.
Most filters now have a "Copy to tag" menu option that will let you copy the filter and it's parameters to a tag. This means that once you find a good combination of filters to clean up your activity, you can have them automatically applied to all data you upload from now on tagged with that tag.

Updates for November 19th

Fri, 19 Nov 2010 04:40:48 GMT

Updates on the updates to GPSLog Labs are lagging a bit, but here's some of what's been happening lately:

The log file chopping logic has been tweaked a little to hopefully avoid creating as many unwanted "unnamed places":
- Auto created places must now be further away from known places before they'll be created.
- Auto created places won't be used to chop logs between known places. If you do want the auto created place, after you name it, if you reprocess the log then it will be cut properly.
- There were also a few bug fixes that should make detection of places more accurate.
There's better support for editing and filtering of large log files.
The little milestone markers now have a small arrow attached to indicate the approximate direction of travel at that point:
There's now a filter on the logs tab for a place that lets you narrow down the activity by tag:
The log file editing page has been tweaked to make finding some of the commands easier, the menu now includes the "Add a Place" and "Manually create segments" commands that were previously a little hard for some users to find:

It's now possible to change the Route or Tags directly from the edit page for an activity after they have been set:
The "Select tags" popup now allows quick selection of sets of tags from those used previously on other activity on the same route:
Speed vs time graphs when you're editing and filtering log files now have a "scale" selection, that will allow you to see detail in log files which was previously hidden by large speed spikes caused by GPS errors:

In this example, the actual activity is completely obscured by the 5,000 km/h errors... Selecting a maximum of 50 km/h on the scale means you can see the activity properly and edit it as required:
There's a new "Inverse position filter" available, that is the opposite of the Position filter. i.e. it will discard points outside the given circle.

Updates for March 28th

Sun, 28 Mar 2010 07:50:40 GMT

Florian is doing a great job translating the site into German, danke schön!
The Summary tab on a tag detail page will show a table breaking down the activity for that tag by other tags where appropriate:
The Distance vs Time graph on a tag detail page now shows a line through the mouse position that let's you see at a glance which logs are faster, slower or the same speed:
Table rows are now highlighted as you hover the mouse over them.
GPS Data Filter editing has been improved to make it more obvious that the items can be dragged to re-arrange the order that they are applied:

Cleaning up a bad GPS log file

Sun, 21 Feb 2010 05:24:00 GMT

On a run the other day, my GPS logger (an AMOD AGL3080) recorded a really noisy signal and the resulting trace was inaccurate for the first 10 minutes. I have no idea why it did this, the second 10 minutes were fine.

The end result was a log which was almost unusable, the distance recorded was twice what it should be and the average speed was way off.

By experimenting with the Speed Filter I was able to find a set of filters that cleaned the log up remarkably, meaning I could include it in my training stats after all. The following graph and map show the discarded points and the resulting "cleaned" data:

The red points are discarded by a Radius Filter as that section of the route goes down a lane-way where the signal is generally very bad.

The green points are discarded by the Speed Filter and show how bad the signal was for the first 10 minutes.

The speed is smoothed with a 10 second Speed Median filter too, but the discard filters do the bulk of the work.

I had to experiment with the Speed Filter parameters by varying the cut-off speed until the log's distance was correct (based on previous logs along the same route). When the cut-off was too high, the resulting distance was still too long, and when it was too low, too many points were discarded and it was too short. Once the distance was right, the resulting average speeds were reasonable enough for me to be able to use them. As the table of before and after stats shows, the results were pretty dramatic:

This is the most successful filtering of a log I've seen so far, generally it's removing much smaller amounts of noise, and it's great to know it can be this powerful. I hope some other GPSLog Labs users can get similar results too as it's very frustrating when a log of your activity doesn't record well and is unusable.