Limit gradient scale in chart to something reasonable instead of autoscaling to silly values

Well and what you expect Andrew when computed track force you to jump 7 metres :).

Check line 3189 in file (14.25 m) vs next 3192 line (7.5 m).

And attachments - I!m aware of this problem, have to write to developers of this system what to do with it.

You would have to agree the value of -400% has made the gradient graph totally useless to view. This is not an imported track where an argument could be mounted that all elevations must be accepted, but a track created by BRouter. A value of -400% is about -75 degrees. For a fast cycling profile this is clearly a bogus value. As I already said I'd expect/ hope these values to be ignored prior to the filtering step. Yes I fully understand a 7m step change has caused this bogus value.

Andrew, I absolutely agree that such value is total nonsense for bike ride. Anyway you cannot expect that Locus should automatically ignore such values. What about cases when such value make a sense and it is expected value?

My practice is, that when I download any track with such crazy values, I then use function "Fill altitude" that for me in all cases fix incorrect altitude values in any track.

>Anyway you cannot expect that Locus should automatically ignore such values

it could ignore such values in situations where it knows they are nonsense, but not to pursue this impossible dream further because...

...I've never used the "Fill alitude" function before. Thanks a lot for making the suggestion. That's a reasonable compromise. The spike reduced from -400% to -30%, and now I can see Y-axis detail for the gradient.

Both Locus and Brouter use SRTM3 altitude data based on 90x90m space shuttle radar survey.

Everybody who uses these data should know there are many SRTM artefacts. Some are just data errors, and some of them are corrected. I supose there may be more SRTM3 versions, with different error correction progress. Also, some software may use SRTM data or track altitude filtering.

But some artefacts are related to fact how terrain features affect averaging of 90x90m radar altitude data. Sources of artefacts causing sharp SRTM altitude jumps are mainly:

narrow valleys, rivers, forest passages, urban areas, tunnels, bridges.

That is the reason why Brouter uses 10m altitude hysteresis filter, that is filtering out many, but not all artefacts, as it is a trade-off between filtering and distorting.

Once can have 20-30 m artefact jump in few metres, if you review SRTM altitude profile of a route you are familiar with.

I now have a phone with a pressure sensor, and testing shows the altitude accuracy is far greater, far more reliable, far closer to known elevations on a topological map. However on a recent 600km ride I again experience a single track point which totally skews the gradient graph with silly autoscaling - gradient less than -150% ! Unlike previous phone where it may made sense to filter a track, I don't think this should now be necessary.

GPS track at - https://drive.google.com/open?id=1aQ-HbrmGdySpIrr4lWUmFqaGs04hbC1Z

I eventually find this single bad track point using Garmin Basecamp point #20078 has zero metre leg distance.

When I deleted this point and re-imported into Locus, the gradient chart is more reasonable -10% to +5% gradient. Unlike previous discussion is it therefore reasonable to ignore points with leg distance = 0m?

I note there is an abrupt jump in elevation from -7 to -15m but it appears this is not the cause of the gradient spike.

Locus Pro 3.32.2 GPS settings:

• optimize GPS values
• use pressure sensor
• light filter

Just a few versions ago I added this condition: compute gradient only in case of non-zero distance. Anyway seems here distance (leg) won't be exactly 0m, but something close to this. Hmm, I'll try to set this condition to 0.25m as the minimal distance (for real slow-walkers). This should help here.

Thanks Menion. Will I be able to check/ verify/ notice any change with this existing track chart when the new Locus version is available? Or will it only affect newly recorded tracks?

Chart is always generated as new, so changes in this will affect the chart immediately.

Not even autopaused due constant location threshold ?

Correction - on closer observation I was not even stationary at the time of 0m glitch. The spike occurred about 500m before this point; I was riding at ~15km/h at this specific track point along a flat road. So no explanation from physical reality.

Hmm. But in spite of altitude having barometric accuracy, position has GPS accuracy. In case one get for proper elevation change too short distance change, the result is a slope spike.

This is the reason an accurate barometric altitude may provide more noisy gradient profile,

compared to SRT based profile.

because

Thanks Libor but certainly in my recent testing with my SRTM data, the pressure sensor has been far better. With the SRTM grid in places where the road traverses hilly terrain the altitude as estimated with SRTM data can be wildly inaccurate.

Hi Andrew.

You may have misunderstood me.

The barometric altitude profile is indeed much better than SRT based one. But for the altitude alone. If the gradient is considered, its calculation is affected by GPS position error.

Purely GPS based gradient is affected by both position and altitude error.

GPS/SRTM based gradient has the least error of all three, as the local errors for altitude and position are correlated and partially cancel each other.

What about some deviation threshold check, the value being overwritten by a suitable interpolation if the threshold is triggerred ? Like to compute running standard deviation with threshold somewhere between 2-3 sigma ? Just thinking.

P.S.: Or, for the chart, to run statistics and use e.g. 3 sigma scale for the gradient values, cutting out outlyers.

The chart is already heavy filtered. My algorithm is based on making the weighted average from before and after 250 meters (or nearest 10 points) around the point, where in case of the slope, is used the strongest parameter that should throw out all these crazy values.

Anyway, I've added your tip to my private notes related to this topic, thanks.

Hi Andrew,

I just tried your GPX track from an older post and it looks quite well I think, so there definitely is a progress.

Three points you send before: what result you expect here? During 6 minutes, 3 points, almost no horizontal movement and 70 m vertical change? Crazy values for hike/bike movement. If you have full recording (GPX) with more points and result is still same, send me it here and I'll gladly check what happen in app, thanks.

Thanks @menion - I've attached the whole GPX file. Note again the workflow - turn on phone, immediately run Locus (auto track record & GPS enabled @ start). Maybe the GPS needs more points to stabilize. This new phone has a pressure sensor and the elevation in general is far less noisy than previous phone, but I have seen with & without a pressure sensor. You said, many years ago, you would ignore some initial points to avoid any startup issue. Although I have a few other examples over the years where it is not just a startup issue.

>what result you expect here?

Yes crazy values (maybe just 1st two), but common phone, nothing special, I imagine it happens more often than people may realize. Would not be obvious unless you graph gradient. It is clearly crazy to a human so I expect/ hope here Locus could be more intelligent like a human too.

In this topic 3 years ago you said "Anyway you cannot expect that Locus should automatically ignore such values. What about cases when such value make a sense and it is expected value?" Could my suggestion to link the elevation filtering/ point rejection "logic" (for want of a better term) with the track recording profile be a way forward? If the profile is for a pilot, paraglider or rockclimber etc. then sure it is difficult to ignore such values. Maybe if the profile is for a road cyclist, Locus can make better assumptions that elevations can not jump by large amounts when the position changes by a small amount? The 1st 3 points are within 30m of each other, and there is a 34m change of elevation just within ~8m for 1st 2 points. Libor may be better thoughts to an actual filtering algorithm and/or whether individual points could be rejected.

sure, signal processing can be effective, but there are also simple ways.

One of practical simple solutions for first derivatives directly from original data is using coefficients for the 1st derivation for Savitzki-golay smoothing (SGM). SGM approximates a time serie by least square method by a local cubic polynom over a sliding set of point.

The point is, it results in just weighting of points around the given point by integer numbers and normalizing the sum..

For the 1st derivative, the gradient in our case, the weighting gets very simple. Typical range of used points is 5-25, i,.e. N = 2 .. 12

If 2N+1 points are taken, gradient at the central point isapproximated as:

g = ( -N * y(-N) - (N-1)*y(-N-1) .... -1 *y(-1) + +1*y(1) + ..... + N * y(N) ) / [ N * ( N+1 ) * h ]

where h is the point step interval.

if x = 0, 2, 4 , 6, 8 and y = 5, 6, 7, 9, 10

then appproximation of the gradient dy/dx at the central point x=4 is

( -2*5 + -1*6 + 0*7 + 1*9 + 2 * 10 ) / ( 2*(2+1) * 2 ) = ( -10 - 6 + 9 +20 ) / 12 = 13/12

In this example it is rather time gradient, the change of altitude with time, not the distance.

But as the altitude values come with 1s interval......

I think the gradient based on HGT files is quite a safe harbour. the option to derive gradient not from GPS altitude, but directly from HGT file may be interesting, as it would be smooth. It could be on user to decide, which way to be used.

Also, it would give an agreement of planned and recorded tracks.

Sure, it would be applicable for special cases like climbing or air gliding.

Correction:( wrong memory )

If 2N+1 points are taken, gradient at the central point isapproximated as:

g = ( -N * y(-N) - (N-1)*y(-N-1) .... -1 *y(-1) + +1*y(1) + ..... + N * y(N) ) / [ 2 * sum[1..N]( N^2 ) * h ]

where h is the point step interval.

if x = 0, 2, 4 , 6, 8 and y = 5, 6, 7, 9, 10

then appproximation of the gradient dy/dx at the central point x=4 is

( -2*5 + -1*6 + 0*7 + 1*9 + 2 * 10 ) / ( 2*(1+4) * 2 ) = ( -10 - 6 + 9 +20 ) / 20 = 13/20=0.65

In this example it would be time gradient, as the altitude values come from GPS with the constant time interval 1s, while the distance interval is variable. But we could easily calculate the gradient = time gradient / speed.

Time gradient 0.1 m/s and speed 10 m/s lead to gradient 0.01 m/m

I did have an example from ~3 months ago where mid-ride there was a spike, but 95% of the time the spike is when Locus is 1st started, or when using pause/resume of track recording. Mostly I don't even check the gradient chart because is well known route & ridden before, so spikes may occur even more frequently than I thought.

Does "ordinary" HGT data have sufficient spatial resolution (30x30m I think) to help in your proposal? What about where the area is very rugged?

For an illustrattion, I created a s testing slope 0.25 per point, with added noise with STDEV=1.0, i.e. like very up/down jumpy GPS based altitude slope.

Below is the table of filtering progress, providing the the average and stdev of omputed slope, given the N of the filter. On thebsecond row is the noise percentage, based on the slope stdev.

you may try this Excel document on Office online ( only a browser is needed ).

( Read Only )

https://1drv.ms/x/s!AmTXM3zwkMZNgRDQabOMDmOiV5kc

Thanks Libor for your suggestions & precise pre-prepared tool. I'll definitely check it next week (once we publish new version).

The filtered value is the exact value of the 1st derivative of a cubic polynomial, used to approximate 2N+1 points via least squares approximation method.

For the filter of the strength N,

create array of the size 2N+1, with values

-N, -(N-1), ...-1 ,0, 1,...N of the weights.

The filtered value would be the scalar multiplication of this array of weights

and of the array of 2N+1 points values, divided by the normalisation constant.

The constant

Is the array of weights,

scalar-multiplied by itself.

Therefore,

Filtered_value = ( points . weights )/(weights . weights)

where . is array scalar multiplication.

Another example today is attached. Elevation of 1st track point is 60m below expected value. I note <pdop> is very high 26 whereas normally far more precise (5 to 10). Maybe elevations with a pdop above some smart threshold should also be ignored? See here for PDOP info.

Correction: The filtered value is the exact value of the 1st derivative of a quadratic polynomial, used to approximate 2N+1 points via least squares approximation method.( cubic one would have harder to compute values ).

Note that the Savitzki-Golay filter has its wikipedia page as well .

Sorry Libor I haven't had a chance to check your filtering yet.

Menion - I have attached another file, and here is also a link to RWGPS. The Locus chart is scaled to +40% but I can see no gradient spike either in the Locus or RWGPS charts that are more than 8%. Also in reality there is no gradient more than 8%.

Attached is another interesting example of track recording, and problems with invalid elevation. Elevation at 2019-01-20T02:11:39.000Z is 107m at lat="-42.484641" lon="146.710361". I enter a store for ~30 minutes. When I come back out the elevation is 94m, and it takes many minutes before it is stable again. I would assume this could be an issue for any Locus user.

Andrew, thanks for the feedback. I've marked this topic in my TODO list. I just do not have time on this as there are currently more important tasks to do, sorry.

That's OK. It was more of interest than anything more, that leaving track recording enabled while passing through tunnel or GPS black spot (eg. indoors) will possibly result in elevation discontinuity. I haven't had a chance to look at Libor's advanced filtering yet either.

Here is a track from yesterday where the Locus chart has autoscaled the gradient -50%/+10% but I can find no spike within the GPX to explain this invalid scaling. When I zoom into the Locus chart I can't see any negative spike. The RWGPS chart shows more reasonable +/-10% scale. Any explanation?

Nice ride ... may you attach here original GPX from Locus Map? It has to be something at the start (first few points).

<trkpt lat="-42.981811" lon="147.180706">
	<ele>270.40</ele>
	<time>2019-03-13T21:57:41.000Z</time>
	<course>151.881</course>
	<pdop>13.00</pdop>
</trkpt>
<trkpt lat="-42.982041" lon="147.180706">
	<ele>238.83</ele>