Threshold for altitude changes

Falco shared this idea 2 weeks ago
Collecting votes

My altimeter pressure sensor does have 0.3m fluctuation and I'm unable to filter them away with locus because locus does record each 0.1m altitude change.


It doesn't help to change altitude filter light to filter very strong because it does only smooth changes out a little bit.

And I notice that some flattening does happen on track save action. It does get reduced from 69m record climb value to 41m climb in saved track statistics on 2 hours testrun on same position. Well my pressure sensor is accurate enoght that even a simple integer convertion would nearly solve this issue.


What about a threshold for altitude changes? Store only new hight values if a specific amount of climb or descent change is reached?

We don't need to round, we just need to skip altitude change flickering from uphill to downhill.


Example

messurement after filtering -> value which get recorded with threshold handling

stard with "record climb only until 1.0m descent"

100.1 -> 100.1

100.4 -> 100.4

100.2 -> 100.4

100.3 -> 100.4

100.5 -> 100.5

100.7 -> 100.7

100.9 -> 100.9

101.1 -> 101.1

100.9 -> 101.1

101.2 -> 101.2

100.9 -> 101.2

100.4 -> 101.2

100.2 -> 100.2 (1.0m threashold reached, switch to descent mode, record decent only until 1.0m climb)

99.9 -> 99.9

100.1 -> 99.9

99.7 -> 99.7

Important: threshold handling have to be applied after filtering because not filtered values would easily hit the threshold.


There is only one minor problem: we get a rough transition from 101.2 to 100.2 but this could easyily fixed by postprocessing on track save if we care about it.


I don't like postprocessing solutions for the whole problem because I would like to rely on the record screen climb indicator.

Comments (9)

photo
1

I did create a sqllite trigger as an prove of concept to find out who well a threshold would work:

ALTER TABLE locations ADD COLUMN elevation_raw FLOAT;
ALTER TABLE locations ADD COLUMN elevation_change FLOAT;

create trigger location_filter_elevation_insert AFTER INSERT ON locations FOR EACH ROW
begin
  -- get previous elevation value into update trigger context
  update locations set elevation_raw=new.elevation, elevation=(SELECT elevation FROM locations WHERE rowid = new.rowid-1) WHERE rowid = new.rowid;
END;

create trigger location_filter_elevation_update AFTER UPDATE OF elevation_raw ON locations FOR EACH ROW
begin
  --new.elevation is the previous record elevation on first elevation_raw update
  update locations set elevation=CASE
	  -- when climb threshold reached or (it was climb and descent threashold not reached)
	  WHEN new.elevation_raw - new.elevation >= 1.0 OR ((SELECT elevation_change FROM locations WHERE rowid < new.rowid AND elevation_change != 0 ORDER BY rowid DESC LIMIT 1) > 0 AND new.elevation_raw-new.elevation > -1.0) THEN
		-- allow climb only
		max(new.elevation_raw, new.elevation)
	  ELSE
		-- allow descent only
		min(new.elevation_raw, new.elevation)
	  END 
  WHERE rowid = new.rowid;
  
  update locations set elevation_change = round((SELECT elevation FROM locations WHERE rowid = new.rowid) - (SELECT elevation FROM locations WHERE rowid = new.rowid-1),1) WHERE rowid = new.rowid;
END;
Results are impressive:
Insert into locations(elevation) VALUES (100.0),(101.3),(101.7),(101.4),(101.2),(101.9),(100.1);
SELECT elevation,elevation_raw,elevation_change, elevation - elevation_raw FROM locations ORDER BY rowid DESC limit 7;
SELECT SUM(elevationfix), SUM(elevation_change_abs) FROM (SELECT elevation - elevation_raw as elevationfix, abs(elevation_change) as elevation_change_abs FROM locations ORDER BY rowid DESC limit 7);


I did write a trigger to get a partial locus integration. This workarround will work only on saved tracks and isn't efficient because of the limitations of SQLlite Triggers. And I can't use this trigger to fix my track recording screen. I need to save the track to trigger altitude threashold filter.

photo
1

Currently I work on a tasker solution based on locus/data/seonsors/ files to extract pressure values. This would even work without GPS.

- use only pressure changes (no calebration needed)

- apply pressure to altitute change calculation based on hight

- calculate pressure change by darksky to remove weather related climb and descent meters.

- apply threshold on altitue change calculation

- display the result as popup


I need to use stream based solutions on comandline like awk for this because tasker read file to memory is to slow because it take a while to move 30mb of sensor data to script execution or other tasks.


Basically I will use 2 Workarrounds:

  1. Database trigger to force elevation threshold for saved tracks for all available GPS positions
  2. locus sensor debug file readout for elevation summaries of active recordings or for saved tracks with missing gps points

photo
1

Hi Falco, I haven't caught too much about your example and further comments but I totally agree that a filtering threshold could be a solution. As a matter of fact I thought that the existing filtering is based on this concept but maybe not. Cumulative elevation error is one of the major issues with locus., Threshold correction/filtering could be applied diredtly during track recording but also with postprocessing, thus allowing to save the corrected track but also keep the original measurement and to try several corrections with different thresholds .

photo
1

Current filtering concept does only slow down the changes by mixing old values together to get the most recent sensor value less prominent. It's not threshold based like you thought. You just select if you want to mix the last second or the last 90s together to get a slow or very slow altitude change to smooth everything out.

It's easy to see how it does work. Use barometric messurement, start trackrecording while your phone is 2m in the air and look at track recording height diagramm while move your phone exaclty one time very fast from 2m in the air to ground. Without filtering 2m altitute change got applied after 1-2s as a steap line in the diagramm and with max filter you need to wait 40 to 90s until you reach the final altitute very slow in the diagramm.

For that reason I spend all the affort to get this threshold prototype. And as long as we don't get this into locus, I will use my 2 workarounds:

  1. fix saved tracks with database hack custom trigger
  2. get live recording climb elevation data by using experimental sensor data recording calculation with rooted device, busybox utilities and massive amount of calculations and assign it to a headfonejack button with tasker to display the correct current total climb evelation as popup. It would be possible to calculate everything with tasker, but tasker actions are to slow to process in worst case 180000 Lines of csv (5 sensor data lines per secound on a 10 hour tripp) you need stream based tools to handle this fast enoght on a smartphone. For that reason I need a rooted device.

Glad to have 6 hardware buttons on my Z3 Compact:

  • power: to turn display off
  • vol+: zoom in
  • vol-: zoom out
  • camera fokus: toggle screen brightness low - normal - max
  • camera release: toggle dashboard - navigation(restore navigation settings)
  • headphonejack button: display total climb elevation of the day

As MTB rider I need the hardware buttons because if I use touchscreen I can't hit the correct buttons and every wrong input does disable map centering and map rotation. I can fix this now with my camera release button by using locus presets (http://docs.locusmap.eu/doku.php?id=manual:user_guide:settings:presets)

photo
1

The main problem is creating workarround for live recording climb indicator because you can't use database triggers as workarround because live recording did not get stored into database.

And my workarround based on sensor data recording is just totally stupid and even with stream based processng tools very expensive. Would be relly nice to have a proper implementation.

photo
1

Here an example of the current filter problem. Even if your device is very accurate, 30cm in my case, only 0.3m errors if it stays in same altitude.

Example is a static position at 100m above see level messured over time

c6c2f54ddd348264ea909358ef30b4cf

On shurt duration:

  • no Filter +2m
  • normal filter +1m
  • max filter +0.5m
  • threashold filter: +0.3m


On normal duration:

  • no Filter +200m
  • normal filter +100m
  • max filter +50m
  • threashold filter: +0.3m

On long duration:

  • no Filter +500m
  • normal filter +250m
  • max filter +125m
  • threashold filter: +0.3m

You may would say, maxium locus filter is great, only 125m error, but this is only for static position.

If you have a trip with alot of very small elevation changes, where a hill climb only takes 10 to 20 seconds, maximum filtering will delete the 10s to 20s short climbs because it does mix the last 60s to 90s together you end up like this:

  • real: +1000m
  • no filter: +1500m
  • normal filter: +1200m (50m get lost by filter and 250m got added by thousands of 0.3m messurement errors)
  • max filter: +725m (400m get lost by filter and 125m got added by thousands of 0.3m messurement errors)
  • threshold filer: +1050m (0m get lost by filter but 50m got added by error because small altitude changes combined with messurement errors could hit threshold to soon)

As you see here the maximum filter does fail in real world. Some tracks are currect with normal filter others with max filter because you can't calculate how much altitude got removed by filter and how much altitude got added by error. That mean if you don't know how accurate this number is you can disable climb elevation display at all because only SRTM based calculations on saved track will be reproducible.

For that reason I did implement my threshold solution as workarround and didn't wait until somebody does vote for this idea. To geat climb elevation values during my trip.

photo
photo
1

I did work on the 2nd Workarround "read sensor data log for calculating live recording climb elevation"

Here is the main part, the awk script for parsing locus sensor log:


cat 2018-07-07_06-56-47.data 2018-07-07_16-38-00.data | 
awk 'BEGIN{FS=";";min=2000}
$3 == "SI_PR" {
  time=substr($2,0,10); 
  pr=substr($4,0,(length($4)-2)); 
  if(time!=prevTime && prevTime!=""){ 
    if(valueCount>2) { 
      valueCount-=2; 
      valueSum-=min+max}
    print time" "valueSum/valueCount; 
    valueCount=0;
    valueSum=0;
    min=pr; 
    max=pr
  } 
  valueCount++; 
  valueSum+=pr; 
  prevTime=time; 
  if(pr<min) min=pr; 
  if(pr>max) max=pr; 
}' | awk '{
pr=$2; 
if(NR==1) prevPr=pr;
change=pr-prevPr;
if(NR % 100 == 0){
 if(pr>1025) prmult=8.25;
 else if(pr>900) prmulti=9.16-((pr-900)/125)*0.91;
 else if(pr>800) prmulti=10.08-((pr-800)/100)*0.92;
 else if(pr>700) prmulti=11.22-((pr-700)/100)*1.14;
 else prmulti=11.22;
} 
if((isDesc && change < 0.15) || change < -0.15) isDesc = 1; 
else isDesc=0; 
if((isDesc && change < 0) || !isDesc && change > 0){
  change*=prmulti; 
  prevPr=pr; 
  if(change>0) climb+=change; 
  else desc+=change;
}
} END{print climb" "desc}' 
prmulti is the elevation related static value for meters per pressure And 0.15 is my threshold simular to database trigger workarround (0.15 pressure * 9 = 1,35m threshold) Here an example 270km roadcycle tour to leipzig without any major climbs
  • locus DB with trigger threashold, track statistics view: 1529hm
  • locus trigger gpx export: 1623hm
  • awk sensor data result: 1305hm
  • SRTM on gpx: 1557hm
  • SRTM on gpx smoothed: 1311hm
  • reduced gpx: 1500hm
  • reduced gpx smoothed: 1200hm
  • GPSies reduced gpx: 1750hm (I don't know why GPSies does fail on road tours, it's much better in the mountains, recalculation of elevation did change from 1793 to 1750)
  • SRTM on reduced gpx: 1184hm
  • SRTM on reduced gpx smoothed: 940hm

As you see it is very close to smoothed SRTM data. I would say SRMT is the correct result because I did select the test tour based on the small very even hills to get a nerly perfect SRTM result as base.


It's interesting to see that even the 1.0m threshold database trigger is not enoght to get close to the SRTM result. And point reduction with Reumann Witkam and 5m corridor does kill the elevation on road tours because there are many segments which goes straight over many small hills like a huge pumptrack.


I did attach 14MB of Sensor data if somebody want to play with. There are 300 000 Lines. And the awk scripts does parse and calculate all of them in just 7 seconds on my sony phone. (5 seconds if you merge both awk scripts and skip the whole print and parse step between them)


If somebody ask why I don't use smoothed SRTM if this is the truth: SRTM does only work on small elevation changes far away from steep trails or even canyon like things. I would to get flat country SRTM accuracy in the middle of the mountains with canyons and trails on the edge of the mountain. And this is only possible with barometric values.

photo
1

Here is my tasker task for calculating weather related climb and descent errors

It does need %SENSORS_DIR = /sdcard/Locus/data/sensors

and %DARKSKYAPI = free api key for dark sky weather

photo
1

And here is my Tasker Task for calculating elevation based on sensor pressure (does even work if you turn GPS off) and just for fun a weather data correction of the elevation based on first GPS position and recording timespan :D

I did add pointsPerSec and maxTimeGap to my awk sensor data parser to track if I lost any sensor data to give me the opportunity to judge the output elevation accuracy. For example in a deep power saving mode I get sensor readouts every 5 minutes which could be accurate enoght if the max time gap between 2 values do not exceed 5 minutes


Don't forgett

%SENSORS_DIR = /sdcard/Locus/data/sensors

and %DARKSKYAPI = free api key for dark sky weather


AWK script optimizations did make alot of fun, it was a little bit difficult to find out how to get the first data line (2nd line) without dooing 300.000 times if(linenumer == 2)