Precise SRTM .hgt file handling

Good day Markus,

if you have data prepared, best is to give it a try. I think it should work (and if not, create a bug report and attach one file for test). Only thing I'm little worried is fact, that app load necessary HGT files directly into memory for a lot faster work. 1m files will be probably really huge so there will be need for some automatic optimization and disabling this system. Anyway give it a try and you will see, I've personally never tried/used such huge HGT files.

Ok, Thanks for fast Answering

I use GlobalMapper to export my Data to the .bil Format. (Motorola BigEndian Byte-Order) rename it to .hgt

Delete the corresponding File that Locus downloaded and replace it with my "new" one.

When I export my Data as .hgt -File (SRTM 1") i get it working but with the standard resolution. And it seems also to be very close to .bil format. (I am just able to export as .bil in custom Resolution)

I think i should keep the square Area of the Data so that Locus maybe is able to use it. (I don´t know how Locus parses the Data).

So I give it a try the next Days.

I'm not sure if simple rename will work, you will see. I remember I worked with bil format some time ago, but it's direct internal support was removed few years ago.

App should anyway work with all HGT files, no matter how precise they are.

It is working when i export as .bil file with same Resolution than 1 arc second. (so export as .bil and just rename works)

So it seems now that locus is just using one of the standard resolutions for srtm-files and can´t handle other resolutions.

Maybe if we say all files contain Data for 1 square arc second tiles. So we are able to calculate the point-density.

And locus load just a maximum count of opints to working memory (ram) ? to save memory.

My task is to use it in a mountain Rescue Team, and when we go to unknown terrain sometimes it helps to see in high detailled height model where are holes or steep rocks. or get back height values.

OK i can load a Hillshade-Map with high detailled Raster hillshade. but then i don´t have the right height data.

Good day Markus,

I'm checking implementation of this feature and for me it should work, no matter how big are HGT files.

May you please share one file with me for test? Thank you.

OK, yesterday i tried again. I exported as 0,1" grided Data. But Locus can´t work with it well.

If i Export as 1" grided Data then it works fine. (same method of Export as .bil, then rename to .hgt) I try to Attach a File with 0,5" Resolution. I hope the Size is not too big.

Are you sure Locus don´t just decide if Data is 1" or 3" ? Because in .hgt Files there is just a Grided Data, not the coordinates for each point. So Locus have to know how the Resolution of the Data is. But Locus also don´t work with Header Files... so... How do Locus find out the Resolution ?

For me Locus is really the best mapping app on Android, so i hope my task is solveable. Now I try to generate 0.5" grided Data and attach. I don´t calculated but i think the Filesize is around 200MB for a 1 degree cell. I try it with my Region N47E009.

Good day Markus,

our help desk most probably refuses upload of such huge file, so please use any Google Drive or similar service, thanks.

And app detects resolution of HGT file thanks to filesize. Size has to be always the same for the same resolutions.

ok, i found out where the Mistake was..

My Data is in Meter accuracy, so exported them to grid resolutions for example: i tried 3m to 3m grid resolution.

But Locus Needs a binary dividable raster resolution of 1 arc Second. So possible is: 0.5 arc second grid spacing, 0.25" spacing, 0.125" spacing, 0.0625, ..., so ist not possible to make exactly 10m Resolution, but something close to.

So now it is working fine. thanks. I get really precise heights and also a Hillshade. (I see little rivers and also roads on Hillshade)

My next Task is to find a Software that generates better .bil data. (=.hgt data with Motorola Byte Order)Because Global-Mapper start produce Little stepps when use resolutions finer than 0.25 arc seconds as Export.

Or Maybe i let this data at a Resolution of 0.125" (close to 3m Resolution) and use a Map-Overlay (Multiplied) with Grayscale shaded Image of Hillshade. (I also use this on my Garmin Device).

You found a problem with your generated files: perfect!

I'm glad it works for you now. If there will be something I may help with, feel free to ask.

Hello

Now i have a question. When i plan a new Route manually (line of sight). And then look at the routes Height- Graph. Locus do not "interpolate" along the Route-Line, while viewing the graph ? Locus just usrs heights of the route points ?

Is it possible somehow to get a path profile ?

I think Garmin is doing it like: when you are in path-profile window, they "interactive" take tighter height points "below" the line of sight, while you zoom in.

Ok, every 100m a point, so Locus don't show meters deep stream ways, when not hit by the 100m raster. Is there possibility to improofe this ?

Understand. Quite a specific task, but let's try it ...

• Expert settings
• New option "Route, interpolation step" in next Beta version

Thanks for sharing of your shaded SQLite map and HGT file. Both downloaded and correctly displayed in app. And now what to do with it :).

Firstly, a quick test in route planner with visible SQLite map and drawn short profile show over some huge terrain changes show no problem to me.

The application uses bicubic interpolation from 16 interpolated points around the required location. In this interpolation is most probably not an issue because otherwise, it will be clearly visible. So the problem really may be in some shift of computed row and column indexes of elevation values.

The whole algorithm is little complex and because if there is an issue, it is really tiny, I cannot invest a day to finding exact reason of possible small shift of interpolated elevations, sorry to say it. We anyway planned to optimize shading in the app in the near future, so I'll keep this in my mind! Thanks for understanding.

Thanks, for me Locus is a really good app. And maybe I manage somehow to shift my .hgt Data some Meters.

I see the Shift best when i plan a manual route, open Route height profile and tap on the Top of a falling Edge, then i see the cursor shifted some meters away from the cliff. Also when i plan a auto-route along a street. (The Street is exactly drawed to the lidar data in openstreetmap). Or by just gwtting heights from cursor position near a cliff.

Hello,

I tried what you describe: "plan a manual route, open Route height profile and tap on the Top of a falling Edge". I tried this maybe on five places and on all it looks correctly. Give me please some start > end coordinates of line, where this will be clearly visible, thanks.

Hello

I work with disabled Hillshade but enabled Overlay "ESG-Hillshade" multiplied mode. (100% non Transparent).As Map I use "Freizeitkarte ALPS+East" or also LoMaps-Austria.

I Start a Manual Route...

Start at Position : N 47°18.209' ; E 009°40.001'

End at Position: N47°18.172' ; E 009°40.022'

Then I tap on the Height-Profile Icon (bottom-right Corner).I tap on the middle of the first flat Region above bottom. (middle of the "Road")

The Cursor on the Map is then shifted. (because the Height-data are shifted to top-left. (About 5.5m to Top and 3m to Left(nearby 0.125 arc-Seconds = the Resolution of my Height-Data = one Pixel)

Or I see it better when I enable the Display of Height-Values at Cursor-Position. And move to an Edge, then the Values start rise at shifted Position.

Now I shifted just my .hgt Data (my Hillshade-Map "ESG-Hillshade" is the same as bevore).Its much better now. For the shifting i had to download the N47E008 Tile and the N48E009 Tile and shift them all. Then cut the Data to the exact Window of the original N47E009. (so I got additional Data at Top and Left but cutted Data from Right and Bottom = shifted Height-Data to Bottom-Right because Locus takes the new Data as N47E009)

I uploaded the New Data to my Google-Drive.

https://drive.google.com/file/d/1QYYcqnRP55ywqjw8W_X2IvbamoTSiXcc/view?usp=sharing

Thanks Markus for the precise description ....

So after maybe two hours of testing (and half of hour of sleep after falling asleep during thinking :) ), I have this result for your two coordinates.

Better right? :). Seems to match your generated map a lot better.

Ah back, sorry. This my "fix" was incorrect and instead, it created another issue.

We here compare the result of Locus Map vs result of some program you use and I tried to "fix" Locus Map to match data from you. Is there any other option how to test, which program gives correct results? Because I'm just unable to find a mistake in my old algorithm. All seems to match!

Hm, yes. There is a projection problem. Its not just a shift. I exported with exactly x and y square 0.125 arcsecond resolution. And a wgs84 projection. I try to check this in my projection header files. The Thing is: when i open both Data in the professional GIS Sodtware it fits cm exact. (Hillshade and .hgt) But Difference is: my GIS Software uses the additional header files. And locus calculate from Filesize

So i check what is inside the additional Files

Ok, found a problem in my Data. I used the bounding-box from the original 1" .hgt file, to cut my Data to. But with my higher Resolution, the box don't hit the right points. So my .hgt had additional data, and locus did a stretch or push

Solution was: to cut Data to exact N47 and N48...

Oki ... so at least I've confirmed that my algorithm is correct.

Menion

Thanks for the Work on Locus. And sorry. My GIS-Software did open the .hgt-File always with the additional Projection files. So there it looked perfect matched. And my Idea to use the Bounding-Box from an original 1" .hgt File to cut my Data, was a mistake.

Hi Markus

I'm also working in the mountains. I've tried to export to BIL in globalmapper, but cant get it to work. It produces a file of size 1kb. No problem exporting directly to the hgt format at 1". And results in that resolution based on DEM from norwegian mapping authority is better than the autodownloaded 3".

But do you have any idea why i cant export correctly in BIL (tried 0.25 0.5 and 1")?

Maybe you have developed your recipe further? Please share :)

Not sure if it helps, but here are the Norwegian LIDAR data in different formats, compiled and provided by Sonny: https://data.opendataportal.at/dataset/dtm-norway

You can download them separately and move them to the right place in the Locus folder structure, or even leave them in separate folder and adapt the MISC setting in Locus for SRTM.

Cheers

Michael

And you Try to use the 1m DTM Data of Norway ? Then it would sure give super Results. At moment i am Downloading Data from Norwey in 1m ground Resolution. Cool that in Norway such high resolution Data is also available.

Høydedata (hoydedata.no)

https://hoydedata.no/LaserInnsyn/

Yes thats the data im using. Until now its only lower parts that is in that resolution, plus some mountain/glacier areas. In a year or two the whole country should be in 1m resolution. Its very cool. I've been using the LIDAR clouds with vegetation to make ski maps of the forest. The big pine trees have a very clear signature....

I did not have anymore time this weekend, but i will try again the next couple of days. For now i'm just trying to get it to work - the whole conversion thing.

Hillshade - i grew up without it, but on my maps with high res slope-steepnes shading in green yellow red brown and black contours the shading really helps getting the 3d feeling. And i'm really excited about how it will work for skiing with the 1m resolution. Maybe its to much info. When it comes to slope-steepnes shading i find a 4m horisontal resolution the best.

Would be great with a video or some other step my step recipe. I tried follw the one you gave, but must have done something wrong along the road.... For now im just trying with the 10m resolution...

Hello,

In France, since 1st of January, the IGN DEM at 5m resolution is available freely. And the DEM at 1m will be available in march.

Not a problem to use it in a lot of applications (possible to do some conversions or transformations with QGIS).

But, how to use it in Locus Map ? I understood that Locus accept only HGT files with 1" resolution, is it true ?

Best regards,

Hi Bernard

Its possible to use higher res in hgt files, as long as they are 0.5", 0.25", 0.125" and so on.

Its all in the discussion here, somehow.

We tried some different mixes of software and approaches, and it would be nice to see if we could get it working in qgis, but i'm just in the beginning of getting familiar with qgis,....

Hi guys,

nice discussion. Just from my point of view: Locus Map does not force HGT files to be in 3601x3601 grid. Grid is determined on the fly from the file size. If some files won't work, just send me it for the test. I'll analyze and wrote back where is a problem.

Menion

Hello,

I've tried to make some highres DEM "pseudo"HGT : I discovered that in fact, Locus needs only square grid, no matter the size. Mine is 16423x16423 (i.e. about 0.2192"), and it works fine ! Not necessary to be exactly 0.5, 0.25, ...

I used QGIS to generate a BIL file (Int16 little indian BIL, QGOS can't write directly big endian HGT for that size), and the Unix "dd conv=swab" command for byte swapping.

Works great !

Bernard

Thx. Will try that in Qgis :)

Unix "dd conv=swab" - i need to look into, But i have a linux server i might do that on. Totally noob when it comes to byte things..

If you have a BIL in INT16 big endian, the exact command is like that : "dd conv=swab < your-dem.bil > NxxEyyy.hgt".

For France, the hires DEM are in format ASC, and Lambert93 projection. So, I'm using QGIS to merge the ASC for a grid, reproject in WGS84, extract the exact square grid and generate the BIL.

The most difficult part is that a HGT file can't be recalibrate. So it is necessary to be very precise when extracting the grid. Recently, Sonny generates new HGT for France, using these IGN data (standard 1" HGT), but most of them are not correctly aligned, so, cells are shifted by one or two... it means between 20m and 40m shifted (E or NE in general). And precise data, but not correctly calibrated... not so useful...

Bernard

Hi,

why do you think my new DTMs of France are shifted? Can you provide some examples with screenshots. If you want, you can directly contact my by e-mail to prove your statement.

Hi,

Here are some examples :

There is no problem with your 20mx20m TIF files.

Bernard

I've checked the correctness of reprojection and resampling of my new HGT-DTMs of Fracnce. And to be short: They are absolutely ok, Reprojetion as well as Resampling has been done correct. There's no systemtic shift of the new HGT files compared to the source data of IGN France.

Since Bernard Perrot accused me that my DTMs are "not correctly calibrated" and shifted, without contacting myself before to clear his accuse, I want to publicly falsify his statement. Usually these explanations are too sophisticated for the Locus forum and I dind't wanted to defend the correctness of my DTMs in public, but anyway, here we go:

1)

If you want to juge the positional correctness of DTMs you have to look for a very good reference to compare with. This can be good positioned Orthophotos where we can identify appropriate objects to compare with (summit cross, streets, river banks etc.), or survey points with surveyed coordinates and altitude.

In general it's better to use a region in the flatlands and its objects (streets, Riverbanks,..) to compare with, than a steep mountain ridge - as provided by Bernard in his examples. Reasons: Orthophots of steep ridges are very hard to rectify correctly for the organisations providing these Orthophotos. Not each peak on the Orthophoto is exactly where the real coordinates of the peak is. I don't want to go into detail of the reason, you can ask professionals who will explain it to you.

Further: Usually LiDAR Source data is finer, more accurate in the lowlands than in steep mountainous regions, especially if the mountains are coated with the snows of glaciers. This fact is also true for IGN 5m. So if you want to prove the correct position of a DTM - better use something in the lowlands!

And of course use the correct position of the summit as reference ;-) Look at Screenshot 1. Bernard's "summit" of Puy Mary is about 50 m west of the real summit. The real summit is marked by an IGN survey point Nr. "1510201 - A" (elevation = 1784.53m). Also watch the blue "lookout" icon just above the survey point which is a optical reference for the summit in the following screenshots.

2)

Check the correct postion of the source data.

That means: did I use the correct projection to import the source data? (for IGN 5m France it is EPSG:2154, called "RGF93 / Lambert-93")

If you look at Screenshot 2 you can see the imported modell of the IGN 5m Source-data. Each colored layer represents an altitude level of 1m, so the outside blue equals 1762m and less, up to 1783m on its highest level where the arrow points. You can see that the highest point of the source data is next to the position of the underlying blue "lookout" symbol, being near the survey point of the peak. So the import of source data is correct.

3)

Reprojection and Resampling.

Finally the Sourcedata hast to be reprojected and resampled to create the 1" files of DTM France 1". This means on predifined, fixed coordinates, which are about 20 x 30 m away from each other, the elevation values of the source modell are taken into the final 1" modell. Have a look at Screenshot 3: Underlying the source modell of Screenshot 2, and above the rectangular 1"-HGT files of the final DTM 1". The elevation of the source data on the CENTER of a DTM 1"-HGT cell equals exactly the elevation of the whole cell. That's the simple magic of resampling.

This is why the highest cell of the final 1" can not always be exactly at the postion of the real summit of source data, but somewhere outside. In the example the highest hgt cell is northwest of the real summit with an elevation of 1778m - which is a correct sample of the underlying "1778.189 m" of the source data. The job of resampling is NOT to reproduce exact locations and elevation of peaks - the job is to keep the exact elevation in the CENTER of the HGT-cell.

Important: the finer the DTM, the better in can reproduce the form, postion and elevations in steep, rocky terrain. That's why I advice anybode to use 1" instead of 3" DTMs if they are used outside the lowlands.

100% Right. And also with 1" resolution (~30m at Equartor) the Middlepoints often dont hit 5m wide Rivers (canyons) or cliffs at Mountains. So i decided or "need" to use at least 0.125" resolution for height data in Locus.

(height data "just" necessary for getting height values or Profiles) 0.125" are roundabout 3 m Resolution at my Region.

And For additional resolution in Visualisation i decided to export highres data as Raster Hillshade Map with ~0.5 m Resolution.

There i get nearby all small cliffs visible exactly. And also paths in Forested Regions get visible.