Locus Action extract doesn’t support large files

develar shared this problem 3 months ago
Solved

Please see http://forum.locusmap.eu/index.php?topic=4036.msg51709#msg51709


I have enough free space, enough RAM (6 gb), but Locus cannot execute action locus-actions://https/s3.eu-central-1.amazonaws.com/gh-routing-data/2018-06-18/de-at-ch.locus.xml (see list https://graphhopper.develar.org) to install routing data.


Manual downloading and extract works for me, so it isn’t a server issue.

Comments (16)

photo
1

Good day develar,

I have quite a hot moments with this issue today.

Problem is that Android library that handles zip files, has some issues with it. The result of open of such huge file is: ZipException: error in opening zip file (simple during call new ZipFile(...)). Unfortunately, I'm unable to find out any method how to bypass this issue. As I try it with "Solid explorer" app and it has no problem with this file. Most probably different zip library.


Is option for you to split files to most probably smaller than 2 GB (expect problems of this size)?

Jiří M. aka Menion

photo
1

> Is option for you to split files to most probably smaller than 2 GB (expect problems of this size)?


It is not an option for users because GraphHopper doesn't support graph merge. https://github.com/graphhopper/graphhopper/issues/293


Well, personally, I live in Bayern, that's why I have prepared special graph "Bayern (Germany), Austria, Czech Republic" (1.15 GB). But for users it will be configuration hell — need to clearly understand what to download and so on.


Maybe Locus can support `tar.gz` files? Or, and it will be awesome, `tar.zstd`? Or maybe 7z (quick google search https://github.com/hzy3774/AndroidUn7zip)?


For me ideally will be support `tar.zstd` because ZSTD decompression speed is very high.

photo
1

ZSTD for Android — https://github.com/luben/zstd-jni (tar is required for zstd because zstd supports only one file, so, need to tar and then zstd).

photo
1

Hmm, not a solution I would like to see. Link C++ library just because of this rare case when we need to download a huge data file, hmm.

I wanted to write Peter Karich if there is any news about this "problem". Seems that your topic is almost a year without an answer and because I know that Peter already uses Locus for a field testing, maybe we may push it a little bit ...


Jiří M. aka Menion

photo
1

Thanks. Actually, proper implementation — do not create temp file, but extract on the fly. But ZIP format doesn't support pipes...

On page http://docs.locusmap.eu/doku.php?id=manual:advanced:customization:actions I see mention of "tar". So, tar supported? Only tar or tar.gz?

photo
1

Answer from Peter is negative > still no time to do something with this limitation (more routing data sources handled at once).

Extract of ZIP during download is really not possible, yes extraction from InputStream does not work on Android. And even if, problem is not download of huge file, but open of this file by ZipFile object. Solution should really be only some kind of external library, like you mentioned Un7Zip.

And TAR: it is my really really old own implementation, that is currently usable only for a special old TAR map format.


Complicated task. I see possible alternative in Apacha Compress library ( http://commons.apache.org/proper/commons-compress/ ). Apache is anyway usually quite a big bug when it comes to size of APK.

I see one more alternative: locus-actions may have defined more files in a pack. What if you split your huge files into more smaller zip files and Locus extract them all into same directory? Should work.


Jiří M. aka Menion

photo
1

> What if you split your huge files into more smaller zip files and Locus extract them all into same directory?


In this case we need actions: "empty dir". e.g. in the XML will be sequence of download actions, but we need action to prepare dir where files will be extracted. If user already have dir downloaded, we need to empty it to ensure that no old files. Is it possible to implement such action (for security, limit it to be under some dir, e.g `/mapsVector/`)?

photo
1

Isn't situation same now as well? When you now download data, you expect that new extracted files overwrite old files. Here it is exactly same.

Option to clear directory before extract is currently not implemented and as I wrote, it looks to me little risky to allow it globally and little limited to allow it only for mapsVector.

If this is not a huge problem for you, please give it a try and for example manually split this huge file into three smaller and set all files into XML files. I'm really sure, it should work without a problem!

photo
1

> Isn't situation same now as well


I was under impression that Locus overwrites dir on extract. Ok. In general it can leads to strange bugs (stale files), but as it is not possible to implement proper solution at this moment (I fully understand "just because of this rare case"), I will pack huge dir as series of small zip files. Thanks!

Maybe later I will implement special app to manage such data, since ZSTD will allow me to greatly reduce traffic and avoid temp files (direct extract). And implement multi-download to reduce latency effect to zero. Users can use existing apps to do it, but it is tedious and error-prone (where to extract, remove old data).

photo
1

Understand. I can imagine improve current mechanism and allow for example some kind of user interaction (like confirmation "Delete directory mapsVector/gh-data?" etc.

A special solution should be of course ideal solution, it depends just on you.

As first, please let's try this solution. If it will work and you will be interested in some improvements in these "Locus-actions", I'll gladly work on it.

photo
photo
1

There is also problem downloading and install maps from openandromaps. I had to use Solid Explorer to extract the Germany map to a Samsung Galaxy S8+ and an Ulefone Power 3s, but Locus Map was able to extract this map on a Samsung S7.

photo
1

Hmm also some huge maps, bigger than X? GB?

photo
1

2.86GB unzipped

photo
photo
1

Hi,

I kindly wrote two weeks ago to Solid explorer developers if they will be willing to share their knowledge about this problem, but no response so far, probably company secret.

Were you able to bypass this issue by splitting content into more zip files?

photo
1

Hi, yes, it works. Sorry for lack of confirmation.


https://github.com/develar/gh-routing-data/blob/master/scripts/locus-action-generator.js#L45 Simply, if `shortcuts_fastest_bike2` file size > 600MB, it is indicator that result zip file will be larger than 2GB and so, should be splitted.

photo
1

Ah perfect. At least this workaround works. Thanks for the information.

photo