This object is in archive!
SQLite maps compression... png vs jpeg
Answered
Are Locus sqlite map tiles always PNG compressed? Or is there also a JPEG option somehow?
I am asking because decoding JPEG on Android might be orders of magnitude more efficient than decoding PNG. To my knowledge, JPEG decoding is hardware accelerated on most Android devices, while PNG (zlib/deflate) is not.
The same question
OTOH, solid computer graphics like maps are better to be compressed by lossless PNG than lossy JPG.
The JPG compressed graphics of the similar size have often around sharp edges JPC Discreet Cosine Transform artefacts.
OTOH, solid computer graphics like maps are better to be compressed by lossless PNG than lossy JPG.
The JPG compressed graphics of the similar size have often around sharp edges JPC Discreet Cosine Transform artefacts.
This is mainly about speed & battery usage, not so much about compression artifacts. I would easily trade some funky pixels for a 50% file size reduction, a 300% speed increase and 30% less battery usage :-). Plus I guess the jpeg artifacts wouldn't really be visible at today's obnoxiously high dpi screens.
Anyway... the whole idea came up due to a recent discussion with an Android programmer, who mentioned that reading JPEG was super-fast and very battery friendly compared to PNG because of hardware decoding support. I do not have any hard evidence for this, but I thought it might be something worth looking into.
Another thing to consider: a JPEG tile might be half the size of a PNG tile (with almost invisible artifacts). Apart from decompression speed, Locus would also just have to read half the bytes from the sdcard / internal memory. If this is a bottle neck, things would improve majorly as well.
Only menion can tell where actual time & cpu cycles are spent though, maybe this is all bullshit. But maybe it isn't... then sqlite maps could be sped up a lot with very little effort :-).
This is mainly about speed & battery usage, not so much about compression artifacts. I would easily trade some funky pixels for a 50% file size reduction, a 300% speed increase and 30% less battery usage :-). Plus I guess the jpeg artifacts wouldn't really be visible at today's obnoxiously high dpi screens.
Anyway... the whole idea came up due to a recent discussion with an Android programmer, who mentioned that reading JPEG was super-fast and very battery friendly compared to PNG because of hardware decoding support. I do not have any hard evidence for this, but I thought it might be something worth looking into.
Another thing to consider: a JPEG tile might be half the size of a PNG tile (with almost invisible artifacts). Apart from decompression speed, Locus would also just have to read half the bytes from the sdcard / internal memory. If this is a bottle neck, things would improve majorly as well.
Only menion can tell where actual time & cpu cycles are spent though, maybe this is all bullshit. But maybe it isn't... then sqlite maps could be sped up a lot with very little effort :-).
I agree about speed/battery.
AFAI remember, MOBAC allows via some Java modules to reencode the tiles to JPG, even with given compression coefficient. The result can be then stored in RMAPs SQLite target format, that is exactly the SQLite format used in locus ( As there are several different map formats withing SQLite DB )
I agree about speed/battery.
AFAI remember, MOBAC allows via some Java modules to reencode the tiles to JPG, even with given compression coefficient. The result can be then stored in RMAPs SQLite target format, that is exactly the SQLite format used in locus ( As there are several different map formats withing SQLite DB )
I prefer WebP which is higher compression than JPG
I prefer WebP which is higher compression than JPG
If someone is able to prepare me two identical files, I offer my service to some more precise measurement directly on various devices. I'm not aware about some precise numbers, how fast and efficient is PNG vs JPEG image handling on Android, so best is test it in real I think.
If someone is able to prepare me two identical files, I offer my service to some more precise measurement directly on various devices. I'm not aware about some precise numbers, how fast and efficient is PNG vs JPEG image handling on Android, so best is test it in real I think.
http://stackoverflow.com/questions/16895036/why-png-compression-is-that-much-slower-than-jpeg-on-android
is somehow related... it only talks about compression instead of decompression, but apparently thats ten times faster for hw accelerated jpg vs PNG. Could also be an indication for decompression speeds.
Apparently it depends on how you read the jpegs though. Using something called SKIA would benefit from hw accel automatically.
http://stackoverflow.com/questions/16895036/why-png-compression-is-that-much-slower-than-jpeg-on-android
is somehow related... it only talks about compression instead of decompression, but apparently thats ten times faster for hw accelerated jpg vs PNG. Could also be an indication for decompression speeds.
Apparently it depends on how you read the jpegs though. Using something called SKIA would benefit from hw accel automatically.
Hmm compression in Locus never happen. All data that are stored into database are exactly same bytes, that are received from web map servers. Except cases, when Locus merge few layers into one of course, but this happen rarely. So hard to say if here will be an significant difference. Even if so, it should only helps people that create some own maps somewhere, because I won't change images to other format after download, because of this. They have to remain in original state. In 100% map scale, problems of JPEG won't be mostly visible, but in higher map scale, ghosts will appear for sure.
Hmm compression in Locus never happen. All data that are stored into database are exactly same bytes, that are received from web map servers. Except cases, when Locus merge few layers into one of course, but this happen rarely. So hard to say if here will be an significant difference. Even if so, it should only helps people that create some own maps somewhere, because I won't change images to other format after download, because of this. They have to remain in original state. In 100% map scale, problems of JPEG won't be mostly visible, but in higher map scale, ghosts will appear for sure.
Maybe... for quick testing... you could simply supply an empty bitmap to your map rendering instead of calling the PNG decode routine? And only read half the bytes from sd? And then see how much faster that is compared to now? That would be the maximum possible gain by using jpg. If you find the difference to be really major, then maybe offering a builting re-encode tool for sqlite png maps might be smart.
Maybe... for quick testing... you could simply supply an empty bitmap to your map rendering instead of calling the PNG decode routine? And only read half the bytes from sd? And then see how much faster that is compared to now? That would be the maximum possible gain by using jpg. If you find the difference to be really major, then maybe offering a builting re-encode tool for sqlite png maps might be smart.
Also... how does Locus implement caching of mapsforge tiles? Do they get turned into png? If they do, it might totally be worth changing that to jpg for testing and measure the results. If there is really a factor of ten in encoding/decoding speed as well as significantly less data writing/reading, it might give a whole new insight on when and what to cache.
Also... how does Locus implement caching of mapsforge tiles? Do they get turned into png? If they do, it might totally be worth changing that to jpg for testing and measure the results. If there is really a factor of ten in encoding/decoding speed as well as significantly less data writing/reading, it might give a whole new insight on when and what to cache.
I have read up a few things on android graphics performance meanwhile and the first suggestion everybody has is to use JPG instead of PNG wherever possible. Most people agree that JPG is done mainly in hardware whereas PNG is completely handled by cpu.
Secondmost suggestion is to use RGB_565 instead of ARGB_8888 when transparency and high color accuracy is not needed. Saves on memory and also increases speed further. I would assume RGB_565 to be "good enough" for sqlite maps or vector map caches.
I have read up a few things on android graphics performance meanwhile and the first suggestion everybody has is to use JPG instead of PNG wherever possible. Most people agree that JPG is done mainly in hardware whereas PNG is completely handled by cpu.
Secondmost suggestion is to use RGB_565 instead of ARGB_8888 when transparency and high color accuracy is not needed. Saves on memory and also increases speed further. I would assume RGB_565 to be "good enough" for sqlite maps or vector map caches.
I rest my case for JPG. After learning a few more things, it appears there are much better formats when speed is of any concern. ASTC comes to mind... this is a very interesting read: http://android-developers.blogspot.de/2015/01/efficient-game-textures-with-hardware.html
Bascially, if Locus did what games do with their textures, it wouldn't have to "decompress" anything at all. You simply read the raw ASTC data from disk and pass it to the GPU without any change whatsoever. The data is never actually "unpacked", the GPU can render ASTC directly. No cpu cycle is ever involved.
I believe a sqlite map (or mapsforge cache?) with ASTC tiles would run circles around the current jpg/png implementation and save shitloads of battery as well. No idea how doable this is, but I would love to see the difference :-).
I rest my case for JPG. After learning a few more things, it appears there are much better formats when speed is of any concern. ASTC comes to mind... this is a very interesting read: http://android-developers.blogspot.de/2015/01/efficient-game-textures-with-hardware.html
Bascially, if Locus did what games do with their textures, it wouldn't have to "decompress" anything at all. You simply read the raw ASTC data from disk and pass it to the GPU without any change whatsoever. The data is never actually "unpacked", the GPU can render ASTC directly. No cpu cycle is ever involved.
I believe a sqlite map (or mapsforge cache?) with ASTC tiles would run circles around the current jpg/png implementation and save shitloads of battery as well. No idea how doable this is, but I would love to see the difference :-).
...or ETC1 instead of ASTC... slightly less efficient but available on basically all Android devices since godknowswhen:
GL_OES_compressed_ETC1_RGB8_texture
Compressing on the fly for a possible "convert sqlite map to accelerated hardware textures" menu item: http://developer.android.com/reference/android/opengl/ETC1Util.html
Time to do some OpenGL-Locus :-).
...or ETC1 instead of ASTC... slightly less efficient but available on basically all Android devices since godknowswhen:
GL_OES_compressed_ETC1_RGB8_texture
Compressing on the fly for a possible "convert sqlite map to accelerated hardware textures" menu item: http://developer.android.com/reference/android/opengl/ETC1Util.html
Time to do some OpenGL-Locus :-).
Nice tries, but Locus do not use OpenGL for now. I'll try to remember and test some JPG/PNG options when I'll have some free time later.
Nice tries, but Locus do not use OpenGL for now. I'll try to remember and test some JPG/PNG options when I'll have some free time later.
Caching of vector tiles at level 12 (current Locus version):
- PNG compression
- average load: 12ms per tile
- average save: 48ms per tile
- average size of tile on SD card: 55kb
Testing based on this discussion
- JPEG/80% compression
- average load: 13ms per tile
- average save: 26ms per tile
- average size of tile on SD card: 17.8kb
Quality is little bit worst, but difference is absolutely not simply visible at 100% scale, maybe later little bit. Size is almost 3x lower and storing speed (faster creating of byte[] + smaller size of result) is also 2x faster. Results are usable, thanks.
In case of vector maps, caching is still enabled only till level 12, shading and auto-loading is disabled. Why this - it belongs to different, already discussed, topic. I'll anyway increase cache to level 13 as this is another worst level for rendering and because of reduced size of tiles and saving speed, it will worth it.
Thanks for a tip, usable now and probably also in future.
Caching of vector tiles at level 12 (current Locus version):
- PNG compression
- average load: 12ms per tile
- average save: 48ms per tile
- average size of tile on SD card: 55kb
Testing based on this discussion
- JPEG/80% compression
- average load: 13ms per tile
- average save: 26ms per tile
- average size of tile on SD card: 17.8kb
Quality is little bit worst, but difference is absolutely not simply visible at 100% scale, maybe later little bit. Size is almost 3x lower and storing speed (faster creating of byte[] + smaller size of result) is also 2x faster. Results are usable, thanks.
In case of vector maps, caching is still enabled only till level 12, shading and auto-loading is disabled. Why this - it belongs to different, already discussed, topic. I'll anyway increase cache to level 13 as this is another worst level for rendering and because of reduced size of tiles and saving speed, it will worth it.
Thanks for a tip, usable now and probably also in future.
OK... so your jpg results... basically... loading time is equal... size is reduced to 1/3... compression is way faster (probably some hw accell magic).
When thinking about downloaded sqlite maps, it means that we could reduce storage requirements by 70%, but would not gain speed nor much battery.
I know you aren't really in love with downloaded maps. Anyway, it would still be worth thinking about an on-the-fly re-coding to JPEG during download and/or a "convert" menu item for offline maps. 70% size reduction without too much effort and no negative side effects is not something to ignore lightly.
OK... so your jpg results... basically... loading time is equal... size is reduced to 1/3... compression is way faster (probably some hw accell magic).
When thinking about downloaded sqlite maps, it means that we could reduce storage requirements by 70%, but would not gain speed nor much battery.
I know you aren't really in love with downloaded maps. Anyway, it would still be worth thinking about an on-the-fly re-coding to JPEG during download and/or a "convert" menu item for offline maps. 70% size reduction without too much effort and no negative side effects is not something to ignore lightly.
ps... another thought... do you cache the calculated hill shading tiles as well? JPEG might be ideal for this: You could easily go for a 10% quality and minimalistic result size without any visible artifacts. And if its 12ms (jpeg tile load) vs 50ms (hill shade rendering), it might be totally worth it.
ps... another thought... do you cache the calculated hill shading tiles as well? JPEG might be ideal for this: You could easily go for a 10% quality and minimalistic result size without any visible artifacts. And if its 12ms (jpeg tile load) vs 50ms (hill shade rendering), it might be totally worth it.
After the discussion here: https://forum.locusmap.eu/index.php?topic=6471.125, this should be revisited, also in light of newer technologies. For example, WEBP might be better than PNG and JPG, as it's developped/endorsed by Google. It would even be possible to offer user choice of lossless/lossy while using the same format.
After the discussion here: https://forum.locusmap.eu/index.php?topic=6471.125, this should be revisited, also in light of newer technologies. For example, WEBP might be better than PNG and JPG, as it's developped/endorsed by Google. It would even be possible to offer user choice of lossless/lossy while using the same format.
Hello Ingo,
thanks for re-opening this topic.
Which map you used, that have small ghosts as you wrote about?
I created in the application support for set a image format and made a test. Here is the result for downloaded "MBT Europe", zoom levels 15 - 19, area selected as "By Country > Czech Republic" (so easily reproduced).
"orig" means directly saving original map data. In case of this map, MBT Europe seems to use optimized PNG images.
As I wrote this change, for my surprise I noticed that except "Kompass.de" and except some multi-layer maps (where Locus Map after download merge multiple map tiles into single tile), always was used this "orig" system. Hope I looked correctly.
Based on this test, for MBT Europe, any kind of compression is quite useless as map tiles are already well optimized on map server. And test every online map? Uff ...
Hello Ingo,
thanks for re-opening this topic.
Which map you used, that have small ghosts as you wrote about?
I created in the application support for set a image format and made a test. Here is the result for downloaded "MBT Europe", zoom levels 15 - 19, area selected as "By Country > Czech Republic" (so easily reproduced).
"orig" means directly saving original map data. In case of this map, MBT Europe seems to use optimized PNG images.
As I wrote this change, for my surprise I noticed that except "Kompass.de" and except some multi-layer maps (where Locus Map after download merge multiple map tiles into single tile), always was used this "orig" system. Hope I looked correctly.
Based on this test, for MBT Europe, any kind of compression is quite useless as map tiles are already well optimized on map server. And test every online map? Uff ...
Thx, Menion, for tring that out. I think I wrote in my original post that it was more a gut feeling - perhaps I'm wrong. I was comparing downloads of Google Classic with SwissTopo. The thing is, SwissTopo has much less texts than Google, and those are most likely to show problems with JPG artefacts.
I also made a screenshot of a Google Maps screen. Zooming into it, it's all much clearer than a similar screenshot of Locus. Then again, it's hard to compare, as Google Maps obviously allows zooming in deeper than the download API. And as it's vector in the app, it's naturally much nicer looking.
Now, did I understand that correctly: Locus doesn't convert/compress the tiles received from the server, or does it? Because if Google delivers the maps like this, well - out of luck.
One more thing: No matter who compresses it, I see one potential problem with Google Maps and JPG: All texts have a very thin white border. Afaik, those are the things that JPG really doesn't handle well...
Thx, Menion, for tring that out. I think I wrote in my original post that it was more a gut feeling - perhaps I'm wrong. I was comparing downloads of Google Classic with SwissTopo. The thing is, SwissTopo has much less texts than Google, and those are most likely to show problems with JPG artefacts.
I also made a screenshot of a Google Maps screen. Zooming into it, it's all much clearer than a similar screenshot of Locus. Then again, it's hard to compare, as Google Maps obviously allows zooming in deeper than the download API. And as it's vector in the app, it's naturally much nicer looking.
Now, did I understand that correctly: Locus doesn't convert/compress the tiles received from the server, or does it? Because if Google delivers the maps like this, well - out of luck.
One more thing: No matter who compresses it, I see one potential problem with Google Maps and JPG: All texts have a very thin white border. Afaik, those are the things that JPG really doesn't handle well...
Hmm, compare Google Maps (raster) in Locus Map with vector data directly in Google Maps ... mainly with the latest Locus Map version where app rescale raster tiles based on device DPI is really not ideal.
It should be as you wrote: app download a certain array of bytes from the map server. From these bytes create an image you see on the screen, any exactly this original array of bytes save into SQLite database (map file). How good is compression or if source array is in JPEG or PNG format, Locus Map does not care.
The result of this topic if I remember correctly was caching of vector tiles for MapsForge V3 maps (LoMaps eg.). This is currently not (yet?) enabled in latest Beta version.
Hmm, compare Google Maps (raster) in Locus Map with vector data directly in Google Maps ... mainly with the latest Locus Map version where app rescale raster tiles based on device DPI is really not ideal.
It should be as you wrote: app download a certain array of bytes from the map server. From these bytes create an image you see on the screen, any exactly this original array of bytes save into SQLite database (map file). How good is compression or if source array is in JPEG or PNG format, Locus Map does not care.
The result of this topic if I remember correctly was caching of vector tiles for MapsForge V3 maps (LoMaps eg.). This is currently not (yet?) enabled in latest Beta version.
Well, looks like there's nothing that can be done right now. You can't improve the quality of what you get from the server. Device DPI-dependency is still the way to go :)
Well, looks like there's nothing that can be done right now. You can't improve the quality of what you get from the server. Device DPI-dependency is still the way to go :)
Replies have been locked on this page!