Page 1 of 3

UserMap: How to convert MBtile into RMapSQLite for FIF

PostPosted: 20 May 2018, 12:03
by farfly
Ahoj,
("Fly is fun" 21.20)

I understand that FIF uses moving maps in RMapSQLite format (zoom level 6 à 14).
Could you provide the specifications for this format (or a link to it) ?

I am looking specifically methods to convert :

- TMS Tile maps, (or Slippymap in XYZ) stored in ZXY directory
- MBTiles (which is also a tile map organised by sqlite, designed for mobil device)

to RMapSQLite.

Regards / děkuji

EDIT: 1 Solution usable proposed by RogerF (See Post 11 June 2018 with SQL script)

1 - download "DB Browser for SQLite" version 3.10.1 from https://sqlitebrowser.org/ and launch it
2 – open “Cartabossy2018.sqlitedb” (copy of Cartabossy2018.mbtiles file renamed)
3 – cut and past your script into the “SQL 1” windows
4 – run the script using the “play” arrow

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 20 May 2018, 23:58
by marpdiaz
Some 3-4 years ago I looked at a free linux software to do that. After a while I bought

MAPC2MAPC (for windows) to write sqlite database maps from geotif and geopdf (accepts several input formats).

It works fine, but there are possibly other options around today.

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 21 May 2018, 07:06
by Tonio
Hi

Most of the maps of FIF library have been created using MOBAC http://mobac.sourceforge.net/ or SAS planet http://www.sasgis.org/
Map sources being online maps as OpenStreetMap or OpenTopoMap (mainly for Europe)

In the past, I used Global Mapper http://www.bluemarblegeo.com/products/global-mapper.php once or twice a year to convert ICAO and other aviation maps to RMapSQLite format
Once, I crashed Global Mapper and since, I stopped

Bons vols

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 21 May 2018, 07:16
by Tonio

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 07 Jun 2018, 16:10
by RogerF
I am a new FIF user and all my other applications support mbtiles.
To feed FIF, I initially used MOBAC, fed by a set of tiles derived from mbtiles using a Python utility mb-util.py found on the internet.

However, this is a bit cumbersome and there is a possible risk of losing details/definition.
I found information about the conversion on the github forum, with an sqlite script + some patches provided by another contributor.
Unfortunately, the result did not display in FIF.
A bit frustrated, I did a contents analysis to compare the result of MOBAC with the result of the script... and some trial/error.
Now "my" sqlite query works fine and I used it to convert my favourite maps.
On my PC the conversion lasts 1-5 minutes, depending on the map size.

I just wonder whether the author would consider implementing mbtiles in FIF as an alternate to sqlitedb:
- Detection is straightforward based on the file type .mbtiles or .sqlitedb
- Mbtiles is also an sqlite database
- There is an easy mapping of the data:
- mbtiles fields are (zoom_level, tile_column, tile_row, tile_data)
- sqlite fields are (z, x, y, s, image):
- conversion is simple:
z = 17-zoom_level
x = tile_column
y = 2^zoom_level - 1 - tile_row
s = 0
image = tile_data

Otherwise, I can send the script to those who are interested.

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 10 Jun 2018, 14:55
by Tonio
Hi Roger
If you have a tool / application allowing automatic conversion of MBTiles to RMapSQLite, I'm sure some of us would appreciate it a lot :D

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 11 Jun 2018, 07:29
by RogerF
[updated => v2 of the script]
OK Petr; here is the tutorial:

All you need is to make a copy of the .mbtiles file and rename the copy as .sqlitedb
Then you open it with an sqlite application, for example the free application "DB browser for sqlite".
Then you run my sql script here below (thanks to the github forum for providing ideas and info; I also needed to analyse map files in both formats to get to this sql script).

Uploading attachments here does not seem to work: I always get a message "The extension xxx is not allowed", whatever extension I use, so I have simply pasted the contents of my
my "Mbtiles_to_sqlitedb_RogerF.sql" file here. Feel free to use it.

-- Conversion of "mbtiles" map into "sqlitedb" map
-- Roger Fraikin - V2 - June 2018

--- Use is simple:
--- 1. Copy the xxxx.mbtiles file to a new xxxx.sqlitedb file
--- 2. Open this file with an sqlite application (e.g. "DB Browser for sqlite")
--- 3. Load this "Mbtiles_to_sqlitedb_RogerF.sql" script as sql query and let it run
--- 4. Confirm the final cleanup (the VACUUM command triggers a warning)
--- 5. Close the sqlite application, confirming the "save the modified database" message.
---
--- Development notes
---
--- N1. The sql command "Rename columns" is not supported by sqlite (despite many requests on the dedicated forum, but probably for good reasons).
--- The workaround is to copy all elements onto a new table with the adequate structure for sqlitedb.
---
--- N2. The formula to translate the mbtiles "tile_row" data into sqlitedb "y" is: y = 2^z - & - tile_row
--- Unfortunately, sqlite only supports basic mathematics like +,-,*,/%, not the "^" exponential operator.
--- The workaround is to perform separate queries for each zoom levels
---
--- N3. Especially with low zoom levels, there is a chance the "y" conversion leads to a "duplicate indexes" error
--- That is why I only create the index after the conversion has been made
---
--- N4. The query can take some time for big maps; that is why I added a series of SELECT messages to monitor the progress
--- Be patient for the final cleanup (VACUUM) as the DB is completely copied/rebuilt.
---
--- Feel free to use this script to convert your maps ;-)

CREATE TABLE sqlitedb_tiles
(
x int,
y int,
z int,
s int,
image blob,
PRIMARY KEY (x,y,z,s)
);
SELECT "New table created";

INSERT INTO sqlitedb_tiles (x, y, z, s, image) SELECT tile_column as x, tile_row as y, zoom_level as z, '0' as s, tile_data as image FROM tiles;
SELECT "Data copied";

--- suppress old table "tiles"
DROP TABLE tiles;
SELECT "Old table dropped";

--- suppress old table "tiles" and rename the new one to "tiles"
ALTER TABLE sqlitedb_tiles RENAME TO tiles;
SELECT "New Table renamed";

--- no need to keep the old "metadata" table
DROP TABLE metadata;
SELECT "Metadata dropped";

--- convert mbtiles data into sqlitedb data assuming that minimum zoom level is 3 and maximum is 16
UPDATE tiles SET y = 7 - y where z = 3;
UPDATE tiles SET y = 15 - y where z = 4;
UPDATE tiles SET y = 31 - y where z = 5;
UPDATE tiles SET y = 63 - y where z = 6;
UPDATE tiles SET y = 127 - y where z = 7;
UPDATE tiles SET y = 255 - y where z = 8;
UPDATE tiles SET y = 511 - y where z = 9;
UPDATE tiles SET y = 1023 - y where z = 10;
UPDATE tiles SET y = 2047 - y where z = 11;
UPDATE tiles SET y = 4095 - y where z = 12;
UPDATE tiles SET y = 8191 - y where z = 13;
UPDATE tiles SET y = 16383 - y where z = 14;
UPDATE tiles SET y = 32767 - y where z = 15;
UPDATE tiles SET y = 65535 - y where z = 16;
UPDATE tiles SET y = 131071 - y where z = 17;
SELECT "y converted";

UPDATE tiles SET z = 17 - z;
SELECT "z converted";

--- create the INFO table (required by sqlitedb) with the min and max zoom level
CREATE TABLE info (minzoom, maxzoom, url);
INSERT INTO info (minzoom, maxzoom) SELECT min(z), max(z) FROM tiles;
SELECT "INFO table created";

--- craete the required index "IND"
CREATE INDEX IND on tiles(x,y,z,s);
SELECT "IND Index created";

--- cleanup possible garbage and reorganise the indexes - this can take a while and you need to confirm a warning message
SELECT "Cleanup running";
VACUUM;
SELECT "Conversion completed";

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 11 Jun 2018, 20:27
by farfly
Many thank Roger for this script.

I am almost there: I get the following error at the beginning of the cleaning:
Code: Select all
cannot VACUUM from within a transaction:
VACUUM;


Can you help ?
The new sqlitedb has been created but is almost twice the size of the original mbtiles file.

In addition, how can you check the resulting file outside FIF ?
I was hoping to visualize it with MOBAC or QGIS, but I haven't found the procedure so far.
(I have concerns with file stored with TMS or slippymap convention, and want to cross check it before loading it in FIF)

Regards

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 11 Jun 2018, 20:58
by RogerF
"Almost there" indeed. ;)
You probably use another SQlite brower.
I have developed and tested with "DB Browser for SQLite" version 3.10.1 from http://sqliterbrowser.org

Either you download the same version as I am using, or you try to reopen the database and execute a simple query with "VACUUM".
Your call !

The purpose of the VACUUM command is to do a cleanup of the former table in the database.
If you cannot execute it, the database does not shrink after its deletion, so the size is indeed double.

To test, I simply used FIF as this is the only app that I have that requires this format; it worked for the 10 personal maps that I tried so far.
Why would you test elsewhere?

Re: Format for UserMap ? (RMapSQLite) and conversion

PostPosted: 12 Jun 2018, 19:10
by Tonio
Hi Roger

Just Great

Despite my ZERO level in this area I was able doing it

Having
1 - downloaded "DB Browser for SQLite" version 3.10.1 from https://sqlitebrowser.org/ and launch it
2 – open “Cartabossy2018.sqlitedb” (Cartabossy2018.mbtiles file renamed following your advice)
3 – cut and past your script into the “SQL 1” windows
and
4 – run the script using the “play” arrow

I did get the converted file YES !!!

File size is almost the same
Original Cartabossy2018.mbtiles file I used is 161,2 Mo
After conversion Cartabossy2018.sqlitedb become 161,4 Mo

Only problem appears transferring the file from my Mac to the Android device. The “mbtiles” extension was automatically added, file name becoming “Cartabossy2018.sqlitedb.mbtile”

To be able using the Cartabossy map with FIF, on the android device, I had to edit “Cartabossy2018.sqlitedb.mbtile” file name to suppress “.mbtile” in order to get only "Cartabossy2018.sqlitedb"

And it work well and full process is less than 5 minutes

You should may be rename or republish your post
Name could something as "How to convert “MBtiles” mapfile in to “RmapSQlite” mapfile to use it with FIF"

Bravo and have safe flights

Tonio