be37f3a565
This PR adds the ability to create SQL tests that ensure that OSM data is properly imported and updated in the OpenMapTiles data schema. The tests work by injecting test OSM data and updates into the database and checking to ensure that the data is properly loaded into the database using standard SQL statements. With this framework in place, developers can now write small tests to inject known data into the database and ensure that imports and updates are working correctly. In addition to the framework, basic tests are provided for four layers. These initial tests are in no way comprehensive, but they provide a structure and framework for developers to add their own tests or expand the existing ones to cover more cases. Usage: `make clean && make sql-test` ## How it works The SQL tests consist of the following parts: 1. **Test import data**, located in `tests/import`. This test data is in the [OSM XML](https://wiki.openstreetmap.org/wiki/OSM_XML) format and contains the data that should be initially injected into the database. The files are numbered in order to ensure that each test data file contains OSM id numbers that are distinct from the other files. For example, the file starting with `100` will use node ids from 100000-199999, way ids from 1000-1999, and relation ids from 100-199. 1. **Test update data**, located in `tests/update`. This test data is in the [osmChange XML](https://wiki.openstreetmap.org/wiki/OsmChange) format, and contains the data that will be used to update the test import data (in order to verify that the update process is working correctly. These files are also numbered using the same scheme as the test import data. 1. **Import SQL test script**, located at `tests/test-post-import.sql`. This script is executed after the test import data has been injected, and runs SQL-based checks to ensure that the import data was properly imported. If there are failures in the tests, an entry will be added to the table `omt_test_failures`, with one record per error that occurs during the import process. A test failure will also fail the build. To inspect the test failure messages, run `make psql` and issue the comment `SELECT * FROM omt_test_failures`. 1. **Update SQL test script**, located at `tests/test-post-update.sql`. This script performs the same function as the import test script, except that it occurs after the test update data has been applied to the database. Note that script will only run if the import script passes all tests.
OpenMapTiles 
OpenMapTiles is an extensible and open tile schema based on the OpenStreetMap. This project is used to generate vector tiles for online zoomable maps. OpenMapTiles is about creating a beautiful basemaps with general layers containing topographic information. More information openmaptiles.org and maptiler.com/data/.
We encourage you to collaborate, reuse and adapt existing layers, or add your own layers. You may use our approach for your own vector tile project. Feel free to fork the repo and experiment. The repository is built on top of the openmaptiles/openmaptiles-tools to simplify vector tile creation.
Please keep in mind that OpenMapTiles schema should display general topographic content. If creating a new layer or expanding an existing layer with a specific theme, please create a fork and invite other community members to cooperate on your topic. OpenMapTiles schema is used in many projects all over the world and the size of the final vector tiles needs to be considered in any update.
- 🔗 Schema https://openmaptiles.org/schema
- 🔗 Docs https://openmaptiles.org/docs
- 🔗 Data for download: https://www.maptiler.com/data/
- 🔗 Hosting https://www.maptiler.com/cloud/
- 🔗 Create own layer https://github.com/openmaptiles/openmaptiles-skiing
- 🔗 Discuss at the #openmaptiles channel at OSM Slack
Styles
You can start from several GL styles supporting the OpenMapTiles vector schema.
🔗 Learn how to create Mapbox GL styles with Maputnik and OpenMapTiles.
We also ported over our favorite old raster styles (TM2).
🔗 Learn how to create TM2 styles with Mapbox Studio Classic and OpenMapTiles.
Schema
OpenMapTiles consists out of a collection of documented and self contained layers you can modify and adapt. Together the layers make up the OpenMapTiles tileset.
🔗 Study the vector tile schema
- aeroway
- boundary
- building
- housenumber
- landcover
- landuse
- mountain_peak
- park
- place
- poi
- transportation
- transportation_name
- water
- water_name
- waterway
Develop
To work on OpenMapTiles you need Docker.
- Install Docker. Minimum version is 1.12.3+.
- Install Docker Compose. Minimum version is 1.7.1+.
Microsoft Windows Subsystem for Linux (WSL)
Please use Linux /home/user/ directory, not Windows e.g. /mnt/c directory.
Build
Build the tileset.
git clone https://github.com/openmaptiles/openmaptiles.git
cd openmaptiles
# Build the imposm mapping, the tm2source project and collect all SQL scripts
make
You can execute the following manual steps (for better understanding)
or use the provided quickstart.sh script to automatically download and import given area. If area is not given, albania will be imported.
./quickstart.sh <area>
Prepare the Database
Now start up the database container.
make start-db
Import external data from OpenStreetMapData, Natural Earth and OpenStreetMap Lake Labels. Natural Earth country boundaries are used in the few lowest zoom levels.
make import-data
Download OpenStreetMap data extracts from any source like Geofabrik, and store the PBF file in the ./data directory. To use a specific download source, use download-geofabrik, download-bbbike, or download-osmfr, or use download to make it auto-pick the area. You can use area=planet for the entire OSM dataset (very large). Note that if you have more than one data/*.osm.pbf file, every make command will always require area=... parameter (or you can just export area=... first).
make download area=albania
Import OpenStreetMap data with the mapping rules from
build/mapping.yaml (which has been created by make). Run after any change in layers definition.
make import-osm
Import labels from Wikidata. If an OSM feature has Key:wikidata, OpenMapTiles check corresponding item in Wikidata and use its labels for languages listed in openmaptiles.yaml. So the generated vector tiles includes multi-languages in name field.
This step uses Wikidata Query Service to download just the Wikidata IDs that already exist in the database.
make import-wikidata
Work on Layers
Each time you modify a layer's mapping.yaml file or add new OSM tags, run make and make import-osm to recreate tables (potentially with additional data) in PostgreSQL. With the new data, there can be new Wikidata records also.
make clean
make
make import-osm
make import-wikidata
Each time you modify layer SQL code run make and make import-sql.
make clean
make
make import-sql
Now you are ready to generate the vector tiles. By default, ./.env specifies the entire planet BBOX for zooms 0-7, but running generate-bbox-file will analyze the data file and set the BBOX param to limit tile generation.
make generate-bbox-file # compute data bbox -- not needed for the whole planet
make generate-tiles-pg # generate tiles
Workflow to generate tiles
If you go from top to bottom you can be sure that it will generate a .mbtiles file out of a .osm.pbf file
make clean # clean / remove existing build files
make # generate build files
make start-db # start up the database container.
make import-data # Import external data from OpenStreetMapData, Natural Earth and OpenStreetMap Lake Labels.
make download area=albania # download albania .osm.pbf file -- can be skipped if a .osm.pbf file already existing
make import-osm # import data into postgres
make import-wikidata # import Wikidata
make import-sql # create / import sql funtions
make generate-bbox-file # compute data bbox -- not needed for the whole planet
make generate-tiles-pg # generate tiles
Instead of calling make download area=albania you can add a .osm.pbf file in the data folder openmaptiles/data/your_area_file.osm.pbf
License
All code in this repository is under the BSD license and the cartography decisions encoded in the schema and SQL are licensed under CC-BY.
Products or services using maps derived from OpenMapTiles schema need to visibly credit "OpenMapTiles.org" or reference "OpenMapTiles" with a link to https://openmaptiles.org/. Exceptions to attribution requirement can be granted on request.
For a browsable electronic map based on OpenMapTiles and OpenStreetMap data, the credit should appear in the corner of the map. For example:
© OpenMapTiles © OpenStreetMap contributors
For printed and static maps a similar attribution should be made in a textual description near the image, in the same fashion as if you cite a photograph.