This PR addresses a pg_deadlock error that would cause an update to fail if a `select getmvt` was queried during update. The deadlock occurred due to an ALTER TABLE DROP COLUMN operation on temporary columns ( `new_source_ids` and `old_source_ids`) in some linestring tables.
**Changes:**
* instead of adding/dropping the columns during update, they are now updated and set to NULL.
* Added new indexes on the source_ids columns to speed up the UPDATE.
* Will refresh `osm_park_polygon_dissolve_z4` concurrently in `park_polygon.refresh()` to avoid blocking.
Co-authored-by: Patrik Sylve <patrik.sylve@t-kartor.com>
Fixes#1640
This ensures that place and poi points computed from areas result in a point node that is actually inside the area that it's derived from. Unusual shaped areas (like a banana shape, for example) may produce points that aren't actually contained.
This PR adds indigenous lands to the `boundary` layer by adding new area features for these objects, along with generalization up to z4. It borrows heavily from the technical processing chain of the `park` layer.
I added new `class`, `name`, and `name:xx` attributes to the boundary layer, in order to support the new area features. Should we (or someone extending this schema) come up with new objects to place in the boundary layer in the future, `class` would support additional categories.
This PR is a **documentation change only**. It documents the fact that `name:xx` values are present in the tiles and documents that `name_de` and `name_en` will be removed in a future release in favor of `name:de` and `name:en` respectively.
1) The trigger name has a typo of `trigger`, which is incorrectly spelled as `tigger` and doesn't match the following create command: `CREATE TRIGGER trigger_flag`. This may result in an error when invoking `make import-sql`.
```
ERROR: trigger "trigger_flag" for relation "osm_park_polygon_gen_z4" already exists
xargs: sh: exited with status 255; aborting
```
2) The `trigger_flag` may be created on the wrong target in line [#251](https://github.com/plainheart/openmaptiles/blob/master/layers/park/update_park_polygon.sql#L251).
This PR adds support for `boundary=aboriginal_lands` by adding it to the `park` layer. While these lands are certainly not "parks", they have similar treatment from a technology perspective and can benefit from the existing processing chain established in that layer. I set all of these objects to `class=aboriginal_lands` in the tiles, including at the lowest zoom in order to separate it from the protected area merging implemented in #1160. In order to distinguish these from general parks, I expose the `class` attribute for these objects at z4 and also ensure that the z4 generalization is performed separately for protected areas versus aboriginal lands.
This unblocks #ZeLonewolf/openstreetmap-americana#105, which describes why having indigenous land boundaries is an important general feature on the map. This is also consistent with my suggested implementation in https://github.com/openmaptiles/openmaptiles/issues/1296#issuecomment-967749403.
This PR addresses that by:
* rendering oneway arrows on more than `subclass=cycleway`
* less opacity for path/footway casing
* adding the thin black casing to bridges
* adding `subclass=steps` & `subclass=bridleway` rendering
* render `bicycle=designated` as `subclass=cycleway` and `horse=designated` as `subclass=bridleway` (bike taking priority)
* improved tunnel rendering with double casing & adding main line
* made casing min zoom consistent across subclasses
All of these changes were made to mimic OSM Carto.
* OpenMapTiles style
* Rename style snippets from layer_name.json to defaultstyle.json
* Add README to icons dir.
* Add README to icons dir.
* Rename snippets to style.json
* Fix spritezero - move icons to svg subdir
* Move icons from style/icons/svg directly to style/icons
Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
This PR implements non-blocking updates in the park layer.
The approach was to use the `REFRESH MATERIALIZED VIEW CONCURRENTLY` feature in postgres. In order to achieve this, a unique index was added on the z4 dissolved park area table. The `ST_Union` / `ST_Dump` sequence was changed to an explicit cluster DB scan (an equivalent operation) so that a unique osm ID could be generated from each dissolved polygon.
Below is a screen shot from Idaho, USA showing that the dissolved z4 still works as expected.
This PR fixes a bug in the park polygon update code introduced in #1160. Because park polygons at zoom 4 are dissolved, the centroid is not present in the table at that layer. Thus, a separate update trigger is needed to avoid errors. This PR adds the separate trigger for the zoom 4 park layer.
This PR likely resolves an unidentified blocker for #1245
* All functions that access database must be declared as `STABLE`, not `IMMUTABLE` -- because database can change at any moment, e.g. during an update
* there are a few functions that could be made `STRICT` -- passing `NULL` as a parameter will always result in a `NULL`, but for some reason that causes a significant decrease in perf.
* tagged one function as parallel safe
NOTE: somehow `ST_AsMVT()` method of tile generation is showing 70-90% slowdown with this patch. I am not sure of why this is happening. If the reason is the `IMMUTABLE` -> `STABLE` change, we may have to dig deeper into PG optimization
I would like to reformat all of our SQL to have a concise coding style.
This makes it far easier to understand the code for a casual contributor,
and lets us spot errors more easily.
Most importantly, it makes it much easier to grep (search) the code because it is more likely to be in the same syntax
Some key changes:
* SQL keywords are always UPPERCASE, e.g. `SELECT WHEN AS END ...`
* types, variables, aliases, and field names (identifiers) are always lower case
* `LANGUAGE 'plpgsql'` is now `LANGUAGE plpgsql` (no quotes)
* a few minor spacing/semicolon cleanups
P.S. Per @TomPohys request, `TABLE` is spelled using upper case despite being a type for consistency with PG Docs. Same for `LANGUAGE SQL` vs `LANGUAGE plpgsql`.
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.
`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).
### Testing procedure
The procedure to test this was:
* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.
<details><summary>Testing query</summary>
```sql
-- Using the function layer_landuse
explain analyze
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>
You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities
### Next steps
Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.
I will post my finding here, even no change in the code should happen.
cc. @nyurik
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
quicker and cleaner diagram image generation.
Remove etl-graph and mapping-graph targets - redundant
Also, the obsolete "fields" is still in Imposm's code and both names are accepted,
but "fields" is not documented anywhere (PR submitted), and could be removed at any moment.
Our docs were not supporting it until this PR, so renaming it at the same time.
Several images have been updated due to a more inclusive mapping scan
Requires https://github.com/openmaptiles/openmaptiles-tools/pull/147 (merged)
Mark all tables that should not be populated with the Wikidata
international labels with a special OMT-specific flag.
This should be ok to merge even before the new tools version
is used because imposm seems to ignore anything it doesn't understand.
The next tools version will remove it when generating imposm mapping file.
Seems like etl and mapping diagrams have been neglected
for a long time. Now it regenerates the files and places
them in the source dir.
This PR also fixes two broken files:
* layers/aerodrome_label/mapping_diagram.png
* layers/housenumber/mapping_diagram.png
They were generated using the newest tools version with the fix
https://github.com/openmaptiles/openmaptiles-tools/pull/65
