142 Commits

Author SHA1 Message Date
Adam Laža
f21b677083
OSM OpenMapTiles style (#1420)
* OpenMapTiles style

* Rename style snippets from layer_name.json to defaultstyle.json

* Add README to icons dir.

* Add README to icons dir.

* Rename snippets to style.json

* Fix spritezero - move icons to svg subdir

* Move icons from style/icons/svg directly to style/icons

Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
2022-10-06 16:09:21 +02:00
Tomas Pohanka
6e73058a23
Class shipway (#1415)
In `transportation_name` layer for `shipway` used Instead of `subclass` use `class` attribute for `ferry`. But `subclass` remains.
2022-09-08 09:10:42 +02:00
Clay Smalley
337f81284d
Add highway=bus_guideway (#1407)
Guided busways (or bus guideways in OSM terminology) are special roadways along which bus operators can travel smoothly at high speeds without steering. These are tagged highway=bus_guideway on OpenStreetMap.

Currently, highway=busway is surfaced in OpenMapTiles as the busway class in the transportation layer, but highway=bus_guideway is not present. These two types of roadways serve generally similar purposes, so it would make sense to have guided busways at the same zoom levels.
2022-07-26 15:26:57 +02:00
SahAssar
dc1e21fccc
Change typecheck to perform within the current namespace/schema (#1389)
This PR fixes #1388 by doing a test cast for the type and if it fails adding the type.
2022-07-26 12:26:46 +02:00
Adam Laža
ef9a68b19a
GE-871 upsert network_type column in update_osm_route_member (#1384)
Fixes #1374 

This PR fixes `update_osm_route_member()` function. Before this the column `network_type` became empty after daily-update.
This PR adds `network_type` into upsert together with `rank` and `concurrency_index` columns.
This bug had occurred once already and had been fixed in https://github.com/openmaptiles/openmaptiles/pull/1239 but then some columns were added and these new columns were not added into the upsert.
2022-05-05 13:14:35 +02:00
Frédéric Rodrigo
bff74511b6
Use diff update on osm_transportation_merge_linestring (#1357)
Replace the `REFRESH MATERIALIZED` on `osm_transportation_merge_linestring_gen_z*` by differential update.

The way of doing this is the same as other differential updates. But in this case there two `GROUP BY` level. So all the the process have to be done twice.
2022-05-04 14:22:15 +02:00
Matt Riggott
b13ab9491f
Add more aerialway tags to transportation layer (#1343)
Before this, only `aerialway=cable_car` and `aerialway=gondola` were included in the tiles. That misses some widely-used tags, notably `aerialway=chair_lift`, `aerialway=drag_lift`, and `aerialway=platter`. The usage of all the tags added in this commit, by fraction of all aerialway way keys (according to TagInfo), is:

| Tag                    | Usage  |
|------------------------|-------:|
| `aerialway=chair_lift` | 25.51% |
| `aerialway=drag_lift`  | 12.93% |
| `aerialway=platter`    | 11.35% |
| `aerialway=t-bar`      |  7.93% |
| `aerialway=j-bar`      |  0.63% |
| `aerialway=mixed_lift` |  0.24% |

Some of these tags were also added in PR #620 back in May 2019, but were removed again before it was merged.

Fixes #1342.
2022-02-17 13:08:39 +01:00
Tomas Pohanka
81ddab9f9a
Updates documentation for 3.13 (#1350)
Adding missing information in `layers.yaml` files for 3.13
2022-02-09 12:38:47 +01:00
Adam Laža
2e04f83166
Fix paths and tracks in transportation layer z12 and z13 (#1334)
This PR fixes a bug that causes that `track` lines disappear at z13. This bug was introduced in https://github.com/openmaptiles/openmaptiles/pull/1190, which adds rendering of paths and tracks at z12 and z13.

Before this PR:
z12: lines with `route_rank = 1` are added (no matter what `highway`).
z13: lines with `route_rank BETWEEN 1 AND 2` and `highway = 'path'` are added.
-> tracks with `route_rank=1` are added at z12 but not at z13.

After this PR
z12: lines with `route_rank = 1` and `highway IN ('path', 'track')` are added.
z13: lines with `route_rank BETWEEN 1 AND 2` and `highway IN ('path', 'track')` are added .
-> only tracks and paths are added at z12 and z13 (which was IMHO the goal of https://github.com/openmaptiles/openmaptiles/pull/1190)

* Add only the most important paths and tracks (route_rank=1) to z12 and more important ones (route_rank between 1-2 or sac_scale or has name) to z13.
2022-01-05 07:28:34 +01:00
Tomas Pohanka
73dcd23497
Run get_basic_name after osml10n_street_abbrev_* (#1326)
PR move `get_basic_names(tags, geometry)` function from `transportation_name_tags` function and adding it after `transportation_name_tags` resolve street abbrevation.

PR runs through `test-sql` test without error. Adding to `osm_transportation_name_network` (table and for update) was done in consequence of `test-sql` errors ([update test 500](a747d98550/tests/test-post-update.sql (L53))). Originally `get_basic_names` was just in `osm_transportation_name_linestring` table.

FIX #1324

![seattle_fix](https://user-images.githubusercontent.com/5182210/145085141-7857e97e-9b83-4467-bb98-84cc4a2e0623.png)
2021-12-08 08:20:15 +01:00
Tomas Pohanka
44ae83718c
Remove STRICT statement (#1320)
Remove `STRICT` statement from `transportation_name_tags` function.

Concurrently with remove `STRICT` from `get_basic_names` and `get_latin_name` in OMT-T

FIX #1319
2021-12-07 17:59:04 +01:00
Brian Sperlongano
7f23feab88
Add expressway tagging to the transportation layer (#1313)
Fixes #1148 

This PR adds expressway tagging to the `transportation` layer, by setting `expressway=1` for non-motorway roads tagged `expressway=yes`, and omitting the tag otherwise.  Additionally, I've added a few unit tests to verify that the expressway tagging is being imported and updated into the intermediate tables.

Here is an example of expressway tagging on US-1 in Rhode Island, USA:

![image](https://user-images.githubusercontent.com/3254090/143501278-db3671b2-2efa-4998-bffd-9ddfec63579b.png)
2021-11-26 07:55:20 +01:00
Brian Sperlongano
ec74480414
BUGFIX: Fix name-based way fragmentation in transportation_name (#1295)
I discovered this bug while investigating issues with the updates process related to #1190 #1292, and #814.

The `transportation_name` layer produces slightly different `tags` hstore values in the `osm_transportation_name_linestring` table during the initial import versus when running an update.  As currently written, the import code produces null-value keys in the `tags` column, while the update code suppresses them.  This PR removes that difference and makes the import code use same method that is currently used in the update code.

With a test case I've written, the import code produces a tags hstore that looks like this:
`"name"=>"OpenMapTiles Secondary 2", "name:de"=>NULL, "name:en"=>NULL, "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`

...while the update code produces a tags hstore that looks like this:

`"name"=>"OpenMapTiles Secondary 2", "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`

Note the missing NULL values.

This bug causes a small amount of space wastage after an update is run, because the update matching code detects the `tags` value as different, resulting in a duplicate copy of the tags value if that row is updated.  This causes duplicate objects and breaks GROUP BY clauses that expect to group same-tagged features together.  I've tested this by inspection of a generated mbtiles, database spot checks, and the unit test code included in this PR.
2021-11-25 10:45:11 +01:00
Brian Sperlongano
75d8c80228
Selective rendering of tracks and paths at z12-13 (#1190)
Closes #271

This PR adds track and path rendering at lower zooms than currently provided, and also achieves near-parity with openstreetmap-carto on track and path rendering. A previously-abandoned attempt, with significant discussion, was #1169.
2021-11-18 17:35:34 +01:00
Falke Design
ebeafc65fa
Add requires.tables to <layer>.yaml (#1236)
**NOTE** this can only be merged after the next tools version is released.

Added required Postgres tables to the `<layer>.yaml` definition.
Close: #1220

PR of tools: https://github.com/openmaptiles/openmaptiles-tools/pull/370
2021-11-05 10:04:54 +01:00
Brian Sperlongano
c7e7fae9ee
Add support for Bus Rapid Transit (highway=busway) (#1271)
This PR adds support for bus rapid transit (BRT) systems, mapped with the [approved tag](https://wiki.openstreetmap.org/wiki/Proposed_features/Tag:highway%3Dbusway) `highway=busway`. BRT systems commonly appear in maps that also show light rail.  Below are two examples of BRT in such maps.

![image (1)](https://user-images.githubusercontent.com/3254090/138018375-ac1f60c3-8059-4f95-bf1d-a659ca0c0103.png)
![image](https://user-images.githubusercontent.com/3254090/138018376-f809aeb3-17ba-495e-b33c-9fe54a0fa840.png)

Since BRT is equivalent in nature to light rail systems, this PR matches the zoom level of light rail, which is rendered at zoom 11+.  BRT systems are relatively uncommon, so adding this support should have negligible impact on tile size.

Below is a data rendering of [a BRT system](https://www.openstreetmap.org/relation/4051383#map=12/40.4068/-79.8646) in Pittsburgh, Pennsylvania.

![image](https://user-images.githubusercontent.com/3254090/138021552-ce974f2f-cfbd-42a0-9848-42ed44da453e.png)
2021-11-01 08:40:07 +01:00
Brian Sperlongano
a0d915942a
Allow oneway=-1,1 only (#1267)
This PR removes cases of `oneway=0` as this is an assumed default.  It is not necessary to have this tag on every road or path and by removing it we can save space in the tiles by only including the meaningful values.  It leaves intact `oneway=1` and `oneway=-1`, the latter of which indicates a oneway in the opposite direction of the way.

Unblocks #1265
2021-10-28 15:39:34 +02:00
Brian Sperlongano
cded42349f
Remove unnecessary ramp tagging (#1266)
This PR removes unnecessary `ramp` tagging from the `transportation` layer.  The `ramp` tag is used for two situations.  First, it is set for all `highway=***_link` tags to indicate a highway ramp.  The second situation is when the `ramp` tag is set on a highway object, which indicates the presence of a ramp, such as one that would be used for a wheelchair.  `ramp=0` is a reasonable default in these situations; by removing cases of `ramp=0` and presenting only `ramp=1` to the tiles, we can save considerable size in the tiles.  Additionally, there appears to be a bug in which all objects tagged `highway=steps` are inexplicably tagged `ramp=1`.

The changes as follows:

1. Remove `ramp=0` where it appears in the tile.
2. `highway=steps` no longer assumes `ramp=1` in cases where `ramp=*` is not explicitly set.

In the current behavior, `ramp=0` is tagged only in at z13+ but suppressed in lower zooms:
![image](https://user-images.githubusercontent.com/3254090/137589213-5a3432df-8c19-47bf-b556-7be15479790c.png)
![image](https://user-images.githubusercontent.com/3254090/137589296-f3126264-fb07-4346-ba70-c1f1b007a709.png)
![image](https://user-images.githubusercontent.com/3254090/137589306-30f9821b-3160-4b45-98b3-f2b8a0ae3a13.png)

This PR unifies `ramp=0` suppression between z13+ and z12-.
2021-10-28 15:08:56 +02:00
Brian Sperlongano
829f28d27b
Fix logic error in transportation layer filter (#1270)
This PR fixes a bug in the `WHERE` clauses in the transportation layer generalization tables for zoom 10.    The intended behavior of the layer is to suppress `highway=tertiary` and `highway=tertiary_link` at zoom 10 and lower.  However, due to this bug, these objects were not suppressed as intended, because an `OR` was used in the SQL where an `AND` was needed instead.

This bug was inadvertently introduced in #1172 😞
2021-10-19 10:31:51 -04:00
Brian Sperlongano
f744f9c009
Upgrade to omt-tools v6, Imposm 0.11.1 and migrate SQL (#1246)
This PR migrates the SQL in the transportation and transportation_name layer to use the new imposm3 mappings which now map a separate primary key ID.
2021-10-14 11:38:28 -04:00
Brian Sperlongano
aaa68048d9
Remove unneeded toll code (#1262)
This PR removes unneeded `CASE` clauses which converts `toll` tags to booleans.  However, the `toll` tag is already mapped as a boolean by imposm, therefore this processing can be simplified.
2021-10-13 20:43:14 +02:00
Brian Sperlongano
88389f2a2c
BUGFIX: Route concurrency data is wiped after route member updates (#1239)
This PR is a bugfix for the `transportation_name` layer.  Currently, the updates process does not work, as noted by @zstadler in #1230.  The issue is that the computed `concurrency_index` column in `osm_route_member` is never updated after the initial updates.  Therefore, whenever route member ways are updated, they will lose their route concurrency index, and hence route concurrencies will be lost in the `transportation_name` tiles.  This PR adds the missing concurrency index update code to the transportation/network update triggers.

This PR as written also incorporates the fixes in #1230 and #1233.

The following SQL is a unit test to demonstrate that these triggers work correctly:

```
select count(*) from osm_route_member where concurrency_index > 5;
select count(*) from osm_transportation_name_network where route_6 is not null;

update osm_route_member set concurrency_index=NULL where concurrency_index > 5;

select count(*) from osm_route_member where concurrency_index > 5;
select count(*) from osm_transportation_name_network where route_6 is not null;
```

If working correctly, both pairs of `count(*)` values should return the same pair of numbers.
2021-09-21 12:56:25 +02:00
Brian Sperlongano
8b6e69e440
BUGFIX: Ignore duplicate route concurrencies (#1233)
While troubleshooting #1230, I discovered that there were cases where a way was a member of the same route relation more than once, such as: https://www.openstreetmap.org/way/17439235

<img src="https://user-images.githubusercontent.com/3254090/133726616-acd01332-343e-4930-ad3d-1fb3c3190a4c.png" width=280/>

This segment would end up showing duplicate highway shields in a rendered style, which is not desirable.  While I personally think this style of tagging is wrong, it's a relatively easy fix to remove those duplicates.  This PR replaces `ROW_NUMBER()` with `DENSE_RANK()` in the concurrency index generation code and adds a `DISTINCT` constraint on the concurrency join to handle multiple member rows with the same concurrency index.

Since I was in this file, I also fixed the copy/paste error in generating the route names.

Screenshot from database showing correct mapping:
![image](https://user-images.githubusercontent.com/3254090/133725554-1aa5dbbc-59a8-4674-a446-b92adb96c2a1.png)
2021-09-20 18:40:36 +02:00
Jaroslav Hanslík
d49f524edd
Removed useless NULLIF calls (#1214)
`NULLIF(wikipedia, '') IS NOT NULL` replaced by `wikipedia <> ''`
https://stackoverflow.com/questions/23766084/best-way-to-check-for-empty-or-null-value
2021-09-20 11:19:24 +02:00
Brian Sperlongano
849aca8da7
Refactor z12-14 transportation filter logic (#1207)
This PR is a refactor.  The purpose of this PR is to refactor the logic in the z12-z14 transportation layer to increase readability and reduce code complexity.

In addition, this fixes bugs in the original filter code:
* Create filter parity between `service` and `service_construction` highway classes.
* Removal of unintended `class` values `minor_construction`, `path_construction`, `service_construction`, and `track_construction` from zoom 12 filter by explicitly listing the highway classes that are included.
2021-09-20 08:07:38 +02:00
Brian Sperlongano
3818979143
Suppress service roads at certain zooms (#1192)
Fixes #1191

This PR suppresses `highway=service` at zoom 12, where it is not a useful level of detail.  This makes OpenMapTiles consistent with openstreetmap-carto, which does not begin showing `highway=service` until raster zoom 14 / vector zoom 13.

Additionally, this PR suppresses `highway=service` + `service=parking_aisle` / `service=driveway` from zooms 12-13, as this detail is excessive below zoom 14.  As a point of comparison, openstreetmap-carto does not begin showing `service=parking_aisle` / `service=driveway` until raster zoom 16 (vector zoom 15).
2021-09-01 09:18:45 +02:00
Brian Sperlongano
7ca751ec7a
Add toll tagging (#1177)
Fixes #366

This PR sets `toll=1` in the `transportation` layer when a road is tagged as a toll road in OSM (with the tag `toll=yes`).  If a road is tagged with any other value of `tag=*`, the value is suppressed in the tile, since non-toll roads can be presumed as the default and therefore this PR should have only negligible impact in the tiles.

Support for toll road tagging is of interest in the American mapping community, because toll roads have historically been styled differently on American-style maps, for example:
![image](https://user-images.githubusercontent.com/3254090/129505967-5916eace-596a-4c89-ac5d-0aab3e641ed7.png)


Screen shot of a toll road being generated in the `transportation` layer tiles:
![image](https://user-images.githubusercontent.com/3254090/129505683-eb315643-95ff-455b-a606-f379f776f92d.png)
2021-08-19 07:12:10 +02:00
Brian Sperlongano
45d825e212
Implement private road tagging (#1174)
Fixes #1066 

This PR adds a new field `access` in the transportation layer, which will be set to `no` if the `access` tag is either `no` or `private`.  While `private` is the more popular value by a 17:1 ratio, I went with `no` because it's smaller in the tiles.  In addition, the text `no` opens up the future possibility of other text-based access values such as `destination`, `customers`, or `permit`.

The screenshot below shows an example of access tagging for a road on a military base:

![image](https://user-images.githubusercontent.com/3254090/129431491-9acbaeca-bf18-4384-8177-2c198834865c.png)
2021-08-14 07:04:17 +02:00
Brian Sperlongano
fdb9ae58cd
Transportation generalization table optimization and cleanup (#1172)
This PR updates the `transportation` layer creation scripts to simplify the SQL and remove unneeded sub-selects, checks and conditionals, fix indexes, and improve inline documentation. 

Currently, there are two sequences of materialized view creations.  There is one from zoom 11 through 9, and a second one for zoom 8 through 4.  This PR removes that break in the sequence of transportation table materialized view creations, in favor of one in which high-zoom views are created first, and then each lower zoom is created from the zoom above.

Instead, the current generalized zoom 8 transportation table is built directly from `osm_transportation_linestring` rather than being built from the zoom 9 transportation table.  This means that when building the zoom 8 table, it must scan the entire transportation network rather than just selecting from the pre-filtered zoom 9 table.

This PR removes an unneeded sub-select in the build of the zoom 8 table, which appears to be a leftover from an old version of the SQL that did some sort of merge.  Once this PR is implemented all zooms from 11 through 4 will be linked via a progressive series of materialized views.

Lastly, this adds in missing materialized view refreshes for zooms 9-11, which appear to have been entirely missing, and as far as I can tell aren't getting updated in the current version of this code.

In addition, the following optimizations were added as part of this commit:

1. Updates the `osm_highway_linestring_highway_partial_idx` partial index to match the `SELECT..WHERE` clause actually present in `transportation/update_route_member.sql`, which is where it appears to be actually used, and update inline documentation to reflect this.

2. Collapses unnecessary sub-select block in `osm_transportation_merge_linestring_gen_z11`, and removes unnecessary ST_IsValid() call, which already provided in `mapping.yaml`, and update inline documentation to reflect these assumptions.

3. Updates `WHERE` blocks to remove unnecesary checks and further document assumptions.  The `highway=construction` check is unnecessary in all cases, because it is sufficient to check the `construction` key alone.  If `construction=*` is set, then `highway=construction` is implied.

4. Two indexes were added to `layers/transportation/update_route_member.sql` to improve route population performance.

In testing locally, I'm seeing performance improvements around 10% in the generation of the `transportation` layer, based on modifying `openmaptiles.yaml` to generate only the transportation layer and then repeatedly running `time make import-sql`, however, this timing might be impacted by docker, so I would ask for confirmation of acceptable performance.

In addition, this PR shortens the length of the transportation update SQL file by 30 lines, which makes it easier for contributors to work with.
2021-08-13 18:00:33 +02:00
Brian Sperlongano
f78d42ec84
Remove unused osm_transportation_merge_linestring table (#1171)
PR #1168 removed several `WHERE` clauses in the `transportation_merge_*` table series.  With those removed, it appears that `osm_transportation_merge_linestring_gen_z8` and `osm_transportation_merge_linestring` are nearly identical, with the former simply adding an `ST_Simplify()` operation.

A grep of the codebase indicates that the _only_ use for `osm_transportation_merge_linestring` is to hold the intermediate result of an `ST_Dump(geometry))` before it is fed into `ST_Simplify()`.  Therefore, it appears that we're holding an entire zoom 8 copy of the transportation layer (all motorway/trunk/primary roads) in a materialized view for absolutely no reason at all.

This PR removes the `osm_transportation_merge_linestring` intermediate table and changes the definition of `osm_transportation_merge_linestring_gen_z8` to perform the `ST_Dump()` and `ST_Simplify()` transformations directly from the `osm_highway_linestring` table in a single operation.
2021-08-11 10:46:17 +02:00
Brian Sperlongano
09078f6d7d
Remove useless WHERE clauses from the transportation layer (#1168)
This PR removes several redundant/unnecessary WHERE clauses from the transportation layer.  Specifically:

The table `osm_transportation_merge_linestring` is a view of `osm_highway_linestring` which exposes only motorway/trunk/primary roads:
9e4be3e3b0/layers/transportation/update_transportation_merge.sql (L122-L123)

However, the create statement for the table `osm_transportation_merge_linestring_gen_z8`, which is a view of `osm_transportation_merge_linestring`, also contains  a `WHERE` clause which selects down to motorway/trunk/primary roads.  This `WHERE` is unnecessary:
9e4be3e3b0/layers/transportation/update_transportation_merge.sql (L144-L145)

This unneeded `WHERE` clause is similarly present in the create statement for `osm_transportation_merge_linestring_gen_z7`, which is a view of `osm_transportation_merge_linestring_gen_z8`:
9e4be3e3b0/layers/transportation/update_transportation_merge.sql (L163-L164)

Likewise, there is a similar redundant `WHERE` clause in the `osm_transportation_merge_linestring_gen_z5` and `osm_transportation_merge_linestring_gen_z6` tables, both of which select down to `motorway` and `trunk`.  This `WHERE` clause is only needed on the z6 table, and is redundant on the z5 table.

I am not sure what the performance penalty is for these redundant `WHERE` clauses, but there does not appear to be any reason to keep them, and they may incur a performance cost.
2021-08-10 17:55:16 +02:00
Brian Sperlongano
08bb2a06c0
Add network to transport layer (#1158)
Closes #1153 

This PR populates the existing `network_type` field in the route table with highway route network information, using the most important value in cases of concurrencies.  In order to expose this value, two files `network_type.sql` and `update_route_member.sql` were moved from the `transportation_name` layer to the `transportation` layer.

Below is a representative zoom level 6 with motorways only shown for `us-interstate` network types.  This sample was generated from this PR using a [custom branch](https://github.com/ZeLonewolf/openstreetmap-americana/tree/selective-highway-zoom) of openstreetmap-americana.  This provides a simplified rendering of the highway network which excludes many minor motorway roads and sections.
![image](https://user-images.githubusercontent.com/3254090/126420985-c380b54b-a991-4a7c-a4c1-40a5cf5ec6b0.png)

Below is zoom level 6 on the current [openstreetmap-americana](https://zelonewolf.github.io/openstreetmap-americana/#6/42.148/-73.712).  Notice the many additional stubs, motorway islands, and minor routes which are present.  Instead of rendering `highway=trunk` to make this network look nice, we can instead suppress non-interstate roads and withhold motorway+trunk rendering to a closer zoom:
![image](https://user-images.githubusercontent.com/3254090/126421148-71e780ec-991d-4e38-a82a-85015b349e97.png)
2021-08-04 10:10:16 +02:00
Brian Sperlongano
79c2ec929d
Add new layer to serve highway=motorway_junction nodes (#1119)
This PR adds a layer for `highway=motorway_junction` features.

This implementation of highway exits in the transportation_name layer add to the existing layer table structure, and renames the internal column name from "construction" (which was already overloaded with non-construction usages) to "subclass", which will be less confusing to future developers.  The string 'junction' is used as the universal sub-class for highway exits.

A new documentation PR has been opened at openmaptiles/www.openmaptiles.org#69 to reflect these changes in the documentation.
2021-06-28 14:35:27 +02:00
Tomas Pohanka
d0ebdde458
Fix transportation road segments disconnection (#1109)
To avoid discontinuous transportation lines between zooms 9 and 11. 
 - Originally limit geometry by length for z9 - z11 (`ST_Length(geometry) > ZRes(11)`)
 - highway z9 to z11 was generalized during import-osm
 - now just create a filtered and generalized z11 table
 - then merge segments in the same way as from (full-featured) osm_highway_linestring and used this merged z11 for mat.view z10 and z9

Close #1107
2021-04-30 15:39:56 +02:00
Frédéric Rodrigo
50d569dfee
Remove useless indices on osm_transportation_merge_linestring_genX (#1079)
On creation of materialized views `osm_transportation_merge_linestring_gen_zX`, an index is created each time the new view and not used else where than the next generalization step.
It does not worth it to to a scan on the table to create index that will be used only once, while the next generalisation step can also be done in one full scan.
So, removing these indices.
2021-03-05 09:42:27 +01:00
Adam Laža
77f5d76e37
Refactor schema, one _gen_z view per zoom. (#1045)
* Refactor layer aeroway.

* Refactor layer boundary.

* Refactor layer landcover.

* Refactor layer landuse.

* Refactor layer park.

* Refactor layer transportation.

* Refactor layer water.

* Refactor layer waterway.

* Re-generate water* layers etl_diagrams.

* Regenerate etl_diagrams for waterway.

* Cast NULL to text.
2020-11-26 13:54:00 +01:00
Eva Jelinkova
0aa6648bd9
adding brunnel into z<9 (#1038)
Following https://github.com/openmaptiles/openmaptiles/pull/1027, solving https://github.com/openmaptiles/openmaptiles/issues/999.

Adding brunnel attributes to merge tables of `transportation` layer, now it is possible to style/filter tunnels, bridges and fords in all zoom levels. Tested on Switzerland and Europe. Changes in generation speed and filesize minimal.
2020-11-11 15:37:09 +02:00
Adam Laža
0dc8c3256c
Rename generic layer.sql to actual_layer_name .sql (#1034)
Following layers had generic `layer.sql` name.

- `aerodrome_label`
- `aeroway`
- `housenumber`
- `mountain_peak`
- `park`
- `place`
- `poi`
- `transportation`
- `transportation_name`
- `water_name`

This PR renames `layer.sql` to `aerodrome_label.sql`, `aeroway.sql`...
2020-11-09 11:43:35 +02:00
Eva Jelinkova
341c4df171
Brunnel changes in transportation and transportation_name layers (#1027)
https://github.com/openmaptiles/openmaptiles/issues/999

- Point 1 solved only partially - there is `brunnel` value added into zooms 9, 10, 11. For zooms 8 and smaller it would need change in osm_transportation_merge_ tables - I am going to try it but maybe its SQL costs will be to high.

- Point 2 solved.
2020-10-14 15:26:43 +03:00
Eva Jelinkova
3d21523fbe
aerialway class for the transportation layer (#1011)
Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
2020-10-12 12:44:33 +02:00
Frédéric Rodrigo
36b7533d3b
Add timers to trigger function (#1006)
Add timer to all trigger refresh functions to help point where the time is spend on update.
2020-10-08 16:57:16 +03:00
Frédéric Rodrigo
883a997dcf
Remove large unused index osm_highway_linestring_highway_idx (#1002)
The index `osm_highway_linestring_highway_idx` is not used. There is also a used partial index `osm_highway_linestring_highway_partial_idx`.

I cross check the usage of the index from postgres query stats and all SQL query code where the table `osm_highway_linestring` is refered.

This index is relatively big.
2020-09-30 11:30:18 +03:00
Frédéric Rodrigo
9bb17792a6
Remove alignment of AS in SQL and few others (#932)
* Remove alignment of AS in SQL

* Remove alignment of CREATE TABLE in SQL
2020-07-22 13:48:25 +02:00
Yuri Astrakhan
2b95d1cffa
Fix & optimize incorrect function declarations (#918)
* All functions that access database must be declared as `STABLE`, not `IMMUTABLE` -- because database can change at any moment, e.g. during an update
* there are a few functions that could be made `STRICT` -- passing `NULL` as a parameter will always result in a `NULL`, but for some reason that causes a significant decrease in perf.
* tagged one function as parallel safe

NOTE: somehow `ST_AsMVT()` method of tile generation is showing 70-90% slowdown with this patch. I am not sure of why this is happening. If the reason is the `IMMUTABLE` -> `STABLE` change, we may have to dig deeper into PG optimization
2020-06-17 12:15:26 -04:00
Yuri Astrakhan
6457419e0d
NOOP: Format all layer's SQL code (#917)
I would like to reformat all of our SQL to have a concise coding style.
This makes it far easier to understand the code for a casual contributor,
and lets us spot errors more easily.

Most importantly, it makes it much easier to grep (search) the code because it is more likely to be in the same syntax

Some key changes:
* SQL keywords are always UPPERCASE, e.g. `SELECT WHEN AS END ...`
* types, variables, aliases, and field names (identifiers) are always lower case
* `LANGUAGE 'plpgsql'` is now `LANGUAGE plpgsql` (no quotes)
* a few minor spacing/semicolon cleanups

P.S. Per @TomPohys request, `TABLE` is spelled using upper case despite being a type for consistency with PG Docs. Same for `LANGUAGE SQL` vs `LANGUAGE plpgsql`.
2020-06-08 12:19:55 -04:00
Jorge Sanz
ace759590e
Parallel capability to layer functions (#728)
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.

`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).

### Testing procedure

The procedure to test this was:

* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze 
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.

<details><summary>Testing query</summary>

```sql
-- Using the function layer_landuse
explain analyze 
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>

You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities

![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png)

### Next steps

Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.

I will post my finding here, even no change in the code should happen.


cc. @nyurik

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2020-01-31 19:36:02 -05:00
Yuri Astrakhan
6801353e15
Declared field mapping 2 (#734)
Make a few more mappings declarative, and removes values declared in both SQL and the yaml file.

Here's the diff comparing `build/tileset.sql` in master vs the new PR. The changes are mostly stylistic, except when a nested `if` statement is expanded into individual `if ... and ...` conditions (logically identical)

```diff
55c55
diff --git a/build/tileset.sql b/build/tileset.sql
index 4e59357..7c6c444 100644
--- a/build/tileset.sql
+++ b/build/tileset.sql
@@ -52,7 +52,7 @@ CREATE INDEX IF NOT EXISTS osm_ocean_polygon_gen4_idx ON osm_ocean_polygon_gen4
 CREATE OR REPLACE FUNCTION water_class(waterway TEXT) RETURNS TEXT AS $$
     SELECT CASE
            WHEN "waterway" IN ('', 'lake') THEN 'lake'
-           WHEN "waterway"='dock' THEN 'dock'
+           WHEN "waterway" = 'dock' THEN 'dock'
            ELSE 'river'
    END;
 $$ LANGUAGE SQL IMMUTABLE;
@@ -1813,24 +1813,41 @@ CREATE OR REPLACE FUNCTION highway_class(highway TEXT, public_transport TEXT, co
         WHEN "highway" IN ('tertiary', 'tertiary_link') THEN 'tertiary'
         WHEN "highway" IN ('unclassified', 'residential', 'living_street', 'road') THEN 'minor'
         WHEN "highway" IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor')
-            OR "public_transport"='platform'
+            OR "public_transport" = 'platform'
             THEN 'path'
-        WHEN "highway"='service' THEN 'service'
-        WHEN "highway"='track' THEN 'track'
-        WHEN "highway"='raceway' THEN 'raceway'
-        WHEN highway = 'construction' THEN CASE
-          WHEN construction IN ('motorway', 'motorway_link') THEN 'motorway_construction'
-          WHEN construction IN ('trunk', 'trunk_link') THEN 'trunk_construction'
-          WHEN construction IN ('primary', 'primary_link') THEN 'primary_construction'
-          WHEN construction IN ('secondary', 'secondary_link') THEN 'secondary_construction'
-          WHEN construction IN ('tertiary', 'tertiary_link') THEN 'tertiary_construction'
-          WHEN construction IN ('', 'unclassified', 'residential', 'living_street', 'road') THEN 'minor_construction'
-          WHEN construction IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor')
-              OR public_transport = 'platform' THEN 'path_construction'
-          WHEN construction = 'service' THEN 'service_construction'
-          WHEN construction = 'track' THEN 'track_construction'
-          WHEN construction = 'raceway' THEN 'raceway_construction'
-        END
+        WHEN "highway" = 'service' THEN 'service'
+        WHEN "highway" = 'track' THEN 'track'
+        WHEN "highway" = 'raceway' THEN 'raceway'
+        WHEN "highway" = 'construction'
+            AND "construction" IN ('motorway', 'motorway_link')
+            THEN 'motorway_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" IN ('trunk', 'trunk_link')
+            THEN 'trunk_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" IN ('primary', 'primary_link')
+            THEN 'primary_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" IN ('secondary', 'secondary_link')
+            THEN 'secondary_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" IN ('tertiary', 'tertiary_link')
+            THEN 'tertiary_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" IN ('', 'unclassified', 'residential', 'living_street', 'road')
+            THEN 'minor_construction'
+        WHEN "highway" = 'construction'
+            AND ("construction" IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor') OR "public_transport" = 'platform')
+            THEN 'path_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" = 'service'
+            THEN 'service_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" = 'track'
+            THEN 'track_construction'
+        WHEN "highway" = 'construction'
+            AND "construction" = 'raceway'
+            THEN 'raceway_construction'
     END;
 $$ LANGUAGE SQL IMMUTABLE;
 
@@ -4073,6 +4090,12 @@ RETURNS TEXT AS $$
         WHEN "subclass" IN ('fast_food', 'food_court') THEN 'fast_food'
         WHEN "subclass" IN ('park', 'bbq') THEN 'park'
         WHEN "subclass" IN ('bus_stop', 'bus_station') THEN 'bus'
+        WHEN ("subclass" = 'station' AND "mapping_key" = 'railway')
+            OR "subclass" IN ('halt', 'tram_stop', 'subway')
+            THEN 'railway'
+        WHEN "subclass" = 'station'
+            AND "mapping_key" = 'aerialway'
+            THEN 'aerialway'
         WHEN "subclass" IN ('subway_entrance', 'train_station_entrance') THEN 'entrance'
         WHEN "subclass" IN ('camp_site', 'caravan_site') THEN 'campsite'
         WHEN "subclass" IN ('laundry', 'dry_cleaning') THEN 'laundry'
@@ -4082,7 +4105,7 @@ RETURNS TEXT AS $$
         WHEN "subclass" IN ('hotel', 'motel', 'bed_and_breakfast', 'guest_house', 'hostel', 'chalet', 'alpine_hut', 'dormitory') THEN 'lodging'
         WHEN "subclass" IN ('chocolate', 'confectionery') THEN 'ice_cream'
         WHEN "subclass" IN ('post_box', 'post_office') THEN 'post'
-        WHEN "subclass"='cafe' THEN 'cafe'
+        WHEN "subclass" = 'cafe' THEN 'cafe'
         WHEN "subclass" IN ('school', 'kindergarten') THEN 'school'
         WHEN "subclass" IN ('alcohol', 'beverages', 'wine') THEN 'alcohol_shop'
         WHEN "subclass" IN ('bar', 'nightclub') THEN 'bar'
@@ -4098,9 +4121,6 @@ RETURNS TEXT AS $$
         WHEN "subclass" IN ('bag', 'clothes') THEN 'clothing_store'
         WHEN "subclass" IN ('swimming_area', 'swimming') THEN 'swimming'
         WHEN "subclass" IN ('castle', 'ruins') THEN 'castle'
-        WHEN (subclass = 'station' AND mapping_key = 'railway')
-          OR (subclass IN ('halt', 'tram_stop', 'subway')) THEN 'railway'
-        WHEN (subclass = 'station' AND mapping_key = 'aerialway') THEN 'aerialway'
         ELSE subclass
     END;
 $$ LANGUAGE SQL IMMUTABLE;
@@ -4301,22 +4321,22 @@ $$
     COALESCE(NULLIF(name_de, ''), name, name_en) AS name_de,
     tags,
     CASE
-      WHEN "aerodrome"='international'
-          OR "aerodrome_type"='international'
+      WHEN "aerodrome" = 'international'
+          OR "aerodrome_type" = 'international'
           THEN 'international'
-      WHEN "aerodrome"='public'
-          OR "aerodrome_type"='civil'
+      WHEN "aerodrome" = 'public'
+          OR "aerodrome_type" = 'civil'
           OR "aerodrome_type" LIKE '%public%'
           THEN 'public'
-      WHEN "aerodrome"='regional'
-          OR "aerodrome_type"='regional'
+      WHEN "aerodrome" = 'regional'
+          OR "aerodrome_type" = 'regional'
           THEN 'regional'
-      WHEN "aerodrome"='military'
+      WHEN "aerodrome" = 'military'
           OR "aerodrome_type" LIKE '%military%'
-          OR "military"='airfield'
+          OR "military" = 'airfield'
           THEN 'military'
-      WHEN "aerodrome"='private'
-          OR "aerodrome_type"='private'
+      WHEN "aerodrome" = 'private'
+          OR "aerodrome_type" = 'private'
           THEN 'private'
       ELSE 'other'
     END AS class,
```
2020-01-31 00:22:21 -05:00
Yuri Astrakhan
9d6dbfc64f
Use one pass docs image generation (#751)
quicker and cleaner diagram image generation.
Remove etl-graph and mapping-graph targets - redundant

Also, the obsolete "fields" is still in Imposm's code and both names are accepted,
but "fields" is not documented anywhere (PR submitted), and could be removed at any moment.

Our docs were not supporting it until this PR, so renaming it at the same time.

Several images have been updated due to a more inclusive mapping scan
Requires https://github.com/openmaptiles/openmaptiles-tools/pull/147 (merged)
2020-01-22 21:55:22 -05:00
Yuri Astrakhan
c9e7ad90c6
Remove unneeded "else null" in conditions (#732)
Minor code cleanup:
SQL already returns NULL in the "WHEN" condition
if it is not matched by any of the cases.

Co-authored-by: Eva Jelinkova <evka.jelinkova@gmail.com>
2020-01-22 17:24:28 -05:00
Yuri Astrakhan
e6e92036a0
Use _resolve_wikidata in layer mapping.yaml (#731)
Mark all tables that should not be populated with the Wikidata
international labels with a special OMT-specific flag.

This should be ok to merge even before the new tools version
is used because imposm seems to ignore anything it doesn't understand.

The next tools version will remove it when generating imposm mapping file.
2020-01-21 09:43:21 -05:00