[Tile-serving] [openstreetmap/osm2pgsql] osm2pgsql fails on 100MB+ .pbf files when using Aurora Postgres Serverless DB (#1056)

[Chris Bailey]

Cross posting [this `imposm` issue](https://github.com/omniscale/imposm3/issues/212), since it also affects `osm2pgsql`. In short, `osm2pgsql` does not work when using the AWS Aurora Postgres Serverless database (works fine on non-serverless), whenever you try to import a `.pbf` file that is roughly 100MB or larger, such as Oregon or California extracts from Geofabrik. Smaller extracts, such as Delaware work fine. Here's the error I see:

```
Processing: Node(18314k 188.8k/s) Way(1396k 107.38k/s) Relation(0 0.00/s)COPY_END for COPY planet_osm_ways FROM STDIN;
 failed: ERROR:  unexpected message type 0x2A during COPY from stdin
CONTEXT:  COPY planet_osm_ways, line 1
``` 

I have a support request open with AWS and they are investigating, but so far, from what we can tell, this is some issue that comes up only via use of osm2pgsl or imposm. 

As a workaround, you can do the import using either Aurora Postgres (non-serverless), or regular RDS Postgres, create a snapshot, and then restore that snapshot to an Aurora Postgres Serverless instance in order to use serverless. Aurora Postgres Serverless is Postgres 10.7, PostGIS 2.4. Note though that using the workaround, I use the same Postgres/PostGIS versions, and it works fine, so it seems specific to the serverless version.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1056
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20200107/d6fee5ce/attachment.htm>