[openstreetmap/openstreetmap-website] Partition large tables (#2076)

[Paul Norman]

>From https://github.com/openstreetmap/openstreetmap-website/issues/375#issuecomment-441660239

> We can't just say that we'll never change anything in our tables ever again! If this does involve large table changes, then we can work out what the zero-downtime migration path would be, and do that. We could practise multi-stage migrations on the smaller tables where there is less impact.

Whenever I describe our large API tables to a PostgreSQL, the number one comment is to partition them. This allows maintenance to be done without needing to sequentially scan billions of rows.

Are there tools for doing this easily in ruby? Should this be a site-specific thing to osm.org? Most users won't have multi-TB tables and it's possible to partition in a way that the rails port doesn't see it.

The benefits of partitioning are

- maintenance is possible
- indexes are smaller
- cache locality is generally improved
- table bloat is lessened
- index bloat is lessened

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/issues/2076
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20181128/960c7a17/attachment.html>