[OSM-talk] Changeset Metadata Db Replication - Enhancements proposal

Pierre Béland pierzenh at yahoo.fr
Sat Jan 9 19:50:37 UTC 2021


Local groups sometimes ask how they could access Changesets metadata to monitor edits. ToeBee ChangesetMD is available since 2012 to replicate OSM changeset metadata in a PostgreSQL database. While the actual script can respond to experienced developpers needs working on linux, the learning curve is a quite steep for many.
https://github.com/ToeBee/ChangesetMD
Note that their are at the beginning of 2021, nearly 100 million records. The complete Replication file is 4Gb at the beginning of 2021.

mvexel has made a proposition to upgrade it to python3 since python2 is not supported since 2021.
I myself propose some enhancements based on Python 3  that could respond to other needs than Complete operational database.Various local groups could for example extract metadata for only a week that correspond to a local event to monitor.

This is a prototype that i Invite you to look, test it if you already have a PosgreSQL Db (9.5+) installed. You can easily create a new Database and schema from the Windows Powershell script provided. A test file is provided for the first Insert to the Db. It is also possible to replicate from the Planet Replication site. 

I have developped and tested this on my laptop. The tests available work with sample data and are completed in minutes.
For many of us, it is a good time to test early 2021. Have fun!

If you create a test Repository and try to replicate from the Planet Replication site, you dont need to let these scripts run for hours. You can at any time interrupt the process and records will have been kept up to the last Bulk rows inserted. You can even restart the replication since the sequence of the last file replicated is kept in the database.

 See documentation at https://github.com/pierzen/ChangesetMD/tree/dopartial-logThe test subdirectory contains Test files (Windows OS Powershell scripts) that make it easy to test. Linuxians should be able to adapt these scripts easily.
 
Pierre 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20210109/e0c62a05/attachment.htm>


More information about the talk mailing list