[openstreetmap/openstreetmap-website] Copies notes details from first comments to notes table (PR #5667)

[Andy Allan]

gravitystorm left a comment (openstreetmap/openstreetmap-website#5667)

In my own projects, we use a different approach for data migrations.

* First, we avoid using rails built-in database migrations. These are great for structure changes, but not for data changes, since they can't be tested. And there's often some edge cases that we want to have confidence about before we run the data migration and tests are the foundation for confidence.
* We create a job to migrate the data, so that it runs in a background process. We have a rake task that simply queues the job and nothing else. This is because rake tasks are convenient for us to run, but again are very hard to test. Jobs are easy to test.
* We have a separate queue for slow jobs, with a separate set of workers, so that if it takes e.g. 10min to run we are still sending out confirmation emails in the meantime.
* For the task itself, we write it in a way that it can be retried multiple times. So for example we only select the objects that have not yet been migrated (e.g. `Note.where(description: nil).in_batches`) so that if the job is retried, it starts from where it left off, not from the start again.

We do this because in so many cases either we forgot about an edge case but picked it up in testing, or there turns out to be some unexpected data that messes things up (an example would be a note with no comments, even if there's no way that could have happened, in theory, somehow it always does). It also avoid any worries about locks, long-running transactions, transactions with large amounts of uncommitted changes, timeouts, network interruptions, db process restarts and so on.

I say all this to provide an alternative approach that I know works. But in this case, since @tomhughes will be running the actual migration process, whatever approach he prefers is what we should do!

-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/5667#issuecomment-2659898440
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/openstreetmap-website/pull/5667/c2659898440 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20250214/c4d63450/attachment.htm>