[openstreetmap/openstreetmap-website] Add a gpx_tracks table and convert trace points into linestrings (PR #7348)

[Tom Hughes]

@tomhughes commented on this pull request.



> @@ -431,6 +431,9 @@ Trace:
   description:
     ColumnPresenceChecker:
       enabled: false
+  gpx_tracks:
+    MissingIndexChecker:
+      enabled: false

Why do we need to disable this? It seems odd that we should be creating new tables that fail consistency checks?

> +
+    puts "Backfilling gpx_tracks from trace #{min_id} to #{max_id} in chunks of #{chunk_size}"
+
+    (min_id..max_id).step(chunk_size) do |chunk_start|
+      chunk_end = [chunk_start + chunk_size - 1, max_id].min
+      started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      traces = 0
+      empty = 0
+      segments = 0
+
+      # Traces that already have rows are skipped, so running a range again is cheap.
+      Trace.visible.imported
+           .where(:id => chunk_start..chunk_end)
+           .where("NOT EXISTS (SELECT 1 FROM gpx_tracks WHERE gpx_tracks.gpx_id = gpx_files.id)")
+           .find_each do |trace|
+             inserted = TraceLinestringJob.perform_now(trace)

I'm wondering if pushing the backfill work through the job queue is a good idea - that's a lot of jobs to run and it means this task isn't really doing any work it's just queueing work to do later.

Also those jobs are going to be in the same queue as regular trace imports so we run the risk of swamping that queue and delaying those imports unless we build extra infrastructure to control how this task is run and those jobs submitted.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/7348#pullrequestreview-5213901045
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/openstreetmap-website/pull/7348/review/5213901045 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20260915/a8e97d0b/attachment.htm>