[openstreetmap/openstreetmap-website] Integrate the high-performance GPX importer (#1852)

[Andy Allan]

For a long time the OSMF has used a [high performance GPX importer](https://github.com/openstreetmap/gpx-import) to process uploaded traces. This runs as a separate daemon on the servers, unrelated to this code.

This has a number of drawbacks:

* There is a lot of ruby code in this repo for importing traces that is [unused and bitrotting].(https://github.com/openstreetmap/openstreetmap-website/issues/281)
* The emails sent by the GPX importer [are not translated](https://github.com/openstreetmap/gpx-import/tree/master/templates).
* There is no queue management for pending traces, so only one import process (on one server) can run at any time.
* The daemon needs its own installation and configuration, its own connections to the database, and its own ability to send mail.

Having a daemon running background tasks is fairly standard in the modern rails world. So I'd like to propose some changes:

* We use the [ActiveJob](http://guides.rubyonrails.org/active_job_basics.html) framework for managing the queue of traces that need importing.
* We rework the C code as a ruby gem, so that we can declare a dependency in the Gemfile and it gets installed and compiled for 'free' with bundler.
* We refactor so that only the parsing of the trace files is handled by the C code, and the email notifications and any other scaffolding comes from this rails app. This will allow us to support translations.
* We ensure that as many worker processes as are necessary can be run in parallel, to improve import speeds.

I'm not familiar at all with the C code, nor the operations of it in production, so any advice on this proposal is welcome!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/issues/1852
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20180509/aeb4bd8a/attachment-0001.html>