[openstreetmap/openstreetmap-website] Integrate the high-performance GPX importer (#1852)

[mmd]

> (perhaps this is what activerecord-import helps with?).

Yes, it seems to be effective. I'm currently playing around with the same tricks employed in the diff_upload pr, i.e. a combination of turning off pointless checks for each gps point, as well as combining multiple inserts into one large insert. Perhaps someone much more familiar with Rails should review, if this all makes sense.

One my test GPX with 4000 points this reduces runtime from 20+s down to some 200ms. 

```
D, [2018-10-03T16:49:57.057361 #20880] DEBUG -- :   Trace Load (0.1ms)  SELECT  "gpx_files".* FROM "gpx_files" WHERE "gpx_files"."id" = $1 LIMIT $2  [["id", 21], ["LIMIT", 1]]
D, [2018-10-03T16:49:57.057420 #20880] DEBUG -- :   ↳ app/models/trace.rb:314
D, [2018-10-03T16:49:57.057880 #20880] DEBUG -- :   User Load (0.1ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
D, [2018-10-03T16:49:57.057956 #20880] DEBUG -- :   ↳ app/models/trace.rb:314

```


```
--- a/app/models/trace.rb
+++ b/app/models/trace.rb
@@ -28,6 +28,8 @@
 class Trace < ActiveRecord::Base
   self.table_name = "gpx_files"
 
+  attr_accessor :skip_uniqueness
+
   belongs_to :user, :counter_cache => true
   has_many :tags, :class_name => "Tracetag", :foreign_key => "gpx_id", :dependent => :delete_all
   has_many :points, :class_name => "Tracepoint", :foreign_key => "gpx_id", :dependent => :delete_all
@@ -37,7 +39,7 @@ class Trace < ActiveRecord::Base
   scope :visible_to_all, -> { where(:visibility => %w[public identifiable]) }
   scope :tagged, ->(t) { joins(:tags).where(:gpx_file_tags => { :tag => t }) }
 
-  validates :user, :presence => true, :associated => true
+  validates :user, :presence => true, :associated => true, :unless => :skip_uniqueness
   validates :name, :presence => true, :length => 1..255
   validates :description, :presence => { :on => :create }, :length => 1..255
   validates :timestamp, :presence => true

```
```

--- a/app/models/tracepoint.rb
+++ b/app/models/tracepoint.rb
@@ -25,9 +25,10 @@ class Tracepoint < ActiveRecord::Base
 
   self.table_name = "gps_points"
 
+  attr_accessor :skip_uniqueness
   validates :trackid, :numericality => { :only_integer => true }
   validates :latitude, :longitude, :numericality => { :only_integer => true }
-  validates :trace, :associated => true
+  validates :trace, :associated => true, :unless => :skip_uniqueness
   validates :timestamp, :presence => true
 
   belongs_to :trace, :foreign_key => "gpx_id"


```

```
    tps = []
    gpx.points do |point|
      if first
        f_lat = point.latitude
        f_lon = point.longitude
        first = false
      end

       tp = Tracepoint.new
       tp.lat = point.latitude
       tp.lon = point.longitude
       tp.altitude = point.altitude
       tp.timestamp = point.timestamp
       tp.gpx_id = id
       tp.trackid = point.segment
       tp.skip_uniqueness = true
       tps << tp
    end
    
    Tracepoint.transaction do
      Tracepoint.import(tps)
    end
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/issues/1852#issuecomment-426683267
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20181003/238a41f0/attachment.html>