[OSM-dev] Rantings about API 0.6

[Iván Sánchez Ortega]

Hi all,

I have good news and bad news.

The good news is, I have a brand new bulk uploader that works with 0.6 and its 
limitations (unless proven otherwise).

The bad news is, I've had a look at the SQL queries the Rails port is making 
to the DB during a mass import, and all I could think is "WTF?!?"

(Rants follows)

So, let's have a look at this:

[2009-02-10 01:00:34.809017 #1664]   User Load (0.000144)   SELECT * FROM 
`users` WHERE (`users`.`id` = 1)
[2009-02-10 01:00:34.810457 #1664]   SQL (0.000144)   SELECT `display_name` 
FROM `users` WHERE (`users`.display_name = 'ivansanchez' AND `users`.id <> 1)
[2009-02-10 01:00:34.811241 #1664]   SQL (0.000108)   SELECT `email` FROM 
`users` WHERE (`users`.email = 'ivan at sanchezortega.es' AND `users`.id <> 1)

WTF does this happen on every object (read: node) I upload? The API already 
knows who I am the second I started to upload the diff!


[2009-02-10 01:00:34.863175 #1664]   SQL (0.000244)   SELECT `id` FROM 
`current_nodes` WHERE (`current_nodes`.id IS NULL)

current_nodes.id is defined as NOT NULL. So, WTF?


[2009-02-10 01:00:34.864872 #1664]   SQL (0.000258)   SELECT `id` FROM 
`changesets` WHERE (`changesets`.id = 44 AND `changesets`.id <> 44)

***WTF***???!!!!


[2009-02-10 01:00:34.859181 #1664]   Changeset Update (0.000314)   UPDATE 
`changesets` SET `num_changes` = 18443 WHERE `id` = 44

Couldn't this wait until the diff upload is complete?

(Rants kind of end)


So, while I'm no expert about Rails, I do suspect that all those extra 
unnecessary queries are done automagically.

I do know OSM will have some brand new DB server (now with 0.1 more API!) that 
should be able to handle the extra overhead, and that most of those queries 
will hit data that will be cached anyway.

I also do think that there is a lot of room for optimizations (namely, removal 
of DB overhead), so...



Is it worth to optimize the code for diff uploads?


Cheers,
-- 
----------------------------------
Iván Sánchez Ortega <ivan at sanchezortega.es>

Q:	What's the difference between Bell Labs and the Boy Scouts of America?
A:	The Boy Scouts have adult supervision.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20090210/20f02ed7/attachment.pgp>