[OSM-dev] full history planet (experimental)

[Anthony]

On Sun, Jan 3, 2010 at 12:14 AM, Lars Francke <lars.francke at gmail.com>wrote:

> On Sat, Jan 2, 2010 at 19:17, Anthony <osm at inbox.org> wrote:
> > 4 nodes + 25 ways + 8 relations=37 changes.  But it's listed as
> > num_changes=35.  And it's not because some elements have multiple
> versions,
> > because excluding duplicates would bring the number below 35.
>
> You're right. I tried to check everything and num_changes seems to be
> updated correctly in every API call. So I don't really have a clue why
> this might happen. Is this the only changeset with that problem or
> could you find multiple problems?
>

I found 31373 of them.  Various different created_by tags too, so it's not
just a Potlatch thing (which was my first guess).  If you want I can get you
the list, but for now here are ten (in whatever order postgres happens to
spit them out):

   id    | num_changes_according_to_changeset | actual_num_changes
---------+------------------------------------+--------------------
 1112512 |                                 84 |                 85
 1112541 |                               2679 |               2873
 1112579 |                                389 |                391
 1112647 |                                538 |                554
 1112677 |                                284 |                290
 1112689 |                                158 |                159
 1112715 |                                 43 |                 44
 1112759 |                                462 |                475
 1112783 |                                475 |                489
 1112792 |                                 13 |                 14

Actual is greater than num_changes in 28745.  Actual is less than
num_changes in 2628.  However, in all 2628 instances of the actual being
less than num_changes, num_changes is 50000.  Here are ten (whatever order
postgres gives me):

  id  | num_changes_according_to_changeset | actual_num_changes
------+------------------------------------+--------------------
 3161 |                              50000 |              49994
 3466 |                              50000 |              49996
 4349 |                              50000 |              49998
 5139 |                              50000 |              49999
 5596 |                              50000 |              49998
 5863 |                              50000 |              49989
 6604 |                              50000 |              49997
 7403 |                              50000 |              49998
 7747 |                              50000 |              49998
 8000 |                              50000 |              49981

I guess I should really take the time to try to validate the history
> export some day.
>

I finally finished importing and indexing everything today, in roughly the
same database schema as Rails Port (plus/minus a few tweaks here and there),
so if you have any queries in particular you're interested in me running
(and they can finish within say 10 hours or so on my machine), let me know.

Right now I'm running a pg_dump on everything.  I'm not sure if that'll
finish overnight or not.  My next step is to go through a random sampling of
dailies and compare everything row by row.  Then I'm all set to start
integrating the minutely-replicate files to get myself up to date.

While I'm thinking of it, I have two questions about things in the db schema
that seem weird:

1) changeset_tags has no primary key?  Based on my experiments with the API
(id, k) seems to be the correct primary key (the API won't let me add two
tags with the same key on a single changeset).
2) Why is the primary key on relation_members so long?  Wouldn't (id,
version, sequence_id) be the correct primary key?  Again experimenting with
the API, it looks like order is preserved regardless of type, role, or id.

In case you haven't seen it, I put the database schema (from a pg_dump of a
Rails Port installation) up at
http://wiki.openstreetmap.org/wiki/Database_schema
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100103/2f365abd/attachment.html>