[OSM-dev] Update osm2pgsql patch mode

Karl Newman siliconfiend at gmail.com
Sun Aug 31 23:57:18 BST 2008


On Sun, Aug 31, 2008 at 1:50 PM, Martijn van Oosterhout
<kleptog at gmail.com>wrote:

> Umm, yeah. There's that. The way I solved it was with the patch below,
> which is a gross hack but it works. Basically it turns every create
> into a modify so it deletes any conflicting rows before inserting. It
> may be the only way, but I'm still thinking on it...
>
> Have a nice day,
>
> Index: osm2pgsql.c
> ===================================================================
> --- osm2pgsql.c (revision 10079)
> +++ osm2pgsql.c (working copy)
> @@ -278,6 +278,7 @@
>         xmlFree(xtype);
>     } else if (xmlStrEqual(name, BAD_CAST "create")) {
>         action = ACTION_CREATE;
> +        action = ACTION_MODIFY;
>     } else if (xmlStrEqual(name, BAD_CAST "modify")) {
>         action = ACTION_MODIFY;
>     } else if (xmlStrEqual(name, BAD_CAST "delete")) {
>
>
> On Sun, Aug 31, 2008 at 7:50 PM, Matt Amos <matt at matt-amos.uklinux.net>
> wrote:
> > Martijn van Oosterhout wrote:
> >>
> >> On Fri, Aug 29, 2008 at 5:23 PM, Robert (Jamie) Munro <
> rjmunro at arjam.net>
> >> wrote:
> >>>
> >>> Great progress - that's now about 14 times real time.
> >>>
> >>> How much disc space does the whole operation require?
> >>
> >> On this system 42.5GB. Transient may be more, not sure.
> >
> > on this system it took 22.8h for the initial planet import, and it is
> > currently consuming 44Gb. from munin graphs it seems that max usage was
> > roughly 9Gb more.
> >
> > when i try to load the next day's osc i get the following error:
> >
> > Reading in file: /home/osm/planets/20080827-20080828.osc.gz
> > Processing: Node(50k) Way(0k) Relation(0k)insert_node failed: ERROR:
> > duplicate key value violates unique constraint "planet_osm_nodes_pkey"
> > (7)
> > Arguments were: 291466881, 6319926.5639616642, -13695169.0319488477,
> (null),
> > Error occurred, cleaning up
> >
> > the node with that ID seems to have been created just after midnight and
> > appears in the both the planet dump and daily diff. is there any way
> around
> > this, or should i just import the hourly diffs from 1am instead?
> >
> > cheers,
> >
> > matt
> >
>

I'm curious why the planet dump doesn't cut off at midnight, so you wouldn't
get these duplicates. Is it because it's way more expensive to compare
timestamps? I doubt it, because the minutely diff generation can run through
the timestamps in the entire database in a few seconds. Am I missing
something, or is it for hysterical raisins (as TomH likes to say) and nobody
ever bothered to change it?

Karl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20080831/4291e714/attachment.html>


More information about the dev mailing list