[OSM-dev] How to clean up Negative IDs

Ian Dees ian.dees at gmail.com
Tue Aug 3 21:55:06 BST 2010


I imagine the bottleneck is the Railsport doing precondition checks for
everything as it's going in.

I don't think I could give an educated guess for time remaining, but on the
api.osm.org server it usually takes 4+ hours to send in a 50k-change diff
file (around 25MB?). Based on that I'd say you have at least half a day of
waiting left.

On Tue, Aug 3, 2010 at 3:46 PM, Eric Wolf <ebwolf at gmail.com> wrote:

> Just how slow is bulk_upload.py?
>
> I am loading a 177MB .osm file into an empty database on a quad 3.6Ghz Xeon
> with 6GB RAM and 700GB of RAID5. The machine is basically idle except for
> this load.
>
> It's already taken almost an hour.
>
> -Eric
>
> -=--=---=----=----=---=--=-=--=---=----=---=--=-=-
> Eric B. Wolf                           720-334-7734
>
>
>
>
>
> On Tue, Aug 3, 2010 at 12:48 PM, andrzej zaborowski <balrogg at gmail.com>wrote:
>
>> On 3 August 2010 20:28, Eric Wolf <ebwolf at gmail.com> wrote:
>> > This is in reference to the USGS OSMCP project - not the real OSM...
>> > When we imported our chunk of data initially (not me - the guy
>> responsible
>> > is on walkabout in the Rockies), we followed the convention of using
>> > negative IDs in the .OSM file. But osmosis was used to load the data
>> into
>> > the database and now all of our data has negative IDs. This seems to
>> have a
>> > really nasty effect on the API - every time something is edited, a new
>> copy
>> > is created with positive IDs and the old version with the negative IDs
>> > persists.
>> > I assume there is something in the API that says "negative IDs == BAD".
>> I've
>> > been trying to test that theory but keep hitting stumbling blocks.
>> Postgres
>> > doesn't seem to want to let me defer integrity constraints, so my
>> efforts to
>> > change a few IDs to positive values keeps failing. Maybe I've lost my
>> SQL
>> > chops (or maybe I just can't do that as the "openstreetmap" database
>> user).
>> > Am I barking up the right tree? Should I just go ahead and destroy the
>> > database and repopulate it using bulk_upload.py instead of osmosis?
>>
>> If there's no way disable the postgres contraints (I'm sure there is..
>> but I'm a sql noob), I'd filter your .osm file through sed removing
>> the '-' in 'ref="-'  and 'id="-' and reimport with osmosis, or modify
>> your conversion script.  Using bulk_upload.py and the API will take
>> ages.
>>
>> Cheers
>>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100803/1b93eedd/attachment-0001.html>


More information about the dev mailing list