[Talk-us] Road Expansions and Seattle Imports (if you're the Seattle importer, please read!)

[andrzej zaborowski]

On 19 May 2012 21:02, Serge Wroclawski <emacsen at gmail.com> wrote:
> On Sat, May 19, 2012 at 1:31 PM, Mike N <niceman at att.net> wrote:
>> On 5/19/2012 12:52 PM, Serge Wroclawski wrote:
>
>>
>>  Ultimately, the only way to get a 100% successful upload is to query the
>> changeset on failure to determine what was actually uploaded, then resume
>> from that point.
>
> While true, that method in isolatiuon is a bit complex and should be
> unnecessary.
>
> The OSC upload method is transactional in nature. If you get a
> positive response from it, you can know the upload succeeded. The
> issue is that there can be a network interruption between the upload
> completion and the return response. During that time the transaction
> has completed successfully but the client won't know.
>
> If there is a 1:1 correlation between OSC and changeset, that's fine,
> but generally uploads are often done with large chagesets.
>
> There are a few ways to handle this (including async upload methods in
> the API code, which has been proposed), but right now, AFAIK, there's
> no public code for making reliable mass uploads to OSM.

As mentioned on irc, I trust the tool set I did the original TIGER
expansion with.  I've used it for many other uploads too and I never
had to revert a single upload, i.e. it was always possible to resume
the process in case of a network error or conflict.  I've heard of two
or three more people who used it successfully.  It also (tries to)
solve the problem of nodes being uploaded in bulk before ways get
uploaded, before relations, so this minimizes the probability of
someone spotting orphan nodes in the middle of an import and removing
them.

Unfortunately it's not very user friendly and requires manual
intervention sometimes.  I'd try improving it instead of starting from
the beginning though.

It's documented here: http://wiki.openstreetmap.org/wiki/Upload.py

It is also limited to xml files that fit in memory because it uses the
python ElementTree parser right now.

Cheers