[Tilesathome] Corrupted data in TRAPI database

[Brett Henderson]

On Tue, Dec 1, 2009 at 10:32 AM, Jeremy Adams <milenko at king-nerd.com> wrote:

> It depends.  If the changeset is uploaded in small chunks to the API, each
>> is written as a separate database transaction and the data will be spread
>> across multiple minute diffs.  If the data is uploaded in a single chunk it
>> is written within a single database transaction and can possibly take longer
>> than 30 minutes to process.  The data only becomes visible to Osmosis after
>> the transaction is committed by which time Osmosis has moved past that time
>> interval and misses the data entirely.
>>
>>
>>>
>>> The fix seems to be using the new replication diffs, but TRAPI doesn't
>>> support those yet.
>>>
>>
>> Yes, they don't use timestamps at all so don't suffer from problems with
>> long running transactions.  If you see any missing data in those files,
>> please let me know.
>>
>>
>
> Say I open a changeset and it takes an hour to upload.  Is the data in the
> changeset being written bit by bit as I'm uploading for that whole hour, or
> is it stored until I'm done uploading and then written to the database as
> one large transaction?  If it's the latter, then I guess it doesn't matter
> how long the changeset is open, only how long it takes the server to write
> the changes to the database.
>

I assume you're referring to uploading an osc xml changeset to the API as
opposed to opening a changeset explicitly and uploading each item
individually.

Honestly I have no idea how the API is implemented.  I believe it makes
changes bit by bit as it receives data but I don't know for sure.  I just
know that some database transactions remain open for longer than half an
hour.


>
> Either way, the solution seems to be using the new replication diffs.  I'm
> no perl wizard, but I might try to muddle my way through and make some
> changes.
>

The new files should be fairly similar to consume.  The key difference is
that each file may contain multiple changes for a single entity.  In other
words, you may receive a version 3 and 4 of an entity in a single diff file.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tilesathome/attachments/20091202/edd3a08b/attachment.html>