[OSM-dev] Fail to get OsmChange for some changesets
zerebubuth at gmail.com
Sat Oct 9 18:15:30 BST 2010
On Sat, Oct 9, 2010 at 1:04 AM, Manchun Yao <manchuny at microsoft.com> wrote:
>> Yes, it's perfectly possible. Individual changes to the data can occur without the changeset being closed. This is completely by design, although many people are confused into thinking that changesets are atomic or delayed until they are closed or similar. They aren't, and that's deliberate. As for how likely it is to happen without someone trying to do it deliberately? I can't answer that.
> I can imagine this now.:
> Suppose we start with: node a, version 7; node b version 7
> Two persons operate on the these two nodes in change set A and change B respectively with the following sequence:
> - A changes node a first: node a version 7 -> 8
> - B changes node b: node b version 7->8
> - A changes node b: node b version 8->9
> - B changes node a: node a version 8->9
> - A change set closes
> - B change set closes
> So we have:
> Change A: node a, version 7->8, node b version 8->9
> Change B: node a, version 8->9, node b version 7->8
> Although change A closes before change B, if we applied change A first without considering change B at all, there will be problems.
> If this is the case, will the accumulative replication process (applying osc changes daily/minutely) ran into problems?
not in my experience. i've been running a tool consuming minutely
replication osc files since feb and i haven't seen these problems.
this is because the replication osc files do not wait to contain whole
changesets - since they're not atomically committed it wouldn't make
sense. however, the osc files do contain the changes which are applied
to the history tables on the level of database transactions rather
> -----Original Message-----
> From: Andy Allan [mailto:gravitystorm at gmail.com]
> Sent: Thursday, October 07, 2010 3:14 AM
> To: Manchun Yao
> Cc: dev at openstreetmap.org
> Subject: Re: [OSM-dev] Fail to get OsmChange for some changesets
> On Wed, Oct 6, 2010 at 5:59 PM, Manchun Yao <manchuny at microsoft.com> wrote:
>> Hi Andy,
>> If all change sets were created and operated strictly based on API 0.6, would the scenario you described (two change sets affecting different objects in non-sequential orders) ever happen? Is the optimistic locking mechanism meant to avoid such things? The problem might exist in the database because of historical reasons though.
> Yes, it's perfectly possible. Individual changes to the data can occur without the changeset being closed. This is completely by design, although many people are confused into thinking that changesets are atomic or delayed until they are closed or similar. They aren't, and that's deliberate. As for how likely it is to happen without someone trying to do it deliberately? I can't answer that.
>> I indeed want to build the history of all the changes so that I can find the map at some time (for example, Jan. 1, 2008) before. The full-planet dump files at (http://planet.openstreetmap.org/full-experimental) contain historical info. However, the file is a bit too big for my computer. I was trying to test whether I can build the history from ground up for a small area by getting the individual change sets one by one. Looks like there are some issues because some "changes" were made before API 0.6 and I have problem to access them using 0.6 API calls.
> You could do it by collecting all the data that those changesets refer to, and then "replay" all the data changes in the order the individual changes occurred (not in the order that the changesets were either opened or closed, which I've explained isn't necessarily the correct order).
> The wider point is that the history dumps are, almost entirely, currently unusable for most people. If anyone has any suggestions for solving this problem, or wants to try making a toolchain to process the history dumps into history extracts, that would be great. There was discussion previously on adapting osmosis to handle history dumps, but I don't know if that resulted in any code.
>> Do you have any suggestions about how to get these problematic change sets? Maybe someone with a database populated with the full-planet dump can get these change sets for me (145015, 4318130, 4318132, 4318136).
> Perhaps this is supported by one of the read-only api services? I'm not sure if any of them (XAPI, ROMA, OWL etc) have access to enough history though.
> dev mailing list
> dev at openstreetmap.org
More information about the dev