<br><br><div class="gmail_quote">On Tue, Oct 28, 2008 at 7:39 AM, Michal Migurski <span dir="ltr"><<a href="mailto:mike@stamen.com">mike@stamen.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">>> Finally, the boundaries between the hourlies and dailies seem<br>
>> misaligned.<br>
>><br>
><br>
</div><div class="Ih2E3d">> This shouldn't be the case.<br>
</div><div class="Ih2E3d">>> After running the remaining hourlies for the 22nd, I attempted to<br>
>> pick up on the 23rd with a daily. The final hourly I used was<br>
>> 2008102223-2008102300.osc.gz. It's my expectation that I should be<br>
>> able to immediately follow that with 20081023-20081024.osc.gz, but<br>
>> this led to duplicate key violation suggesting that there's an<br>
>> overlap between the two files. Continuing with hourlies *works*,<br>
>> but is tedious and I suspect slower than the dailies.<br>
>><br>
><br>
</div><div class="Ih2E3d">> You should have been able to do what you've suggested. If you are<br>
> finding problems, please provide me with some example data which is<br>
> misaligned between the two types of changesets.<br>
<br>
</div>Try the two files mentioned above - that's where I saw this behavior,<br>
they're quite recent.<br>
<br>
2008102223-2008102300.osc.gz<br>
20081023-20081024.osc.gz</blockquote><div><br>I need you to provide some specific examples of broken data. If you can say that "way 27123456 is created in both of the above files even though they are for different time periods" then I can take a look at why this may have occurred. Just saying that there is misalignment between those two files doesn't help me at all. Presumably you ran into a specific problem and received a specific error message, this is the kind of information I need. I only do this project in my spare time and can't go looking for problems that I'm not sure even exist, I have enough known problems to look into already :-)<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div class="Ih2E3d"><br>
<br>
>> My sense from reading other people's experiences has been that it's<br>
>> a common pattern to rely solely on the weekly planet dumps,<br>
>> incurring the substantial overhead of parsing and importing the<br>
>> full 5GB dump once every week, and then re-rendering the complete<br>
>> set of tiles.<br>
>><br>
><br>
</div><div class="Ih2E3d">> For a long time weekly planet dumps were the only bulk data<br>
> available. Osmosis changesets have been on the scene for some time<br>
> now though and are gradually being utilised by more and more<br>
> clients. As the planet grows, this will become more critical. Who<br>
> knows, if the kinks gradually get ironed out of the osm2pgsql<br>
> program we may even begin to see the main mapnik tile generator move<br>
> to using changesets.<br>
<br>
</div>I would love to rely on these exclusively, it's much more efficient.<br>
But, I was seeing a fair bit of information fall through the cracks so<br>
that's why I'm re-synching to planet every four weeks.</blockquote><div><br>Again, please provide some specific examples. If data is being missed I'd like to know about it. Osmosis provides some tools that may be useful here. You can download a planet, apply changesets for a week, then compare against the next planet and see what the differences are. Obviously both planets would need appropriate changesets applied to make them consistent before performing a comparison to eliminate noise.<br>
<br>I probably should do some of these comparisons myself, but again just haven't found time yet and nobody else has complained about missing data. The minute changesets run 5 minutes behind the API so could potentially miss data if a lock is held for several minutes. The daily and hourly changesets run at least 20 minutes behind API (forget off the top of my head) and should be extremely unlikely to miss data.<br>
</div></div>Brett<br><br>