Hi Andrew,<br><br><div class="gmail_quote">On Wed, Oct 19, 2011 at 11:54 PM, Andrew Ayre <span dir="ltr"><<a href="mailto:andy@britishideas.com">andy@britishideas.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
On 7/21/2011 9:19 AM, Andy Allan wrote:<br>
<div class="im">> On Thu, Jul 21, 2011 at 9:07 AM, Andrew Ayre <<a href="mailto:andy@britishideas.com">andy@britishideas.com</a>> wrote:<br>
>> Keeping my copy of the planet up to date is a two-step process with<br>
>> Osmosis. Get the latest changes and applying them. This takes about an<br>
>> hour on my server which is enough time for some other user to reboot the<br>
>> server without realizing/knowing.<br></div></blockquote><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="im">
>><br>
>> What protection is there in Osmosis to recover from this without missing<br>
>> any changes? If none, how are people solving this in their scripts?<br></div></blockquote><div><br>Osmosis only updates the local state file after downstream pipeline tasks have completed. This means that if you're writing a PBF or XML file, that file will be complete before the local state file gets updated by the --rri task. If any problems occur prior to that point, the next time you run Osmosis it will simply start again from the previous state file potentially doing the same changes again. It aims to be Idempotent.<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="im">
><br>
</div>> It depends if you run things as two separate stages or not. I try to<br>
> combine anything involving the updates into one command, so that if<br>
> it's interrupted then the state file hasn't changed and can simply be<br>
> run again.<br>
><br>
> So that means doing something along the lines of --rri --simc --rx<br>
> --ac --wx to fetch the changes, simplify, read the planet, apply the<br>
> changes and write it out, all in one go. If you split this into<br>
> multiple osmosis calls, then you'll need to approach it differently -<br>
> perhaps testing whether or not the local changes.osm.gz exists, and if<br>
> it does, skip downloading and go straight to applying it to the<br>
> planet.<br>
<br>
Hello, I am still trying to get this to work properly. Some of the time<br>
I end up with a state.txt that looks like this:<br>
<br>
#Sat Oct 15 00:24:20 CEST 2011<br>
sequenceNumber=16402<br>
timestamp=2011-10-04T08\:00\:00Z<br>
<br>
The timestamp in the comment is correct. The timestamp on the last line<br>
is not - it's stale.<br></blockquote><div><br>The timestamp in the comment specifies when the state file was created, it is informational purposes only and not used anywhere. The timestamp field specifies what point in time the state file represents.<br>
<br>In other words, you downloaded this file at Sat Oct 15 00:24:20 CEST 2011:<br><a href="http://planet.openstreetmap.org/hour-replicate/000/016/402.state.txt">http://planet.openstreetmap.org/hour-replicate/000/016/402.state.txt</a><br>
<br>After that processing run, your planet file only contained data up to October 4th.<br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
On the next attempt to update the planet it fails and I end up with a<br>
planet size of zero bytes. I don't know the error message because I<br>
can't get it to always fail. It seems more likely to do this from a<br>
cronjob I think...<br></blockquote><div><br>Osmosis could be bombing out for any number of reasons. Do you have logs? It may simply have not completed processing when it was interrupted. It downloads all files before sending data through the pipeline. It is also possible that Osmosis is bombing out because you have a large time window to process and it is trying to download a huge number of change files and apply them. It might be running out of file handles (I think the number of threads is constant but I'm vague on the details now). The output planet file will be empty if it fails before writing any data to the destination. You shouldn't lose data though, because the failure should occur prior to the local state.txt being updated.<br>
<br>What value do you have for maxInterval and baseUrl in the --rri configuration file? I'm assuming you're using baseUrl=<a href="http://planet.openstreetmap.org/hour-replicate">http://planet.openstreetmap.org/hour-replicate</a> based on your example sequence number. maxInterval shouldn't be set to 0, but given an upper bound. Some experimentation may be required to find the upper limit for your system.<br>
<br>For updating planet files you may be better off using the --read-change-interval task instead of --read-replication-interval, simply because you'll then be able to download daily change files. I personally use the following settings in the configuration.txt file:<br>
baseUrl=<a href="http://planet.openstreetmap.org/history">http://planet.openstreetmap.org/history</a><br>intervalLength=86400<br>changeFileBeginFormat=yyyy/MMdd<br>changeFileEndFormat=MMdd<br>maxDownloadCount = 100<br> </div>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
I'm using osmosis 0.39.<br>
<br>
/tools/osmosis --rri workingDirectory=/data/ --simc --read-bin<br>
/data/planet-current.osm.pbf --buffer bufferCapacity=12000<br>
--apply-change --buffer bufferCapacity=12000 --write-bin<br>
/data/planet-111019-135247.osm.pbf<br>
<br>
Any ideas what I am doing wrong?<br></blockquote><div><br>Hopefully something above will help.<br><br>Brett<br> <br></div></div>