[Tile-serving] [openstreetmap/osm2pgsql] --append is not working properly (data is deleted when it shouldn't) (#938)

[crnm21]

So I did one of the biweekly updates again this night and maybe found a probable reason for the problem.

I do fetch the data each second friday at 1:15 UTC with

`osmupdate --day $lastrundate $tmpfile --kepp-tempfiles` where `$lastrundate` ist the date of the lastrun, in todays case it would be 2020-03-13T00:00:00Z as it did last run at Friday 13th of March at 1:15 UTC.

Now this is the list of tempfiles I got (I started the copied list some days earlier than this times update run):

```-rw-r--r--  1 [osmuser] [osmuser]   83 Mar  6 00:06 temp.d000002732.txt
-rw-r--r--  1 [osmuser] [osmuser]  85M Mar  7 00:06 temp.d000002733.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar  7 00:06 temp.d000002733.txt
-rw-r--r--  1 [osmuser] [osmuser]  64M Mar  8 00:06 temp.d000002734.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar  8 00:06 temp.d000002734.txt
-rw-r--r--  1 [osmuser] [osmuser]  76M Mar  9 00:06 temp.d000002735.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar  9 00:06 temp.d000002735.txt
-rw-r--r--  1 [osmuser] [osmuser]  94M Mar 10 00:06 temp.d000002736.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 10 00:06 temp.d000002736.txt
-rw-r--r--  1 [osmuser] [osmuser]  76M Mar 11 00:06 temp.d000002737.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 11 00:06 temp.d000002737.txt
-rw-r--r--  1 [osmuser] [osmuser]  93M Mar 12 00:06 temp.d000002738.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 12 00:06 temp.d000002738.txt
-rw-r--r--  1 [osmuser] [osmuser]  95M Mar 13 00:06 temp.d000002739.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 14 00:07 temp.d000002740.txt
-rw-r--r--  1 [osmuser] [osmuser]  72M Mar 15 00:06 temp.d000002741.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 15 00:06 temp.d000002741.txt
-rw-r--r--  1 [osmuser] [osmuser]  68M Mar 16 00:06 temp.d000002742.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 16 00:06 temp.d000002742.txt
-rw-r--r--  1 [osmuser] [osmuser]  92M Mar 17 00:07 temp.d000002743.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 17 00:07 temp.d000002743.txt
-rw-r--r--  1 [osmuser] [osmuser]  88M Mar 18 00:06 temp.d000002744.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 18 00:06 temp.d000002744.txt
-rw-r--r--  1 [osmuser] [osmuser]  88M Mar 19 00:07 temp.d000002745.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 19 00:07 temp.d000002745.txt
-rw-r--r--  1 [osmuser] [osmuser]  95M Mar 20 00:07 temp.d000002746.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 20 00:07 temp.d000002746.txt
-rw-r--r--  1 [osmuser] [osmuser]  94M Mar 21 00:06 temp.d000002747.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 21 00:06 temp.d000002747.txt
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 22 00:06 temp.d000002748.txt
-rw-r--r--  1 [osmuser] [osmuser]  88M Mar 23 00:06 temp.d000002749.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 23 00:06 temp.d000002749.txt
-rw-r--r--  1 [osmuser] [osmuser] 100M Mar 24 00:07 temp.d000002750.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 24 00:07 temp.d000002750.txt
-rw-r--r--  1 [osmuser] [osmuser]  89M Mar 25 00:06 temp.d000002751.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 25 00:06 temp.d000002751.txt
-rw-r--r--  1 [osmuser] [osmuser]  85M Mar 26 00:07 temp.d000002752.osc.gz
-rw-r--r--  1 [osmuser] [osmuser]   83 Mar 26 00:07 temp.d000002752.txt
-rw-r--r--  1 [osmuser] [osmuser]  96M Mar 27 00:06 temp.d000002753.osc.gz```

As you can see, the 2739.txt from March 13th is missing
as well as the 2740.osc.gz from March 14th.
So it does look as if I miss a day the way I'm doing it.
But I don't get why this is happening.

a) could I solve this by updating every two weeks but setting the update-from date to 15 days ago?

b) could I repair this by doing the update to the db again but this time running the last 15 days instead of the last 14?


(Also, as a sidenote, I had osm2gsql producing an error last time about not being able to close the zip at the end but I thought that is hopefully not important as it being the last step of cleaning up the operation - or am I wrong here?).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/938#issuecomment-604900780
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20200327/7880f702/attachment-0001.htm>