[OSM-dev] Osmosis Replication Statistics
brett at bretth.com
Wed Aug 22 14:46:34 BST 2007
I'm inching ever closer towards osmosis being ready for wider use. This
is just an FYI for anybody who is interested to know where I'm at. I've
recently been testing osmosis replication features against full osm
history data and fixing up the various issues that I've encountered as a
result of this.
This email is pretty long, it starts with some stats breaking down the
time taken for osmosis to perform various change operations. For now
I'm focussing on the generation of snapshot and change files. Applying
them to a database will come later, the tasks are feature complete but
not heavily tested yet and a lower priority. Later in the email I
describe a couple of problems I'm facing.
I produced a set of monthly changesets from the start of 2006. The file
sizes are as follows:
The most recent one (20070701-20070801.osc) is quite large so I'll focus
on it. It took 43m26s to produce.
Splitting into 7 day intervals and ignoring the last couple of days
produces the following stats.
File Size Duration
20070701-20070708.osc 164027848 10m37s
20070708-20070715.osc 220498572 10m10s
20070715-20070722.osc 347431979 14m42s
20070722-20070729.osc 253686702 13m40s
Selecting the biggest file in the interval (20070715-20070722.osc) and
breaking into 1 day intervals produces the following stats.
File Size Duration
20070715-20070716.osc 16684056 1m41s
20070716-20070717.osc 44242820 2m55s
20070722-20070729.osc 62863288 3m02s
20070718-20070719.osc 45727744 2m31s
20070719-20070720.osc 54787426 2m27s
20070720-20070721.osc 77041093 3m07s
20070721-20070722.osc 63953029 2m44s
Selecting the biggest file in the interval (20070720-20070721.osc) and
breaking into 4 hour intervals produces the following stats.
File Size Duration
2007072000-2007072004.osc 8932383 10.229s
2007072004-2007072008.osc 17590822 17.930s
2007072008-2007072012.osc 12615890 16.762s
2007072012-2007072016.osc 14360116 21.107s
2007072016-2007072020.osc 13339044 23.095s
2007072020-2007072100.osc 10841897 31.845s (Note: running a second time
Selecting the biggest file in the interval (2007072004-2007072008.osc)
and breaking into 1 hour intervals produces the following stats.
File Size Duration (run 3 times)
2007072004-2007072005.osc 3160144 4.872s,4.655s,4.985s
2007072005-2007072006.osc 4156641 5.249s,5.185s,4.729s
2007072006-2007072007.osc 5766073 9.204s,8.871s,8.638s
2007072007-2007072008.osc 4584598 4.250s,4.650s,3.983s
Another bit of news is that I've just written merge tasks that will
allow multiple files (both base osm files and change files) to be merged
together. If 3 minutes to produce the daily change file extraction is
too long, the extraction can be broken down into smaller time intervals
and the result files merged together afterwards. Martijn requested this
feature and may use it to help process his AND files.
So how long will it take to generate a new daily planet from the
previous daily planet. My next task is to pick the biggest daily change
file (20070720-20070721.osc) and apply it to 20070720.osm but I need to
generate 20070720.osm first. I hope to do this in the next day or so.
In the meantime, this will give some indication. Applying all 2006
change files in a single pipeline against 20060101.osm to produce
20070101.osm took 59m29s. The result is 6146181638 bytes in size (yeah,
huge for some reason, more on that in a sec). So it appears that
generating a new daily change set takes less than 4 minutes in its
current state. Applying this to an existing planet will take very
approximately one hour and can be performed offline.
Now, back to the huge planet file. I have no idea why the planet file
is so big. My first thought was that my 12-month change application
process was wrong so I took a snapshot at 20070101 to verify the
results. The files were *almost* identical in size. The differences
are contained in the file attached to this email.
So two problems to focus on:
1. Why is the planet file is so big?
2. Why do I have differences between my snapshotted 20070101 planet and
my derived 20070101 using 12 months of changes?
I have no idea what is causing this huge planet. My only thought is
that perhaps data exists in the history tables that doesn't exist in the
current tables. I'm really not sure what's going on here. I need to
look into this further.
I've examined a random sample of the changes between my two 20070101.osm
files. For each change I examined the history of the entity in
question. In every case I've checked the change can be explained by the
fact that the two most recent history rows (as of beginning 2007) have
identical timestamps. This means my queries sometimes return one row,
sometimes the other depending on the particular query characteristics.
I don't think there's much I can do about this. Given that it is a very
small set of changes, it is probably something we can live with and fix
on a case by case basis as problems are picked up.
Congratulations if you've made it this far :-) Next up I'll produce
some timings for creating daily planets and then I'll start looking into
the huge planet in more detail. But for now, bed time.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 21157 bytes
Desc: not available
More information about the dev