[OSM-dev] Bin-test failed

Scott Crosby scrosby at cs.rice.edu
Tue Sep 28 20:09:16 BST 2010


On Sat, Sep 25, 2010 at 2:51 AM, Christian H. Bruhn <brogo at arcor.de> wrote:

> Friday, September 24, 2010, 6:06:59 PM, Scott Crosby wrote:
>
> > Is there an error message thrown? If so, please paste it.
>
> No error message, everything seems to be OK.
>
> > If the error message is something about a possibly corrupt file..... I'm
> > aware of the issue. To detect bad files and fail gracefully, I put in
> some
> > size limits that I later found might be too small. The patch with the
> > increased limits is waiting for me to push out a new jar.
>
> I've just tested it with the planet-100922.osm.bz2 (MD5 checked) and
> 20100922-20100923.osc.gz. All osmosis tasks run with the -v option
> under Windows 7 Prof. 64 Bit with osmosis-snapshot 0.37 from
> 23.09.2010.
>

I'm not replicating the problem. I'm running under my working tree. The only
relevant differences are increased maximum blocksize limits.


Here is what I ran (under ubuntu linux. pbzip2 is a parallel bzip2
decompressor, which is then buffered with mbuffer.):

## I converted the planet twice, once for faster writing, one not.
pbzip2 -d < foo/planet-100922.osm.bz2.1 |
~/source/Map2/osmosis/package/bin/osmosis --read-xml file=- --lp --b
bufferCapacity=12000 --write-bin file=planet.osm.pbf compress=none
pbzip2 -d < foo/planet-100922.osm.bz2.1 |
~/source/Map2/osmosis/package/bin/osmosis --read-xml file=- --lp --b
bufferCapacity=16000 --write-bin file=planet-default.osm.pbf

## Resulting sizes are as expected

## I then tried merging changesets.
~/source/Map2/osmosis/package/bin/osmosis --read-xml-change
file=20100922-20100923.osc.gz --read-bin file=planet.osm.pbf --b
bufferCapacity=12000 --apply-change --lp --b bufferCapacity=12000
--write-bin file=out.osm.pbf
~/source/Map2/osmosis/package/bin/osmosis --read-xml-change
file=20100922-20100923.osc.gz --read-bin file=planet-default.osm.pbf --b
bufferCapacity=12000 --apply-change --lp --b bufferCapacity=12000
--write-bin file=out-default.osm.pbf
pbzip2 -d <foo/planet-100922.osm.bz2.1 |
~/source/Map2/osmosis/package/bin/osmosis  --read-xml-change
file=20100922-20100923.osc.gz --read-xml file=-  --lp  --b
bufferCapacity=12000  --apply-change --lp --b bufferCapacity=12000
--write-bin file=planet-xmlin.osm.pbf


## All three run to completion, returning identical output files of about
the expected sizes:
 8835376 -rw-r--r-- 1 scrosby scrosby  9047420819 Sep 28 11:00
out-default.osm.pbf
 8835376 -rw-r--r-- 1 scrosby scrosby  9047420819 Sep 28 10:14 out.osm.pbf
 8835376 -rw-r--r-- 1 scrosby scrosby  9047420819 Sep 28 12:06
planet-xmlin.osm.pbf
 8824112 -rw-r--r-- 1 scrosby scrosby  9035884739 Sep 28 10:37
planet-default.osm.pbf
18576056 -rw-r--r-- 1 scrosby scrosby 19021875191 Sep 28 10:12
planet.osm.pbf

Is your problem repeatable?

If it is repeatable, one explanation is that a 'too big' block was
generated, and somehow the warning was missed. That can be tested by running
a smaller batchlimit. Can you try a '--write-bin file=FOO batchlimit=2000'
on the initial conversion from *.osm to *.osm.pbf and the later
--read-xml-change tests and see if the problem goes away? If so, please
report the result to help me fine-tune the block size and batch limit
settings.


Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100928/0e8462d9/attachment.html>


More information about the dev mailing list