[osmosis-dev] osmosis performance optimization

[Aurélien FILEZ]

Hi,

I saw the new (for me) "read-pbf-fast" option.

Also, I checked the documentation of the --filter-tag option.

As I know strictely what I need, I first began using the options :
--tag-filter reject-nodes \
--tag-filter reject-ways \
--tag-filter reject-relations \
And then, put all my "accept-nodes / ways / relations" with the desired
tags I need.

But it produced a file of 98 bytes.. bad !

So I followed the example in the documentation to produce 3 pipes (1 for
nodes, 1 for ways with the used-node option, 1 for relations with the
used-node & used-ways option) and finishing by merging them.

It produce these command :

./osmosis/bin/osmosis \
--read-pbf-fast file=europe-latest.osm.pbf workers=4 --b
bufferCapacity=500000 --log-progress \
--bounding-box top=51.28 left=-5.48 bottom=41.25 right=9.88
completeWays=yes completeRelations=yes \
--tag-filter reject-nodes \
--tag-filter accept-nodes
place=country,state,region,county,city,town,village,hamlet,suburb,locality \
--tag-filter accept-nodes
addr=addr:housenumber,addr:housename,addr:street,addr:postcode,addr:city,addr:country
\
--tag-filter accept-nodes highway=motorway_junction,turning_circle,bus_stop
\
--tag-filter accept-nodes
railway=station,halt,tram_stop,crossing,level_crossing,subway_entrance \
--tag-filter accept-nodes
amenity=university,school,library,fuel,hospital,fire_station,police,townhall
\
outPipe.0=nodes \
\
--read-pbf-fast file=europe-latest.osm.pbf workers=4 --b
bufferCapacity=500000 --log-progress \
--bounding-box top=51.28 left=-5.48 bottom=41.25 right=9.88
completeWays=yes completeRelations=yes \
--tag-filter reject-ways \
--tag-filter accept-ways
addr=addr:housenumber,addr:housename,addr:street,addr:postcode,addr:city \
--tag-filter accept-ways
highway=motorway,motorway_link,trunk,trunk_link,primary,primary_link,secondary,secondary_link,tertiary,road,path,track,service,footway,bridleway,cycleway,steps,pedestrian,living_street,unclassified,residential
\
--tag-filter accept-ways
railway=rail,tram,light_rail,subway,narrow_gauge,preserved,funicular,monorail
\
--tag-filter accept-ways waterways=stream,river,canal,drain \
--tag-filter accept-ways aeroways=runway,taxiway \
--used-node \
outPipe.0=ways \
\
--read-pbf-fast file=europe-latest.osm.pbf workers=4 --b
bufferCapacity=500000 --log-progress \
--bounding-box top=51.28 left=-5.48 bottom=41.25 right=9.88
completeWays=yes completeRelations=yes \
--tag-filter reject-relations \
--tag-filter accept-relations admin_level=* \
--tag-filter accept-relations admin=boundary \
--tag-filter accept-relations buildings=* \
--tag-filter accept-relations waterway=riverbank \
--tag-filter accept-relations natural=water \
--tag-filter accept-relations landuse=basin,reservoir \
--tag-filter accept-relations railway=station \
--tag-filter accept-relations aeroway=aerodome,terminal,helipad,apron \
--tag-filter accept-relations
landuse=park,forest,residential,retail,commercial,industrial,railway,cemetery,grass,farmyard,farm,farmland,wood,meadow,village_green,recreation_ground,allotments,quarry
\
--tag-filter accept-relations
leisure=park,garden,playground,golf_course,sports_centre,pitch,stadium,common,nature_reserve
\
--tag-filter accept-relations natural=wood,land,scrub \
--tag-filter accept-relations highway=pedestrian,footway \
--tag-filter accept-relations
amenity=university,school,college,library,fuel,parking,cinema,theatre,place_of_worship,hospital
\
--used-node \
--used-way \
outPipe.0=relations \
\
--merge inPipe.0=nodes inPipe.1=ways outPipe.0=intermediate \
--merge inPipe.0=intermediate inPipe.1=relations \
--write-pbf file=france2-bbox.osm.pbf omitmetadata=true

Am I in a good way ?

Thank you very much, and have a nice weekend :)

Kin

On Sat, Jun 1, 2013 at 12:30 PM, Aurélien FILEZ <kinju59 at gmail.com> wrote:

> Hello,
>
> I have an ubuntu server installed on a computer which have an i7 processor
> (4 physical CPU, 8 logical) with 8Gb of memory.
>
> My objective is to extract a country data with completeWays &
> completeRelations = true, to have a better cutting than what we can have
> with the geofabrik cutting. For the moment, I can make it, but it's long,
> very very long (9/10 days of processing for the bounding-polygon
> operation). I'm sure I can improve this treatment by a better osmosis
> usage, but I don't understand stories about pipes, buffer etc.
>
> Here is how I actually process (for example, for the France country) :
> - Download european set (10.2 Go)
> - Apply a "bounding-bbox" of the country (2.7 Go)
> - Apply a "bounding-polygon" of the country (1.5 Go)
>
> The commands provides from a basic usage of osmosis :
> --read-pbf file=xxx.pbf --bounding + options --write-pbf yyy.pbf
>
> I first remark that with htop, I have 8 processors, but only one used.
> Maybe we can exploit the all computer performance by a "spacial command" ?
>
> Also, I don't need all osm data, and I know exactly which entity / tags I
> need, for example : Ways (tunnel, bridge, oneway, ref, z_order), Points
> (addr:housenumber, addr:housename, addr:street, addr:postcode, addr:city),
> Relations (admin_level), etc.
>
> Can you, please, help me to find the better steps / commands to make this
> job in the spirit of "what I need, how I need, where I need, when I need,
> not too much, not too less" ?
>
> Thank you for your help, very much, and sorry for my bad english (I will
> improve it ;)
>
> See you later,
>
> Kin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20130601/9adb0396/attachment.html>