[OSM-dev] How do I build the latest db schema?

[Brett Henderson]

After my previous email, I kicked off an import to my local database.  
Before I added date handling it used to complete in 70'ish minutes.  
Since then the dataset appears to have grown considerably so I had 
expected it to take somewhat longer.

It crashed after 4 hours with an out of memory error.  This is 
surprising because it was supposed to be processing in a streamy 
fashion.  I made some changes to the database layer so I've probably 
forgotten to release something.  I changed my handling of prepared 
statements and suspect that may be the problem.  Coincidentally it 
failed after it had completed all nodes, all segments and had only 
processed a small number of way records (40000), this may be related.

Looks like I have some work to do before a load will work properly.

Brett Henderson wrote:
> It is able to populate a MySQL database from scratch.  A Postgres DB 
> would require the creation of a new writer task.  To improve speed, I'm 
> using MySQL specific things like multi-record inserts.  Postgres will 
> require smart use of transactions to obtain good speed which aren't 
> being used currently.  If the app was using small datasets I would have 
> written a generic task to handle any database engine but I've tried to 
> wring as much speed out of it as I can.
>
> All versions are available at:
> http://www.bretth.com/osmosis/
> The latest is:
> http://www.bretth.com/osmosis/osmosis-latest.zip
> which is currently:
> http://www.bretth.com/osmosis/osmosis-0.1.8.zip
>
> I'll release v0.2 when I finish my mysql change set writer.  The reader 
> half is complete but untested.
>
> I haven't tested this bit of functionality for a little while but I've 
> just kicked it off and it appears to be working okay.  It's slower than 
> it used to be because it's now performing full date parsing (using 
> example code from JOSM), date parsing is incredibly CPU intensive but it 
> seems to be the Java SimpleDateFormat that is the problem, not the hacks 
> that are required to support all the possible OSM date formats.
>
> If you're running on Linux and you install the app using the 
> instructions in the zip, the command line for loading a planet file is:
> osmosis --read-xml file="planet-xxx.osm" --write-mysql host="myhost" 
> database="osm" user="osm" password="xxx"
>
> If you're running on Windows, you'll need to run the full java command 
> which can be extracted from the osmosis script in the bin directory.  I 
> need to figure out how to write an equivalent batch file for Windows but 
> will need to find some examples (ant maybe?).
>
> More instructions are available here:
> http://www.bretth.com/wiki/Wiki.jsp?page=OpenStreetMap
>
> Some caveats:
> Recently I've learnt more about the OSM schema and I'm not loading the 
> data correctly, you will need to correct a few things after load.  For 
> example, it's not populating the current tables so a separate query will 
> need to be run to copy this data from the history tables.  It won't be 
> hard to fix the code, but it takes time unfortunately.
>
> Frederik Ramm wrote:
>   
>> Hi Brett,
>>
>>     
>>> Is there a script somewhere for building the current database schema?
>>>       
>> [...]
>>
>> I'd like to set up a "clone" of the production MySQL database 
>> (possibly trying to make that a Postgres instead of MySQL but using 
>> the same structure). Is "Osmosis" already in a state where I could 
>> feed it a planet file and ask it to populate a database?
>>
>> Bye
>> Frederik
>>
>>     
>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>