[OSM-dev] AND upload almost ready

[Marc Kessels]

On Fri, 2007-08-24 at 21:37 +0100, Jon Burgess wrote:
> On Fri, 2007-08-24 at 20:11 +0100, Tom Hughes wrote:
> > In message <46CF2B55.3050306 at rjb.za.net>
> >           Ray Booysen <raybooysen at rjb.za.net> wrote:
> > 
> > > Tom Hughes wrote:
> > >
> > > > Not really - to start with I don't think even Brett is claiming it's
> > > > ready for production use yet.
> > > >
> > > > Plus I thought we'd already agreed to go through the rails object model
> > > > rather than hitting the database directly. I'd like some sort of checking
> > > > that database integrity constraints aren't being broken you see.
> > >
> > > Thats a fair point.  Everything should be going through the API not the
> > > DB directly.  Seems like too many mistakes could be made any other way.
> > 
> > The point is to not go through the API as such (too slow due to all
> > the network round trips) but to still go through the rails object
> > module by writing code that does Node.create(...) or whatever.
> > 
> 
> What about a script which could run on dev issuing requests to
> localhost. Surely that would be very fast without bypassing the API?
> 
> Unfortunately even a 'headless' JOSM would not work since it would
> require too much RAM to load the current AND.osm file. The file was
> roughly 1GB last time I ran and2osm. In my experience JOSM needs RAM of
> several times the size of the OSM file, e.g. a 150MB osm file failed to
> load into JOSM with 1GB allocated to the JVM, but worked with 2GB.
> 
> 	Jon
another way to get this solved, is by subdividing the data in several
pieces (each 10 MByte or so), and process them one-by-one. I am still
planning to improve the and2osm/2AND program, such that it is much less
memory consuming. This would also enable an output which is not
overlapping, apart from the connecting nodes. These nodescould be
flagged, and need some attention during/after the import in osm. 

just my 2 cents...
greetz,
Marc