[OSM-dev] Hourly Diffs
80n80n at gmail.com
Fri Nov 30 17:14:15 GMT 2007
It's a non-issue now - I've adapted to use the gz format.
On Nov 30, 2007 5:01 PM, Karl Newman <siliconfiend at gmail.com> wrote:
> On Nov 29, 2007 5:05 PM, Brett Henderson <brett at bretth.com> wrote:
> > Sorry, I think I've run out of time, I'm hoping to get out of here in a
> > few hours and I still have a fair bit to get done. I can fix it up when
> > I get back in two weeks although that is probably too late.
> > The reason I'm using gz is because I'm streaming the data direct from db
> > to file and the java bz2 implementation I'm using isn't very fast. The
> > daily scripts are using the command line bzip2 running at nice 10 after
> > the database read has completed. I suspect the java gz implementation
> > uses native code, it's very fast (hourly extract takes approx 30 seconds
> > clock time). I was trying to avoid holding locks on the main db for too
> > long.
> If the Java bz2 is slow, what about the possibility of writing the XML
> to stdout and then piping into bzip2? I'm haven't looked into how
> Java's stdout handling works; I'm guessing it's not going to be as
> easy as "--write-xml file=-" but it might not be too difficult.
> dev mailing list
> dev at openstreetmap.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the dev