[OSM-dev] Hourly Diffs

[80n]

It's a non-issue now - I've adapted to use the gz format.

On Nov 30, 2007 5:01 PM, Karl Newman <siliconfiend at gmail.com> wrote:

> On Nov 29, 2007 5:05 PM, Brett Henderson <brett at bretth.com> wrote:
> > Sorry, I think I've run out of time, I'm hoping to get out of here in a
> > few hours and I still have a fair bit to get done.  I can fix it up when
> > I get back in two weeks although that is probably too late.
> >
> > The reason I'm using gz is because I'm streaming the data direct from db
> > to file and the java bz2 implementation I'm using isn't very fast.  The
> > daily scripts are using the command line bzip2 running at nice 10 after
> > the database read has completed.  I suspect the java gz implementation
> > uses native code, it's very fast (hourly extract takes approx 30 seconds
> > clock time).  I was trying to avoid holding locks on the main db for too
> > long.
>
> If the Java bz2 is slow, what about the possibility of writing the XML
> to stdout and then piping into bzip2? I'm haven't looked into how
> Java's stdout handling works; I'm guessing it's not going to be as
> easy as "--write-xml file=-" but it might not be too difficult.
>
> Karl
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20071130/8999ed3c/attachment.html>