[OSM-dev] Hourly Diffs

[Karl Newman]

On Nov 29, 2007 5:05 PM, Brett Henderson <brett at bretth.com> wrote:
> Sorry, I think I've run out of time, I'm hoping to get out of here in a
> few hours and I still have a fair bit to get done.  I can fix it up when
> I get back in two weeks although that is probably too late.
>
> The reason I'm using gz is because I'm streaming the data direct from db
> to file and the java bz2 implementation I'm using isn't very fast.  The
> daily scripts are using the command line bzip2 running at nice 10 after
> the database read has completed.  I suspect the java gz implementation
> uses native code, it's very fast (hourly extract takes approx 30 seconds
> clock time).  I was trying to avoid holding locks on the main db for too
> long.

If the Java bz2 is slow, what about the possibility of writing the XML
to stdout and then piping into bzip2? I'm haven't looked into how
Java's stdout handling works; I'm guessing it's not going to be as
easy as "--write-xml file=-" but it might not be too difficult.

Karl