[OSM-dev] osm2pgsql: "unexpected chunk size"

[Jon Burgess]

On Thu, 2008-06-05 at 18:20 +0200, Martijn van Oosterhout wrote:
> On Thu, Jun 5, 2008 at 4:58 PM, Frederik Ramm <frederik at remote.org> wrote:
> >> Smaller files, including the whole europe.osm, work fine. Planet
> >> gives me the said error message, and in the log file it says:
> >>
> >> ERROR: unexpected message type 0x58 during COPY from stdin
> >>
> >> I'll re-run it with some monitoring to find out if there's a resource
> >> shortage.
> >
> > No error this time. Strange. Lesson learned: If you get strange
> > Postgres errors, just retry.
> 
> FWIW, the most common cause for this error on the pgsql mailing list
> is people trying to use the same connection from different threads at
> the same time. osm2pgsql only recently started using threads, perhaps
> there is something there?

It certainly could be but each multiple thread should be acting on an
individual connection. I guess it might still want some flushing or
synchronisation to be done in the main program before launching the
threads. Or maybe some older versions of the libpq are not thread safe.

I did wonder about changing the model to move all the data offload into
different threads. The output copy buffer might be a suitable place to
move the data to the output worker threads.

For the main tile server the rendering takes far longer than the data
import so I'm not too bothered about minor speed improvements in the
import. Right now the threading adds very little extra performance and
could easily be disabled (just remove the -DHAVE_PTHREAD from the
Makefile).

	Jon