[OSM-dev] osm2pgsql Windows build

[Kai Krueger]

On 12/13/2012 11:04 AM, Dominik Perpeet wrote:
>
>>
>> It seems like osm2pgsql for windows used to be kept at
>> http://tile.openstreetmap.org/osm2pgsql.zip and is still from 2010.
>>
>> I wonder if your version can be put up as a replacement for this at
>> the same location?
> Maybe eventually. For now I have my own server I can use.

The idea behind the same location was so that any documentation or links 
that still point to that location would automatically be up-to-date 
again. That would probably be easier than finding any and all 
documentation for windows and update them. Although perhaps it would be 
good anyway to check if the rest still works.

>
>>
>> Using MinGW or the Intel compiler seems to be one route? Coverting
>> things to C++ might be another?
>>
>> Any suggestions what the most sensible route is in general for the
>> best portability?
> I started removing the C99 code myself, but quit because it was too much
> work (and a bit error-prone). Since I don't see a good reason to use C,
> I'd choose the C++ route.

Sounds like the same reason I gave up on it too. But if it were of help, 
I could finish it off fairly quickly.

How much work would it be to move things to C++? Any disadvantages for it?

>
>>
>> The current code now has a configure check to test if fork is
>> available and if doesn't compile the multi-process code. I haven't
>> looked at the advantages and disadvantages of OpenMP, but perhaps
>> indeed that is an option. However, I suspect the big issue there are
>> the connections to postgresql, as each thread / process needs to setup
>> its own connections and transactions to postgresql.
>>
> Jochen's clang argument seems persuasive. I don't have much OSX
> developing experience, so I wouldn't know. One reason I didn't add
> Windows threaded code is that the postgresql connection stuff is a bit
> scattered. I would be willing to refactor the code into C++ classes.
> Once the connection variables are a bit more concentrated, it wouldn't
> affect fork() and other threading systems could be implemented.

I was trying to remember why I chose the fork() route, as I started off 
with threads. I think the reason was that part of the code (possibly 
text-tree.c) used global or static variables and thus wasn't thread 
safe. It was easier at the time to use fork() instead of fixing the rest 
of the code to be thread safe.

However, going the fork() route has caused issues on the unix side as 
well. Osm2pgsql uses a large amount of RAM for its node cache. Forking 
the process then results in having e.g. many processes using 16GB each. 
Theoretically that isn't a problem, as thanks to copy-on-write that 
memory is shared and therefore only allocated once. However, as the OS 
has no knowledge that that memory will never be written to and thus 
remains shared, it may deny the fork request to prevent too much over 
commit. Going back to threads would eliminate that issue.

So probably the best way forward in this case is to make the rest of the 
code properly thread safe and use threads for everything instead of fork.

>
> Once the code is refactored, there are alternatives to using openmp:
> - Qt (personal preference, but big dependence)
> - tbb (requires experience with tbb and can be a bit tricky to use
> correctly)
> - boost (don't really like using it for basic stuff like threads, also a
> big dependence)
> To my knowledge, all three have support for OSX. Qt is also nice because
> it has different sql backends.
>
> Without adding another library dependency, it should be possible to use
> threads just for Windows. While it wouldn't break code for anyone
> currently using osm2pgsql, it would lead to diverging paths in the
> codebase which are a bit more difficult to maintain. For starters, I
> think this would be the best option though - nobody likes too many
> dependencies.
>
>>
>> Yes, I think there is serious interest in this. And if reasonably
>> possible, I would like to get as many of those changes into the svn
>> master repository as possible to make it easier to compile on windows
>> in the future.
> Ok, I'll look into it. I think once I switched to the Intel Compiler I
> didn't have to change too much. The biggest hassle in Windows is
> building and linking the dependencies anyway. =)
>
> Dominik
>