[osmosis-dev] Continuous Integration

[Brett Henderson]

Jochen Topf wrote:
> On Fri, Mar 20, 2009 at 09:09:49AM +1100, Brett Henderson wrote:
>   
>> It's not ideal running it on my home server because I don't want to give 
>> out access to it.  If anybody has other suggestions let me know.  The 
>> osm dev server may be more appropriate but I need a tomcat server to run 
>> hudson under and I'm not sure how easy that will be to organise.
>>     
>
> Depending on what exactly you need we could also maybe run this on the
> server where openstreetmap.de is running on or use the server that the
> wikipedia people are buying for experimental OSM stuff.
>   
Cool.  Using one of those servers sounds great if they're available.
> How much ressources would you need? Tomcat and the Osmosis dev stuff ?
> I guess the whole thing could be swapped out most of the time and only
> needs a few resources for the actual build every few hours? Do several
> people need access? root access?
>   
It is running on a virtual machine here which is allocated a total of 
512MB RAM and a single CPU (E6600 2.4GHz Core 2 Duo).  It doesn't stress 
that machine very much and it runs a number of other services.

It needs the following applications to be running at all times:
Tomcat
PostgreSQL
MySQL

The two databases are tiny because they only hold data required for 
functional unit tests.  They have no more than 10-20 entities in them at 
a time and get cleared out before each test run.  If the two db servers 
are already running then the overhead of Osmosis tests will be 
negligible.  If the db servers need to be installed then they can be 
configured to use minimal RAM due to the limited amount of data being used.

Tomcat is the heaviest piece.  I've had a look at my install to see how 
much RAM it is consuming.  Being java it is hard to get exact numbers 
and I'm not sure how to get the jconsole tool to connect under a 
different user (Tomcat is running under a tomcat user) to examine it 
properly.  Using top, at first check Tomcat had 88MB resident in RAM.  
Using the memhog tool to consume RAM I managed to get all but 5MB to 
swap out.  On browsing the hudson dashboard page it climbed to just over 
40MB resident in RAM.  After performing a full build it was consuming 
around 88MB RAM again.  If necessary Hudson can be run standalone as a 
java app instead of running within Tomcat, that would possibly bring 
resource usage down further.

As for access, there are two types of access that may be required.
1. Console access to log into the server.  This is probably only 
necessary for upgrading db schemas, and longer term the build should do 
that itself.  The number of people needing this should be very small (2 
or 3 perhaps).
2. Access to Hudson.  Hudson allows fairly fine-grained security but it 
should be sufficient to give Hudson admin access for a small number of 
users who configure build jobs, and allow anonymous users to trigger 
builds.  If anonymous builds aren't appropriate we can create a larger 
group of users allowing builds to be triggered.

The steps to install are something like this:
* Install Tomcat and connect via the AJP connector to the Apache server 
if there is one.
* Install PostgreSQL, create an osm6_test postgis schema, and install 
the osmosis pgsql schema with all optional features.
* Install MySQL, create an osm6_test schema, and install the latest 0.6 
mysql osm schema.
* Copy the hudson.war file into the tomcat webapps directory.
* Browse to http://hudsonurl, configure a new Osmosis build job, enable 
security, create users to administer the osmosis job, and allow 
anonymous users to trigger builds.
* Via a shell, navigate to the hudson managed osmosis checkout directory 
and modify the unit test db authorisation files so the tests can connect 
to the database instances.

Jochen, is this the type of info you need?  Let me know if I can do 
anything to help.

Brett