[Tilesathome] setting up the api was: [OSM-dev] t at h API still turn off...

[Florian Lohoff]

On Mon, Sep 01, 2008 at 03:29:31PM +0200, spaetz wrote:
> Subject: Re: [Tilesathome] setting up the api was: [OSM-dev] t at h API still
>  turn	off...
> 
> Florian Lohoff wrote:
> 
> > Okay - import done and osmosis is keeping it up to date with the minute
> > diffs - currently the offset is roughly 5 minutes it seems.
> 
> > Most likely 90% of the whole rails stuff would not be needed for
> > only a read only api e.g. session tracking, logins, friends whatever.
> 
> I think ruby is not the best of all choices for this anyway as it seems
> to not be able to stream XML but actually stores everything in RAM
> first. Which is bad for complex tilesets.

You cant stream everything in case of the XML export.

1. You export all nodes in the BB you have (Need to memorize node ids)
2. Detect all ways connected to nodes you exported. (Store ways ids in
memory)
3. Export all nodes on these ways which have not been exported in 1)
4. Export all ways including their tags defined by ids in 2)
5. Find all relations your nodes or ways participate in (Store relation
ids)
6. Find all relations where relations from 5 participate in (Store
relids)
7. Export all relations with tags and members


In 7 you can choose to either store members or tags in memory you cant
get all in on sql query without exponential growth of your sql query
result.

I am done except 7 which is just 3-4 lines perl - I stripped usernames,
timestamps and visibile tag from exporting - not the database so the xml
gets a little smaller.

I'll announce a url for others to test output.

Currently i am a little unhappy about the size growth of the mysql - not
because of the data i guess but rather non optimal index and reclaimed
space handling - so probably i'd need to ditch it and reimport from time
to time. Currently its 200GB with innodb_file_per_table set. It seems it
grows ~500-1000MB a day which is monstrous.

Flo
-- 
Florian Lohoff                  flo at rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.openstreetmap.org/pipermail/tilesathome/attachments/20080902/ce5ba935/attachment.pgp>