[OSM-dev] Anyone with a speedy gazetteer

[Sascha Silbe]

On Mon, Jan 12, 2009 at 10:57:24AM +0100, Erik Johansson wrote:

> I have two questions; why is gazetteer.openstreetmap.org so slow  at
> 20-50 seconds per request,
I remember David Earl mentioning a scheduled database rebuild. Perhaps 
its currently running. Otherwise, I'd really like to know the reasons 
(so I can test my own implementation accordingly).

> and if anyone has code for faster variants of name finders? Speed is 
> essential..
I have done an alternative, but suspended its development because
a) the current Namefinder was sufficiently fast when I tried to compare 
them => don't know under which conditions it's slow, so I cannot test 
whether my implementation really is faster in these cases; and
b) because I haven't had enough time to finish it.

The core (written in C/C++) is almost finished (only some minor changes 
for name canonicalization needed). What's really missing is the 
"front-end" part, i.e. the web server and search plan generation. A 
Python module implementing the protocol between Core and Frontend is 
included, so someone else could start implementing it right away. :)


To give some funny, unscientific numbers:

General conditions: planet-080813.osm.gz, Athlon 64 BE-2300 dual-core 
1.9Ghz, 4GB DDR2-800, Limit 100 ways + 100 nodes per step

Data base size: ~8GB
import time: 206 minutes
startup time: 41 seconds
RAM usage after startup (-> names and hash tables): 501MB

name search ("Provenceweg", exact):            0.001117s
loc search (48.51 9.07 48.52 9.08):            0.061777s
exact name + loc:                              0.001331s
loc + exact name:                              0.005612s
name search ("Provenceweg", regex):            2.582227s
loc + regex:                                   0.232248s
exact name + loc (whole world):                2.263841s
loc (whole world):                             0.069157s
Name search ("Provenceweg", substring):        0.600431s


Unfair comparison with current namefinder (my code: local, no frontend; 
gazetteer: remote; with caching, i.e. first result is discarded):

					old		new
Beethovenweg				0.681s		0.008970s
random string (=> 0 results)   		0.310s		0.000352s



I've released my code under GPLv2 in my arch repository [1] as 
osmsearch--devel--0.1 (haven't had an idea for a better name yet, 
sorry).
You'll need some data files [2-4] as well.

Instructions for fetching the source (assuming the GNU arch client "tla" 
is installed):

1. tla register-archive 
http://sascha.silbe.org/arch/sascha-arch@silbe.org--2008
2. tla get sascha-arch at silbe.org--2008/osmsearch--devel--0.1 
osmsearch--devel--0.1

This will put the source into a newly-created directory called 
"osmsearch--devel--0.1".


[1] http://sascha.silbe.org/arch/sascha-arch@silbe.org--2008
[2] http://sascha.silbe.org/tmp/osmsearch.conf
[3] http://sascha.silbe.org/tmp/nodesDisplay.rules
[4] http://sascha.silbe.org/tmp/waysDisplay.rules

CU Sascha

-- 
http://sascha.silbe.org/
http://www.infra-silbe.de/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20090112/3ef322ed/attachment.pgp>