[OSM-dev] Effort in the US wasted until TIGER import is complete?

Don Smith dcsmith at gmail.com
Fri Mar 23 03:53:03 GMT 2007


Thomas,
do you have a machine setup? I'll look at the code tonight, but you  
seem to have a better grasp of the operations going on. Any ideas in  
what specifically needs to be done?

Right now, the two tasks appear to be aggregation of related segments  
into ways (?) and the 1 sec insert cycle. I would suggest instead, if  
possible that we load the data into mysql, and then when it's ready,  
batch import the sql into the master db(does this make sense? Instead  
of running the script twice, run it once, dump the db, and then  
import the dump?).

Don Smith

On Mar 22, 2007, at 3:23 PM, SteveC wrote:

> Don, Thomas,
>
> Please run with this, don't wait for permission :-)
>
> Thomas Lunde wrote:
>> It was previously written (but I can't find by who in my email  
>> chain):
>>>> The original approach on one machine at 1 sec insert cycle produces
>>>> a huge
>>>> amount of data quite quickly (it by far swamped the volume of data
>>>> for the
>>>> rest of the world put together in the time it ran) it will still
>>>> take a
>>>> considerable time to import everything, many many months rather
>>>> than weeks.
>>
>> Oy!  There must be a way to avoid months of waiting.
>>
>>
>> On 3/21/07, Frederik Ramm <frederik at remote.org> wrote:
>>> Would it not be sensible - for this special case, where such a large
>>> amount of data is imported - to import directly into the central
>>> MySQL database? Of course this would require some admin cooperation
>>> and oversight but for an amont of data that will (my guess)  
>>> instantly
>>> double what we already have, one shouldn't do anything without admin
>>> cooperation or oversight anyway;-)
>>
>>
>> If an environment is a problem, I can provide one for this testing
>> purpose.  I'm thinking of a dump of the current MySQL database which
>> is then imported into a new database on a server that is neither
>> production nor the main dev environment, if there are concerns on the
>> impact of doing so there.  Obviously, if this is the sort of thing
>> that can be done in dev, so much the better.
>>
>>
>> I don't speak Ruby either, but I've heard so much about its
>> readability that perhaps the current scripts can be used as
>> "inspiration" for a import script that is more widely grokked.  (I do
>> _not_ want to turn this into a language war, and I'm well aware of  
>> the
>> reaction to an outsider who wanders into a project and just starts
>> firing off "Why don't you use language X?" questions.  I'm merely
>> trying to help in any way that I can.  Again:  I'm not trying to  
>> start
>> a debate on the use of Ruby.)
>>
>>
>> In addition to kick-starting the TIGER project again, I am hopeful
>> that someone can answer my original question:
>>
>> Would pre-TIGER-import mapping efforts in the US be wasted/blown-away
>> by the eventual import?
>>
>> Thanks
>> Thomas
>>
>> _______________________________________________
>> dev mailing list
>> dev at openstreetmap.org
>> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>>
>
> -- 
> have fun,
>
> SteveC steve at asklater.com http://www.asklater.com/steve/
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev





More information about the dev mailing list