[Talk-ca] CanVec and potential 'pounding' servers

[Sam Vekemans]

Hi Talk-ca@ im just looking for feedback and some ideas :)

On Mon, Jun 14, 2010 at 7:54 AM, Tom Hughes <tom at compton.nu> wrote:
> On 14/06/10 15:47, Sam Vekemans wrote:
>
>> So my question for Tom is, what would be a 'normal volume'  that you
>> would recommend that users stick to so that it doesnt over-load the
>> servers?
>
> I have no idea what a "normal volume" is.
>
>> Do you have a list of recommendations that we can follow as a 'best
>> practice?'
>
> The normal rules apply - don't do anything that causes you to appear on my
> radar and we'll all be happy ;-)

There is 2 angles to this

1 - if there aren't that many people who actually know that the files
will be available.
ie. dont actively 'announce to the world', then there will be less
volume that way.
the way that it's listed at now... people need to ask on the talk-ca
list for instructions.   So keeping a 'low profile'  is actually a
benifit on the server side..

2 - on the NRCan server side, the rate of data being available can be
slowed to pace that makes the OSM servers happy.  Ie.  only make tiles
available in a 'stream flow' instead of Niagara falls'.  Just 1 NTS
tile at a time and take spot-requests.

And so, what would i have todo to get on this radar?  :)   How much of
a volume is 'too much'  and what exactly would qualify for 'pounding'
the servers.   How much can it currently handle?

Or maybe a better question is, how can we learn from the TIGER method,
so that we
1 - dont have people 'remotly drop in data that will need to be fixed later'
  answer:  Dont import it until a local mapper 'finds it' and copies
it in themselves.
2 - perhaps list the availability as a WMS layer  ie.  Like the
Toporama WMS, but just listing the NTS tile boxes & have them coloured
in for what the current status of the tile is.

>From Halifax:
I've started to copy in some of the Address data (remotly) but since i
am not actually local (i just plan on visiting)  it would probably be
better if i simply hold off on actually copying the local data in,
until i am actually there.
As it would require a whole lot of 'local fixing'.
So in learning from TIGER, we dont want OSM Canada to be a 'FIXME'
project, where all that is bring done is 'fixing' from an import that
was 'plopped in'.
So i would recommend that the data actually remain on the NRCan
server, until someone local is working on it.
So i ended up, only copying in the address data for the roads that i
indent on 'virtually traveling'.

So here's the plan
1 - Simply dont announce that the data is available ...
2 - Dont list instructions for people to copy in the data, (let them
ask on this talk-ca@ and IRC chat.
3 - Host good old fashioned 'mapping parties'  where local area
mappers work in .. the local NTS tile and copy the data in (in 1
meetup session)
4 - Treat this canvec.osm.zip data in the same manor that the
Austrailian Nearmap data is. ... as it's available, all people need
todo is copy it in.      Where copying it in FAST will produce errors
... Guarenteed.  So just because the data is available, having more
data in the map doesnt mean that it's a quality map.
.... and so, we just croud source it.   Once more people see that the
map is getting better, it will attract more.
Sure, it will take a few years for all the data to be imported, but
the ent result is a MUCH better map.

And for those of up who are the most ambitious. ... we just have to
keep in mind that we are responsible for our OWN edits.. which
includes copying in the data.   So if you copy in stuff that you know
needs to be fixed..  It's MUCH better to just fix it yourself, and
hold off on other mapping until it's done. :)

Ideas?  Comments?

Cheers,
Sam

>
> I would also add that you might want to consider holding off for a while as
> we're getting quite short of disk space at the moment. There will be some
> more coming on line this week probably which will give us a bit of headroom.
>
> Tom
>
> --
> Tom Hughes (tom at compton.nu)
> http://compton.nu/
>