[Talk-ca] CanVec and potential 'pounding' servers

[Sam Vekemans]

Hi again Tom,
cc: Daniel @NRCan, talk-ca

Now that the canvec.osm files are becoming available, i'm afraid that
I (myself) wont be the one who will be 'pounding' the servers, but
rather a whole lot of happy Canadians from all over the country doing
work simultaneously.


On the one-hand, we will have newbie mappers who are all excited to
help, and jumping in and just uploading (and not thinking, or rather,
more accurately no knowing exactly what they are doing.

Yes, it's great to have more help. ... but....

My  recommendation is that people jump in the #osm-ca IRC chat so then
help is right here, however, there is no way to be able to see that
'bulk_imploppers' dont start dropping in data and making a big mess.
  (a TIGER mess)  ... but not as big.

For example, i was working on the Halifax water area and connecting
the boundary of the multi-polygon to the coastline edge by adding in
more nodes, when another mapper (more local)  was actually working on
residential area... just as i was.   But not aware of what  IRC is.
(It took me some time to actually know WHAT IRC is  and WHAT Pigin
is.)

Daniel, So is there anyway to slow down the volume of data so that
only requested areas are made available?

Tom, Basically... there is NO way of controlling how much data is
being loaded.. when the tap is running.

Like the Oil Spill in the Gulf.  If the data is being made available,
we instantly have hundreds of (boats) out there ready to help... and
doing work.   But alot of the work ends up being redundent as there
are better methods...
ie. a solution would be that  all the boats work together to put out a
GIANT wall (net) around the area, so to control the size of the flow.
working as 1 boat dropping the net & the others behind it  cleaning up
the stuff that got left over.  & re-inforcing the net with more
heavier plastic....   to put the weight down.

But .....  since there are many languages and boats and companies
involved, there is no coordination.  and the oil keeps on flowing all
over the place... and little gets picked up. and flow is left to go to
the shore. (coastline)

.... and yes, we will mess up the worlds coastline :-)  lol....  ...
unless we fix this now :)

I have a Google Docs chart available that is Open Access (listing the
tile #'s and names)  that people can write in what area they are
working on.

... and a Wiki-chart is not big enough to handle it all in real-time...

And the other idea (#OSM IRC folks)  say "dont clutter the map with
Squares' that are not part of the map".    However, having these
squares in the map... people WILL see them, and (at least ask).. then
people can use these squares as 'nets'  and write in the name of the
net & the status of it.   This way, everyone can see what everyone is
working on.


And i HIGHLY RECOMMED :)   that we use "landuse=construction" (or
whatever else you decide) when the area is active (.osm data
available) and being worked on.    Daniel can be the one to watch the
volume, (hopefully) only make the requested files available)  until
all these bugs are found.   Then  (myself or somoene else) can easily
add in the 'nets'  for the data available.      We just need KMZ files
for the quad-tree... or even just blanket the NTS area, as the entire
NTS area will be under construction.

So, (for the sake of not pounding the OSM servers) with enthusiastic
Canadians, speaking many languages & many levels of experience, all
being so 'helpful' ....   Would you permit the use of 'Nets'  (NTS
tile squares) in the OSM database, so then they can get listed as
'under construction'  as people start working in (and as the data is
available)???

We DO plan on removing these 'nets'  as soon as all the data (oil) has
been imported & all the fish (dupNodes)  saved.   ... until the next
(available .canvec.osm) data becomes available.

Cheers,
Sam



On Mon, Jun 14, 2010 at 7:54 AM, Tom Hughes <tom at compton.nu> wrote:
> On 14/06/10 15:47, Sam Vekemans wrote:
>
>> So my question for Tom is, what would be a 'normal volume'  that you
>> would recommend that users stick to so that it doesnt over-load the
>> servers?
>
> I have no idea what a "normal volume" is.
>
>> Do you have a list of recommendations that we can follow as a 'best
>> practice?'
>
> The normal rules apply - don't do anything that causes you to appear on my
> radar and we'll all be happy ;-)
>
> I would also add that you might want to consider holding off for a while as
> we're getting quite short of disk space at the moment. There will be some
> more coming on line this week probably which will give us a bit of headroom.
>
> Tom
>
> --
> Tom Hughes (tom at compton.nu)
> http://compton.nu/
>