[Tilesathome] MySQL and iowait

[Frederik Ramm]

Hi,

> The tiles at home server allows lots of people we barely know to upload images, 
> which we then display to the public.  That's fine if everyone uploads maps, 
> but could get a bit dodgy if someone "experiments" with their privileges.  

I still maintain that this is not a risk that we should consider. If 
someone desperately wants to upload kiddie porn, he can do that at 
YouTube, Flickr and millions of other places, where accounts are handed 
out automatically by email. And if someone desperately wants to hurt 
OSM, then there are numerous avenues where he can do more harm than with 
tiles at home. There are 60 people uploading data to tiles at home, and maybe 
one day there'll be 200. Honestly, let's not be paranoid!

I almost feel offended if you say you must trace uploads because we who 
freely give away our CPU time & bandwidth for tiles at home cannot be 
trusted and must be watched!

> Lots of people are talking about "batch uploads" now, where each upload 
> contains all 1376 images in one tileset, and just one piece of meta-info is 
> stored for the entire tileset.  This will require some extra code on both the 
> server and the client, but will reduce the SQL load while maintaining 
> traceability for images.  

Problems with that:

1. Layers. Does one upload have to contain all tiles for all layers, and 
how do we know which layers go up to which zoom level.

2. Will we discard uploads deemed incomplete, or just those containing 
tiles they shouldn't contain?

3. What about lowzoom uploads?

4. Upload zipfile size. Central London Level-12 tile, everything in one 
zip file - too big?

5. Performance.

I seriously suggest that we trust our users and don't bother checking 
their every step. Since you unzip files into a temp dir anyway, do a 
quick "ls -l >> /some/logilfe.log" together with the user name and keep 
these log files for a week if you must, for manual inspection in an 
abuse case - but we don't need full traceability through a data base.

> I can probably do the server side of that, but if anyone wants to create a PHP 
> function which looks in a directory full of "layer_z_x_y.png" images,  checks 
> that it's one complete tileset (and nothing outside that tileset), then 
> returns which tileset it is, that would be very helpful.

The following code can be used to check a directory for "foreign" 
elements - i.e. all files must be named layer_z_x_y.png, and if 
something with z>12 exists, the corresponding z==12 must also exist. It 
does not check for completeness. If one wanted to check for completeness 
also, this can be done by simply counting the files (if the total number 
is right and every file is contained in the top level-12 tile, then the 
tileset MUST be complete) - but I wouldn't advocate that because you 
would have to define what completeness means.

Bye
Frederik

<?
$dir = "/tmp/test";

$found = checkTileDir($dir);
if (is_array($found))
{
     echo "found the following tilesets:\n";
     foreach ($found as $tileset)
     {
         $ex = explode("/", $tileset);
         echo "Layer $ex[0], x=$ex[1], y=$ex[2]\n";
     }
}
else
{
     echo "invalid directory entries\n";
}


function checkTileDir($dirname)
{
     $got = array();
     $want = array();

     $dhandle = opendir($dirname);
     while(false !== ($file = readdir($dhandle)))
     {
         if ($file == "." || $file == "..") continue;
         if (!preg_match("/^(.*)_(\d\d?)_(\d+)_(\d+)\.png$/", $file, 
$matches))
         {
             echo "cannot parse filename $file\n";
             return null;
         }
         $layer = $matches[1];
         $zoom = $matches[2];
         $x = $matches[3];
         $y = $matches[4];

         if ($zoom<12)
         {
             # what now?
         }
         else if ($zoom == 12)
         {
             $got["$layer/$x/$y"] = 1;
         }
         else
         {
             $x = $x >> ($zoom-12);
             $y = $y >> ($zoom-12);
             $want["$layer/$x/$y"] = 1;
         }
     }
     closedir($dhandle);

     # now "want" has the names of level-12 tiles for which at least one 
high-zoomlevel
     # tile was present, and "got" has the names of level-12 tiles that 
were actually
     # there.

     foreach ($want as $wanttile => $dummy)
     {
         if (!array_key_exists($wanttile, $got))
         {
             echo "expected level-12 tile for $wanttile but didn't get 
it\n";
             return null;
         }
     }
     return array_keys($got);
}

?>


-- 
Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00.09' E008°23.33'