[Tile-serving] [mod_tile] Addition of Amazon S3 tile store module (#122)

[estadtherr]

I added support to mod_tile for using an Amazon S3 bucket for metatile storage, using the libs3 library to implement the S3 API communication. This allows a cluster of EC2 tile servers to share a tile store. The changes include:

 * new store_s3 module with all storage methods implemented. configuration of S3 storage is done using a connection string formatted as follows:

     ```s3://<S3 access key ID>:<S3 secret key>[@<S3 host>]/<bucket name>[/<base path>]```
 * autoconf support for detecting presence of the libs3 library
 * S3 unit tests in the gen_tile_test program

If autoconf detects the libs3 library, the S3-related tests will be included in gen_tile_test. These tests require environment variables to be set containing the S3 access credentials: ```S3_ACCESS_KEY_ID```, ```S3_SECRET_ACCESS_KEY```, and ```S3_BUCKET_NAME```. If libs3 support is compiled in, and these environment variables are not defined, gen_tile_test will have failed assertions on that section of the test. I'm certainly open to other options for behavior, but this was simplest.

As a possibly optional change, I changed the parameters to the ```daemon()``` call to allow the working directory and stdout/stderr handles of ```renderd``` to be specified in the systemd unit specification (which is launching ```renderd``` in our setup). This is certainly not directly required for libs3 support, but should cause no change in behavior in the vast majority of instances. I'm more than willing to drop this change from the PR if you're concerned about the impacts.


You can view, comment on, or merge this pull request online at:

  https://github.com/openstreetmap/mod_tile/pull/122

-- Commit Summary --

  * started working on S3 storage backend
  * added libs3 autoconf support, completed initial store_s3 implementation
  * allow case where default host name for s3 access isn't specified in connection string
  * add url-decoding to s3 connection string
  * add url detection for s3 storage
  * add new storage module to apache module build
  * fix error logging
  * added s3 testing to unit test, got s3 working with metatiles
  * add environment variable support to S3 credential configuration
  * don't chdir when forking - this allows systemd to configure the working directory
  * let systemd handle stdout/stderr too
  * clean up some uninitialized values
  * fix more uninitialized values to make valgrind happy
  * fix buffer write quantity computation
  * removed some debug to help performance
  * cleanup of logging
  * added S3 reference to README

-- File Changes --

    M Makefile.am (8)
    M configure.ac (5)
    M includes/metatile.h (29)
    M includes/store_file.h (5)
    M includes/store_file_utils.h (1)
    A includes/store_s3.h (16)
    M readme.txt (4)
    M src/daemon.c (2)
    M src/gen_tile_test.cpp (304)
    M src/metatile.cpp (1)
    M src/store.c (6)
    M src/store_file.c (9)
    A src/store_s3.c (551)

-- Patch Links --

https://github.com/openstreetmap/mod_tile/pull/122.patch
https://github.com/openstreetmap/mod_tile/pull/122.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/mod_tile/pull/122
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20160119/456b692b/attachment.html>