Check behavior of tile requests during database crash

[strk]

Desired behavior on database crash is whatever can result in a relatively "quick" response.

Could be an error on timeout or crash detection.
There could be some attempts at reconnecting too, maybe, but in that case there should be an upper limit on the time of attempting that.

For the reconnecting mapnik attempts,
See mapnik/mapnik#499 for the crash detection.

For a generic timeout, see #169

JIRA REF: https://cartodb.atlassian.net/browse/CDB-1935

[strk]

The test should be performed with and without pgbouncer in the middle
\cc @lbosque @zenitraM

[strk]

Killing postgresql on direct connection results in immediate response:

[2014-03-05 17:31:42.170] [INFO] console - [TILE RENDER ERROR] -- 400: Error: layer0: Postgis Plugin: 
Full sql was: 'SELECT ST_AsBinary(ST_Simplify("the_geom_webmercator", 1.91093)) AS geom FROM (select * from nyc_subway_entrance_export) as cdbq WHERE "the_geom_webmercator" && ST_SetSRID('BOX3D(-8181819.507645266 4928659.583828164,-8137791.779353004 4972687.312120426)'::box3d, 3857) LIMIT 65535'

[2014-03-05 17:31:42.170] [ERROR] [default] - undefined GET dev.localhost.lan:8181/tiles/layergroup/5d1263cee6b3d898b6d8041f091fda86:1390320700831.33/12/1212/1543.png?map_key=0d4ec64ac2b6bdb6aa7b0379368ef9a35dd8f4c6&cache_policy=persist&cache_buster=wb_strk_4573_1394037101762 400 36 ms -> application/json; charset=utf-8 (TOT:35;authorizedByAPIKey:17;authorize:6;setDBConn:7;render-png:5)

[strk]

Improvement in the mapnik error message have been submitted as a pull request:
mapnik/mapnik#2177

Also a possible segfault on crash has been submitted:
mapnik/mapnik#2176

[strk]

Next stop: test with pgbouncer

[strk]

So, what happens with pgbouncer is highly dependent on pgbouncer configuration.
With the default configuration any query sent will be in a queue until a connection can be estabilished with the backend. No timeout ever.

The default can be overridden by using the query_wait_timeout configuration setting, but done in a central place could give more problem then it solves.

For this reason we'd better implement a tiler-specific timeout for the database connection,
which is already ticketted as #169 -- closing this ticket as completed