ingestion-sink update

changes since release 20200415 for ingestion-sink/:

• Emit json logs in ingestion-sink (#1258)
• Detect errors on bq load in ingestion-sink (#1259)
• Fail storage.create if the blob already exists (#1255)
• Add dependency on core test-jar in IDE profile (#1253)
• Remove nulls from arrays before writing to BQ (#1244)

in particular #1259 and #1255 both fix potential data-loss cases.

ingestion-sink update

changes since release 20200325 for ingestion-sink/:

• Always provide executor to CompletableFuture::*Async (#1243)
  • Fix BIG_QUERY_OUTPUT_MODE=file_loads performance issue where the only executor thread would block via Thread.sleep
• Record gcs and bq batch stats (#1240)
• Update maven profiles to fix IDE issues and skip missing submodules (#1242)
• Reduce DEFAULT_BATCH_MAX_DELAY to avoid OOM errors (#1233)
  • Reduce DEFAULT_BATCH_MAX_DELAY to 10s reverted by #1243
  • Reduced gcs and bigQueryFiles defaults for MAX_OUTSTANDING_ELEMENT_COUNT to 50,000 and MAX_OUTSTANDING_REQUEST_BYTES to 100MB
• Close GCS batches from a synchronized context (#1237)
  • Fix message loss case where BatchWrite$Batch#write could be called after BatchWrite$Batch#close
• Limit Gcs.Write to single storage.create api call (#1238)
  • Minimize storage api cost for very small batches

Prod kube live sink

For bug 1612372