[iot-ingestion-manager] go down due OutOfMemoryError

[fabiana001]

After one day running the Spark streaming consumer went down due the following error:
bash [Stage 5:===============================================> (10 + 2) / 12]Exception in thread "Yarn application state monitor" java.lang.OutOfMemoryError: Java heap space [Stage 5:===============================================> (10 + 2) / 12]Exception in thread "dispatcher-event-loop-2" java.lang.OutOfMemoryError: Java heap space [Stage 5:===============================================> (10 + 2) / 12]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space [Stage 5:===============================================> (10 + 2) / 12]Exception in thread "dispatcher-event-loop-1" java.lang.OutOfMemoryError: Java heap space 2017-09-24 16:32:01.064 [netty-rpc-env-timeout] ERROR o.a.s.s.c.YarnSchedulerBackend$YarnSchedulerEndpoint.logError(91) - Error requesting driver to remove executor 27 for reason Container marked as failed: container_1501855380192_0278_01_000028 on host: slave2.platform.daf.gov.it. Exit status: 1. Diagnostics: Exception from container-launch. Container id: container_1501855380192_0278_01_000028 Exit code: 1

Each time that an RDD is cached it needs to be un-persisted (i.e. rdd.unpersist())