Background job processor

[swistak35]

There is recurring question of whether or not we should have our own background processor, provided as an option.
I will cover few options, with my opinion on them, I would gladly accept additional input.

1. Build on existing RDBMS background job solution. Build it on top of good_job, que, or whatever is now most popular with rails.

• RES customers have some tool about they don't have to think too much (when someone just has to start with asynchronous processing, these tools are not really different since processing power is not critical in the beginning)
• RDBMS solutions avoid having to use outbox pattern, are consistent with event publishing etc.
  Cons:
• If we already are doing it then in-house solutions may have few more benefits which may be critical
• performance of RDBMS has it's limits

2. Build our own, RDBMS solution.
   Pros:

• all from (1)
• outbox is already about 60% of solution (with the locks, correctly handling split keys and everything). One would need to add spawning multiple threads, but that work would probably have to be done anyway for the projector. Big part of the code could be shared
  Cons:
• maintenance. I guess? On the other hand I don't see much maintenance which has to be done with outbox, for example.

3. Build our own, Redis-based solution. I don't see much of a profit here. Sidekiq is there, it's stable and supported. The only benefit I see is providing pro-level reliability in our version.

• we would have to build it from scratch
• there is already a stable solution with which we just integrate

5. Build our own, X-based solution. I don't really know what's there. For a customer which does not have any solution, something else than redis may be preferable if it would provide better performance. That would need probably quite a research about performance of our options, if we would go this route.

• would need quite a research to find something more performant than redis