melonade-process-manager

Workflow manager that implemented SAGA, written in typescript

To start

  nvm install v10.16.3
  nvm use

  npm install
  npm run start:watch

Docker .env

melonade.namespace=docker-compose
runners.max=1
kafka.conf.bootstrap.servers=localhost:29092
server.hostname=0.0.0.0
server.port=8080
server.enabled=true
topic.kafka.number_partitions=10
topic.kafka.replication_factor=1
task-definition.type=ZOOKEEPER
task-definition.zookeeper.connections=localhost:2181
workflow-definition.type=ZOOKEEPER
workflow-definition.zookeeper.connections=localhost:2181
task-instance.type=REDIS
task-instance.redis.host=localhost
task-instance.redis.port=16379
workflow-instance.type=REDIS
workflow-instance.redis.host=localhost
workflow-instance.redis.port=16379
transaction-instance.type=REDIS
transaction-instance.redis.host=localhost
transaction-instance.redis.port=16379

Recomended kafka's brokers config

offsets.topic.replication.factor=3 default.replication.factor=3 min.insync.replicas=2 offsets.retention.minutes=5000000 unclean.leader.election.enable=false auto.create.topics.enable=false

Tasks

• Workflow Definition
• Task Definition
• Workflow Instance
• Task Instance
• State Translation
• Store
• Refacetor code, move to their's domain
• Dispatcher
• Consumer
• Config
• Logger
• Use custom error
• Clean up completed workflows/tasks
• Delay dispatcher
• Cron job transaction support
• Failed workflow handling
• Timeout workflow handling
• Cancel workflow
• Event store
• Compensate workflow
• Publish event to kafka
• Update Workflow/Transaction's output
• Time keeper
• Refactor redundant action in "STATE"
• Watch for workflowDefinition changed
• Rewrite all the test XD
• Instance delete retention => clean up when transaction finished
• Graceful shutdown
• Procress system task immediately intread of dispatch to itself
• Use json schema
• Document
• Remove sub-workflow
• Test parallel task inside another parallel task
• Standardise the error, so it easilier to summary.
• List running transaction redis not work correctly (paginates)
• Pause workflow

Known issues

• parallel tasks can be empty
• Sub workflow won't get compensate => Not support sub-workflow anymore
• Task/Workflow data send as string's ISO time format instead of number
• Workflow Definition can have task that not existed
• MongoDB not fast enough for 5000 concurent (Lag about 1 min before task updated) => Added redis store
• Transaction did not cancelled if compensating
• Parent task did not failed if child failed
• Delay retry task sometime did not retry until limit
• Transaction will failed if send cancel transaction while on compensate workflow