You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Each harvester microservice should store its state and make it available to other harvesters. This way multiple harvesters can communicate with each other and work on different ranges of data. It prevents processing duplicate messages more than necessary. Such stateful information would include the current page of search results, etc.
This should be easily done using boltdb and exposed as a RPC via Kite. Should a service go offline and the same data be processed by another service (duplicate work), it should still be OK. While slightly inefficient it is "safer" and more consistent. The architecture is also setup in such a way that duplicate messages can not exist in the database.
The text was updated successfully, but these errors were encountered:
Each harvester microservice should store its state and make it available to other harvesters. This way multiple harvesters can communicate with each other and work on different ranges of data. It prevents processing duplicate messages more than necessary. Such stateful information would include the current page of search results, etc.
This should be easily done using boltdb and exposed as a RPC via Kite. Should a service go offline and the same data be processed by another service (duplicate work), it should still be OK. While slightly inefficient it is "safer" and more consistent. The architecture is also setup in such a way that duplicate messages can not exist in the database.
The text was updated successfully, but these errors were encountered: