Infinite dead loop caused by a network failure

[jopemachine]

Hello, I am suffering from an infinite dead loop caused by a network failure.

I'm wondering if anybody else has experienced a similar issue.

In the screenshot below, the commit index is 1808, and in the handle_append_entries of node 2, the logTerm of entry 1808 is 1, while the logTerm of the message is 6, resulting in match_term returning False and maybe_append also returning False, leading to the rejection of the message. At this point, since the reject_hint_index is 1807, it requests to send the entry 1807, and the leader sends entry 1807.
However, since commits are already made up to 1808, the MsgAppend for entry 1807 logs a message saying ‘got message with lower index than committed’ and once again, through the MsgAppendResponse message, notifies that it has committed up to entry 1808. Then, the leader sends the message for entry 1808 again, but similarly, due to the mismatch in logTerm, it sends entry 1807, and this process repeats.

[BusyJay]

since commits are already made up to 1808

If a log is committed, then leader must have the exact same log. But in your case leader thinks the term of 1808 is 1, and follower thinks the term is 6, there is obvious a inconsistency. It can be caused by either incorrect implementation of storage trait (especially the term method) or data loss.

[jopemachine]

Thanks for your reply!

I agree that there seems to be an issue with the term method.
In my opinion, it doesn't make sense to return the term from the hard_state in the function below.
This is because the commit in hard_state can be updated independently of the term. In the implementation of MemStorage, I found that the term from the snapshot metadata is returned, but in the LMDB storage implementation I am working on, the snapshot metadata is not updated every time the entries are committed, so this approach seems unfeasible.

Simply removing this line won't make it functional, so I'm contemplating how to fix it.
By any change, do you have any advice or suggestions on this matter?

    def term(self, index: int) -> int:
        first_index = self.first_index()
        last_index = self.last_index()
        hard_state = self.hard_state()
	
		# I think this line appears to be wrong.
        if index == hard_state.get_commit():
            return hard_state.get_term()

        if index < first_index:
            raise StoreError(CompactedError())

        if index > last_index:
            raise StoreError(UnavailableError())

        try:
            entry = self.entry(index)
        except Exception:
            raise StoreError(UnavailableError())

        return entry.get_term() if entry else 0

[BusyJay]

hard_state.term has nothing to do with hard_state.commit. hard_state.term is the latest term the peer has advanced to, hard_state.commit is the latest commit index the peer knows.

There should be only one correct term for log at 1808, you may need to find out the correct term first, and then trace if the returned value is correct after fixing the above error.

[jopemachine]

I have confirmed that by modifying the term method as below, the infinite dead loop is resolved and the logs are synchronized normally.
Thank you very much!

    def term(self, index: int) -> int:
        snapshot = self.snapshot(0, 0)
        if snapshot.get_metadata().get_index() == index:
            return snapshot.get_metadata().get_term()

        try:
            entry = self.entry(index)
        except StoreError as e:
            self.logger.error(f"Unknown error occurred while getting entry. Err: {e}")
            raise StoreError(UnavailableError())

        if not entry:
            if index < self.first_index():
                raise StoreError(CompactedError())
            else:
                raise StoreError(UnavailableError())

        return entry.get_term()