Auth: Prune pending TLS identities

[markylaing]

As part of the TLS fine-grained authorization specification, pending TLS identities must be pruned when their associated token expires. This pull request adds logic to the autoRemoveExpiredTokens task such that the leader will retrieve all the pending TLS identities and delete those that have expired. Non-leaders will continue to cancel "certificate add" and "cluster join" operations in-memory in each member.

There was a lot of duplication of logic to identify the database leader. This PR adds a method to Daemon and State to determine if the member is a leader and refactors existing code to use that method. If the Daemon is standalone it is considered to be the leader. It is the responsibility of the caller to check (*State).ServerClustered before calling (*State).LeaderInfo when necessary.

Marked as draft until #14207 is merged.

[markylaing]

@tomponline @hamistao third time lucky, it has passed! Unfortunately this is probably bad news as it means we have a racy test that fails intermittently in CI. I'll keep looking but I can't see how my PR is affecting this.

[markylaing]

@tomponline @hamistao third time lucky, it has passed! Unfortunately this is probably bad news as it means we have a racy test that fails intermittently in CI. I'll keep looking but I can't see how my PR is affecting this.

I just realised that I posted this comment on the wrong PR. It's meant for #14324

[tomponline]

Please can you review the use of s.ServerClustered and in functions where there is also a call to s.LeaderInfo check leaderInfo.Clustered instead, that way we are keeping checks for cluster state to the leaderInfo struct. Perhaps ultimately we may unexport s.Clustered from State....

[markylaing]

Please can you review the use of s.ServerClustered and in functions where there is also a call to s.LeaderInfo check leaderInfo.Clustered instead, that way we are keeping checks for cluster state to the leaderInfo struct. Perhaps ultimately we may unexport s.Clustered from State....

I have kept these checks because checking s.Clustered is quick, whereas s.LeaderInfo may call d.gateway.LeaderAddress which may be slow. I think it's fine to keep s.Clustered because sometimes we just need to know if we're clustered, and not necessary get the leader information.

[tomponline]

Please can you review the use of s.ServerClustered and in functions where there is also a call to s.LeaderInfo check leaderInfo.Clustered instead, that way we are keeping checks for cluster state to the leaderInfo struct. Perhaps ultimately we may unexport s.Clustered from State....

I have kept these checks because checking s.Clustered is quick, whereas s.LeaderInfo may call d.gateway.LeaderAddress which may be slow. I think it's fine to keep s.Clustered because sometimes we just need to know if we're clustered, and not necessary get the leader information.

Yeah but wont we do an s.LeaderInfo call if we are clustered in those situations, and s.LeaderInfo is cheap because it tests s.ServerClustered internally first?

If we werent also calling s.LeaderInfo in the same function i would agree testing s.ServerClustered directly is fine.

[markylaing]

I think I've addressed everything. Let me know if you have any more thoughts on #14261 (comment) but otherwise ready for another round.