Skip to content

Commit

Permalink
feat!: introduce epochs (#1660)
Browse files Browse the repository at this point in the history
* cleanup ./changelog entries

* rebase

* fix!: Validation of SlashAcks fails due to marshaling to Bech32  (backport #1570) (#1577)

fix!: Validation of SlashAcks fails due to marshaling to Bech32  (#1570)

* add different Bech32Prefix for consumer and provider

* separate app encoding and params

* remove ConsumerValPubKey from ValidatorConfig

* update addresses in tests

* make SlashAcks consistent across chains

* add comments for clarity

* Regenerate traces

* Fix argument order

* set bech32prefix for provider to cosmos

* add changelog entries

* add consumer-double-downtime e2e test

* update nightly-e2e workflow

* fix typo

* add consumer-double-downtime to testConfigs

* remove changes on provider

* skip invalid SlashAcks

* seal the config

* clear the outstanding downtime flag for new vals

* add info on upgrading to v4.0.0

* fix upgrade handler

* fix changeover e2e test

* Update tests/e2e/config.go

Co-authored-by: Philip Offtermatt <57488781+p-offtermatt@users.noreply.github.com>

* Update tests/e2e/config.go

Co-authored-by: Philip Offtermatt <57488781+p-offtermatt@users.noreply.github.com>

* add AccountPrefix to ChainConfig

* fix docstrings

* update AccountAddressPrefix in app.go

* fix consumer-misb e2e test

---------

Co-authored-by: Philip Offtermatt <p.offtermatt@gmail.com>
Co-authored-by: Simon Noetzlin <simon.ntz@gmail.com>
Co-authored-by: Philip Offtermatt <57488781+p-offtermatt@users.noreply.github.com>
(cherry picked from commit 8604692)

Co-authored-by: Marius Poke <marius.poke@posteo.de>

* docs: update changelog for v4.0.0 (#1578)

update changelog

* docs: prepare for v4.0.0 (#1581)

* unclog build

* update release notes

* update release date

* added proto declaration

* temp commit

* temp commit

* more changes

* first commit

* add param and fix tests

* reduce epoch size for e2e

* clean up

* mbt fix

* fix diff bug

* cleaning up

* cleaning up

* cleaning up

* cleaning up

* cleaning up

* cleaning up

* added more tests

* more fixes

* nit fixes

* cleaning up

* increase downtime by one block

* fix logs

* took into account Marius' comments

* tiny fixes

* Update x/ccv/provider/keeper/params.go

Co-authored-by: Simon Noetzlin <simon.ntz@gmail.com>

* use Bech32 addresses as keys for maps

* refactor nextBlocks(epoch) to nextEpoch

* fixed comment

* Remove new block creation during consumer chain setup

* Revert "Remove new block creation during consumer chain setup"

This reverts commit 85a52b7.

* added simple param test

* added upper bound and addressed a comment

* Add another edge case for diffing

* used smarted solution (based on Philip's comment) for diffing validators

* refactor!: remove key-assignment replacements (#1672)

* initial commit

* removed KeyAssignmentReplacementsKey

* refactor: simplify key-assignment logic (#1684)

* fixed typo: depreciated to deprecated

---------

Co-authored-by: Marius Poke <marius.poke@posteo.de>

* add the epoch param in the docs

---------

Co-authored-by: mpoke <marius.poke@posteo.de>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Simon Noetzlin <simon.ntz@gmail.com>
Co-authored-by: Philip Offtermatt <p.offtermatt@gmail.com>
  • Loading branch information
5 people committed Mar 11, 2024
1 parent 157efcb commit 6c0549c
Show file tree
Hide file tree
Showing 35 changed files with 1,313 additions and 628 deletions.
116 changes: 17 additions & 99 deletions docs/docs/adrs/adr-001-key-assignment.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ title: Key Assignment

## Changelog
* 2022-12-01: Initial Draft
* 2024-03-01: Updated to take into account they key-assigment-replacement deprecation.

## Status

Expand All @@ -30,10 +31,6 @@ ConsumerValidatorsBytePrefix | len(chainID) | chainID | providerConsAddress -> c
```golang
ValidatorsByConsumerAddrBytePrefix | len(chainID) | chainID | consumerConsAddress -> providerConsAddress
```
- `KeyAssignmentReplacements` - Stores the key assignments that need to be replaced in the current block. Needed to apply the key assignments received in a block to the validator updates sent to the consumer chains.
```golang
KeyAssignmentReplacementsBytePrefix | len(chainID) | chainID | providerConsAddress -> abci.ValidatorUpdate{PubKey: oldConsumerKey, Power: currentPower},
```
- `ConsumerAddrsToPrune` - Stores the mapping from VSC ids to consumer validators addresses. Needed for pruning `ValidatorByConsumerAddr`.
```golang
ConsumerAddrsToPruneBytePrefix | len(chainID) | chainID | vscID -> []consumerConsAddresses
Expand Down Expand Up @@ -67,20 +64,6 @@ if _, consumerRegistered := GetConsumerClientId(chainID); consumerRegistered {
oldConsumerAddr := utils.TMCryptoPublicKeyToConsAddr(oldConsumerKey)
vscID := GetValidatorSetUpdateId()
AppendConsumerAddrsToPrune(chainID, vscID, oldConsumerAddr)
} else {
// the validator had no key assigned on this consumer chain
oldConsumerKey := validator.TmConsPublicKey()
}

// check whether the validator is valid, i.e., its power is positive
if currentPower := stakingKeeper.GetLastValidatorPower(providerAddr); currentPower > 0 {
// to enable multiple calls of AssignConsumerKey in the same block by the same validator
// the key assignment replacement should not be overwritten
if _, found := GetKeyAssignmentReplacement(chainID, providerConsAddr); !found {
// store old key and power for modifying the valset update in EndBlock
oldKeyAssignment := abci.ValidatorUpdate{PubKey: oldConsumerKey, Power: currentPower}
SetKeyAssignmentReplacement(chainID, providerConsAddr, oldKeyAssignment)
}
}
} else {
// if the consumer chain is not registered, then remove the previous reverse mapping
Expand Down Expand Up @@ -129,89 +112,24 @@ func (k Keeper) MakeConsumerGenesis(chainID string) (gen consumertypes.GenesisSt
}
```

On `EndBlock` while queueing `VSCPacket`s to send to registered consumer chains:
Note that key assignment works hand-in-hand with [epochs](https://github.com/cosmos/interchain-security/blob/main/docs/docs/adrs/adr-014-epochs.md).
For each consumer chain, we store the consumer validator set that is currently (i.e., in this epoch) validating the consumer chain.
Specifically, for each validator in the set we store among others, the public key that it is using on the consumer chain during the current (i.e., ongoing) epoch.
At the end of every epoch, if there were validator set changes on the provider, then for every consumer chain, we construct a `VSCPacket`
with all the validator updates and add it to the list of `PendingVSCPacket`s. We compute the validator updates needed by a consumer chain by
comparing the stored list of consumer validators with the current bonded validators on the provider, with something similar to this:
```golang
func QueueVSCPackets() {
valUpdateID := GetValidatorSetUpdateId()
// get the validator updates from the staking module
valUpdates := stakingKeeper.GetValidatorUpdates()

IterateConsumerChains(func(chainID, clientID string) (stop bool) {
// apply the key assignment to the validator updates
valUpdates := ApplyKeyAssignmentToValUpdates(chainID, valUpdates)
// ..
})
// ...
}

func ApplyKeyAssignmentToValUpdates(
chainID string,
valUpdates []abci.ValidatorUpdate,
) (newUpdates []abci.ValidatorUpdate) {
for _, valUpdate := range valUpdates {
providerAddr := utils.TMCryptoPublicKeyToConsAddr(valUpdate.PubKey)

// if a key assignment replacement is found, then
// remove the valupdate with the old consumer key
// and create two new valupdates
prevConsumerKey, _, found := GetKeyAssignmentReplacement(chainID, providerAddr)
if found {
// set the old consumer key's power to 0
newUpdates = append(newUpdates, abci.ValidatorUpdate{
PubKey: prevConsumerKey,
Power: 0,
})
// set the new consumer key's power to the power in the update
newConsumerKey := GetValidatorConsumerPubKey(chainID, providerAddr)
newUpdates = append(newUpdates, abci.ValidatorUpdate{
PubKey: newConsumerKey,
Power: valUpdate.Power,
})
// delete key assignment replacement
DeleteKeyAssignmentReplacement(chainID, providerAddr)
} else {
// there is no key assignment replacement;
// check if the validator's key is assigned
consumerKey, found := k.GetValidatorConsumerPubKey(ctx, chainID, providerAddr)
if found {
// replace the update containing the provider key
// with an update containing the consumer key
newUpdates = append(newUpdates, abci.ValidatorUpdate{
PubKey: consumerKey,
Power: valUpdate.Power,
})
} else {
// keep the same update
newUpdates = append(newUpdates, valUpdate)
}
}
}

// iterate over the remaining key assignment replacements
IterateKeyAssignmentReplacements(chainID, func(
pAddr sdk.ConsAddress,
prevCKey tmprotocrypto.PublicKey,
power int64,
) (stop bool) {
// set the old consumer key's power to 0
newUpdates = append(newUpdates, abci.ValidatorUpdate{
PubKey: prevCKey,
Power: 0,
})
// set the new consumer key's power to the power in key assignment replacement
newConsumerKey := GetValidatorConsumerPubKey(chainID, pAddr)
newUpdates = append(newUpdates, abci.ValidatorUpdate{
PubKey: newConsumerKey,
Power: power,
})
return false
})

// remove all the key assignment replacements

return newUpdates
}
// get the valset that has been validating the consumer chain during this epoch
currentValidators := GetConsumerValSet(consumerChain)
// generate the validator updates needed to be sent through a `VSCPacket` by comparing the current validators
// in the epoch with the latest bonded validators
valUpdates := DiffValidators(currentValidators, stakingmodule.GetBondedValidators())
// update the current validators set for the upcoming epoch to be the latest bonded validators instead
SetConsumerValSet(stakingmodule.GetBondedValidators())
```
where `DiffValidators` internally checks if the consumer public key for a validator has changed since the last
epoch and if so generates a validator update. This way, a validator can change its consumer public key for a consumer
chain an arbitrary amount of times and only the last set consumer public key would be taken into account.

On receiving a `SlashPacket` from a consumer chain with id `chainID` for a infraction of a validator `data.Validator`:
```golang
Expand Down
7 changes: 7 additions & 0 deletions docs/docs/introduction/params.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,10 @@ This param would allow provider binaries to panic deterministically in the event

`RetryDelayPeriod` exists on the consumer for **ICS versions >= v3.2.0** (introduced by the implementation of [ADR-008](../adrs/adr-008-throttle-retries.md)) and is the period at which the consumer retries to send a `SlashPacket` that was rejected by the provider.


## Epoch Parameters

### BlocksPerEpoch
`BlocksPerEpoch` exists on the provider for **ICS versions >= 3.3.0** (introduced by the implementation of [ADR-014](../adrs/adr-014-epochs.md))
and corresponds to the number of blocks that constitute an epoch. This param is set to 600 by default and cannot exceed 1200.
Assuming we need 6 seconds per block, the default value corresponds to 1 hour and the maximum to 2 hours.
14 changes: 14 additions & 0 deletions proto/interchain_security/ccv/provider/v1/provider.proto
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,9 @@ message Params {
// The fee required to be paid to add a reward denom
cosmos.base.v1beta1.Coin consumer_reward_denom_registration_fee = 9
[ (gogoproto.nullable) = false ];

// The number of blocks that comprise an epoch.
int64 blocks_per_epoch = 10;
}

// SlashAcks contains cons addresses of consumer chain validators
Expand Down Expand Up @@ -295,3 +298,14 @@ message ConsumerAddrsToPrune {
uint64 vsc_id = 2;
AddressList consumer_addrs = 3;
}

// ConsumerValidator is used to facilitate epoch-based transitions. It contains relevant info for
// a validator that is expected to validate on a consumer chain during an epoch.
message ConsumerValidator {
// validator's consensus address on the provider chain
bytes provider_cons_addr = 1;
// voting power the validator has during this epoch
int64 power = 2;
// public key the validator uses on the consumer chain during this epoch
tendermint.crypto.PublicKey consumer_public_key = 3;
}
14 changes: 13 additions & 1 deletion tests/e2e/actions.go
Original file line number Diff line number Diff line change
Expand Up @@ -1431,6 +1431,12 @@ func (tr TestConfig) relayPacketsGorelayer(
target ExecutionTarget,
verbose bool,
) {
// Because `.app_state.provider.params.blocks_per_epoch` is set to 3 in the E2E tests, we wait 3 blocks
// before relaying the packets to guarantee that at least one epoch passes and hence any `VSCPacket`s get
// queued and are subsequently relayed.
tr.waitBlocks(action.ChainA, 3, 90*time.Second)
tr.waitBlocks(action.ChainB, 3, 90*time.Second)

pathName := tr.GetPathNameForGorelayer(action.ChainA, action.ChainB)

// rly transact relay-packets [path-name] --channel [channel-id]
Expand All @@ -1455,6 +1461,12 @@ func (tr TestConfig) relayPacketsHermes(
target ExecutionTarget,
verbose bool,
) {
// Because `.app_state.provider.params.blocks_per_epoch` is set to 3 in the E2E tests, we wait 3 blocks
// before relaying the packets to guarantee that at least one epoch passes and hence any `VSCPacket`s get
// queued and are subsequently relayed.
tr.waitBlocks(action.ChainA, 3, 90*time.Second)
tr.waitBlocks(action.ChainB, 3, 90*time.Second)

// hermes clear packets ibc0 transfer channel-13
cmd := target.ExecCommand("hermes", "clear", "packets",
"--chain", string(tr.chainConfigs[action.ChainA].ChainId),
Expand Down Expand Up @@ -1763,7 +1775,7 @@ func (tr TestConfig) invokeDowntimeSlash(action DowntimeSlashAction, target Exec
// Bring validator down
tr.setValidatorDowntime(action.Chain, action.Validator, true, target, verbose)
// Wait appropriate amount of blocks for validator to be slashed
tr.waitBlocks(action.Chain, 10, 3*time.Minute)
tr.waitBlocks(action.Chain, 11, 3*time.Minute)
// Bring validator back up
tr.setValidatorDowntime(action.Chain, action.Validator, false, target, verbose)
}
Expand Down
33 changes: 20 additions & 13 deletions tests/e2e/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,8 @@ func SlashThrottleTestConfig() TestConfig {
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
".app_state.provider.params.slash_meter_replenish_fraction = \"0.10\" | " +
".app_state.provider.params.slash_meter_replenish_period = \"20s\"",
".app_state.provider.params.slash_meter_replenish_period = \"20s\" | " +
".app_state.provider.params.blocks_per_epoch = 3",
},
ChainID("consu"): {
ChainId: ChainID("consu"),
Expand All @@ -384,7 +385,7 @@ func SlashThrottleTestConfig() TestConfig {
IpPrefix: "7.7.8",
VotingWaitTime: 20,
GenesisChanges: ".app_state.gov.params.voting_period = \"20s\" | " +
".app_state.slashing.params.signed_blocks_window = \"15\" | " +
".app_state.slashing.params.signed_blocks_window = \"20\" | " +
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
Expand Down Expand Up @@ -522,7 +523,8 @@ func DefaultTestConfig() TestConfig {
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
".app_state.provider.params.slash_meter_replenish_fraction = \"1.0\" | " + // This disables slash packet throttling
".app_state.provider.params.slash_meter_replenish_period = \"3s\"",
".app_state.provider.params.slash_meter_replenish_period = \"3s\" | " +
".app_state.provider.params.blocks_per_epoch = 3",
},
ChainID("consu"): {
ChainId: ChainID("consu"),
Expand All @@ -531,7 +533,7 @@ func DefaultTestConfig() TestConfig {
IpPrefix: "7.7.8",
VotingWaitTime: 20,
GenesisChanges: ".app_state.gov.params.voting_period = \"20s\" | " +
".app_state.slashing.params.signed_blocks_window = \"15\" | " +
".app_state.slashing.params.signed_blocks_window = \"20\" | " +
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\"",
Expand All @@ -551,7 +553,8 @@ func DemocracyTestConfig(allowReward bool) TestConfig {
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
".app_state.transfer.params.send_enabled = false"
".app_state.transfer.params.send_enabled = false | " +
".app_state.provider.params.blocks_per_epoch = 3"

name := string(DemocracyTestCfg)
if allowReward {
Expand Down Expand Up @@ -583,7 +586,8 @@ func DemocracyTestConfig(allowReward bool) TestConfig {
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
".app_state.provider.params.slash_meter_replenish_fraction = \"1.0\"", // This disables slash packet throttling
".app_state.provider.params.slash_meter_replenish_fraction = \"1.0\" | " + // This disables slash packet throttling
".app_state.provider.params.blocks_per_epoch = 3",
},
ChainID("democ"): {
ChainId: ChainID("democ"),
Expand Down Expand Up @@ -625,7 +629,8 @@ func MultiConsumerTestConfig() TestConfig {
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
".app_state.provider.params.slash_meter_replenish_fraction = \"1.0\"", // This disables slash packet throttling
".app_state.provider.params.slash_meter_replenish_fraction = \"1.0\" | " + // This disables slash packet throttling
".app_state.provider.params.blocks_per_epoch = 3",
},
ChainID("consu"): {
ChainId: ChainID("consu"),
Expand All @@ -634,7 +639,7 @@ func MultiConsumerTestConfig() TestConfig {
IpPrefix: "7.7.8",
VotingWaitTime: 20,
GenesisChanges: ".app_state.gov.params.voting_period = \"20s\" | " +
".app_state.slashing.params.signed_blocks_window = \"10\" | " +
".app_state.slashing.params.signed_blocks_window = \"20\" | " +
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\"",
Expand All @@ -646,7 +651,7 @@ func MultiConsumerTestConfig() TestConfig {
IpPrefix: "7.7.9",
VotingWaitTime: 20,
GenesisChanges: ".app_state.gov.params.voting_period = \"20s\" | " +
".app_state.slashing.params.signed_blocks_window = \"10\" | " +
".app_state.slashing.params.signed_blocks_window = \"20\" | " +
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\"",
Expand Down Expand Up @@ -684,7 +689,8 @@ func ChangeoverTestConfig() TestConfig {
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
".app_state.provider.params.slash_meter_replenish_fraction = \"1.0\" | " + // This disables slash packet throttling
".app_state.provider.params.slash_meter_replenish_period = \"3s\"",
".app_state.provider.params.slash_meter_replenish_period = \"3s\" | " +
".app_state.provider.params.blocks_per_epoch = 3",
},
ChainID("sover"): {
ChainId: ChainID("sover"),
Expand All @@ -694,7 +700,7 @@ func ChangeoverTestConfig() TestConfig {
IpPrefix: "7.7.8",
VotingWaitTime: 20,
GenesisChanges: ".app_state.gov.params.voting_period = \"20s\" | " +
".app_state.slashing.params.signed_blocks_window = \"15\" | " +
".app_state.slashing.params.signed_blocks_window = \"20\" | " +
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
Expand Down Expand Up @@ -784,7 +790,8 @@ func ConsumerMisbehaviourTestConfig() TestConfig {
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\" | " +
".app_state.provider.params.slash_meter_replenish_fraction = \"1.0\" | " + // This disables slash packet throttling
".app_state.provider.params.slash_meter_replenish_period = \"3s\"",
".app_state.provider.params.slash_meter_replenish_period = \"3s\" | " +
".app_state.provider.params.blocks_per_epoch = 3",
},
ChainID("consu"): {
ChainId: ChainID("consu"),
Expand All @@ -793,7 +800,7 @@ func ConsumerMisbehaviourTestConfig() TestConfig {
IpPrefix: "7.7.8",
VotingWaitTime: 20,
GenesisChanges: ".app_state.gov.params.voting_period = \"20s\" | " +
".app_state.slashing.params.signed_blocks_window = \"15\" | " +
".app_state.slashing.params.signed_blocks_window = \"20\" | " +
".app_state.slashing.params.min_signed_per_window = \"0.500000000000000000\" | " +
".app_state.slashing.params.downtime_jail_duration = \"60s\" | " +
".app_state.slashing.params.slash_fraction_downtime = \"0.010000000000000000\"",
Expand Down
11 changes: 10 additions & 1 deletion tests/integration/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ func delegateAndRedelegate(s *CCVTestSuite, delAddr sdk.AccAddress,
srcValTokensAfter := s.getVal(s.providerCtx(), srcValAddr).GetBondedTokens()
s.Require().Equal(srcValTokensAfter.Sub(srcValTokensBefore), amount)

s.providerChain.NextBlock()
s.nextEpoch()

dstValTokensBefore := s.getVal(s.providerCtx(), dstValAddr).GetBondedTokens()

Expand Down Expand Up @@ -625,3 +625,12 @@ func (s *CCVTestSuite) mustGetStakingValFromTmVal(tmVal tmtypes.Validator) (stak
s.Require().True(found)
return stakingVal
}

// nextEpoch moves `chain` forward by an epoch
func (s *CCVTestSuite) nextEpoch() {
blocksPerEpoch := s.providerApp.GetProviderKeeper().GetParams(s.providerCtx()).BlocksPerEpoch

for i := int64(0); i < blocksPerEpoch; i++ {
s.providerChain.NextBlock()
}
}
Loading

0 comments on commit 6c0549c

Please sign in to comment.