Height Coordinated Upgrade - Jan 2024

This update includes the following changes.

👨‍💻 Improvements for developers

The last release added the basic support for script execution on the access node. This update builds on the previous work and includes several improvements and bug fixes to get script execution on the access node close to completion.

🎵 Getting ready for Flow EVM

Flow EVM is coming soon (see: https://flow.com/upgrade/crescendo). To unlock integration with existing middleware, this update includes the Application Binary Interface (ABI) Encode and Decode methods for Flow EVM (issue).

Candence updates

The upgrade will update the supported Cadence version to v0.42.9.

🦾 Performance improvements

This upgrade includes the new cryptography module, improving the overall performance of the consensus algorithm.

Change to resource identifier generation

This release introduces an implementation change for Cadence resource identifier generation. The resource identifiers will no longer be generated sequentially and can have values greater than the max value for Uint32 and, for the foreseeable future, less than 2^56 -1 (the MAX_SAFE_INTEGER for JavaScript). If you were previously storing resource IDs in a Unit32 data type, please make sure to update it to a larger data type that can, at the very least, accommodate IDs greater than Uint32 and up to 2^56 -1 but preferably up to Max Uint64.

Commit list

Access

• Make script exec configurable by @peterargue in #5037
• Script execution coded errors by @sideninja in #4895
• Allow get blocks script calls by @sideninja in #4894
• Improve logging and validation in local script exec by @peterargue in #4920
• Add metrics for script exec failure from missing data by @peterargue in #4907
• Log script exec mismatches as error by @peterargue in #5194
• Handle script canceled and timeout errors by @peterargue in #5036
• Improve script exec compare logging by @peterargue in #4936
• Cleanup script execution comparisons by @peterargue in #4956
• Get account bugfix with tests by @sideninja in #4862
• Add wait in integration tests for index to be synced by @sideninja in #4902
• Validate addresses match network in rest api by @peterargue in #4930
• Use local event for AccessAPI get events endpoints by @peterargue in #4851
• Use local event for event streaming API by @peterargue in #5160
• Index collections from execution data by @peterargue in #5163
• Add endpoints to Execution nodes to support getting Transaction Result error messages by @AndriiDiachuk in #5042
• Cache Transaction Result error messages by @Guitarheroua in #5066
• create get register async API by @koko1123 in #4871
• Make RegisterID request size configurable by @koko1123 in #5067
• Fix GetRegisterValues input types by @peterargue in #5138
• Update websockets event streaming to return JSON-CDC encoded events by @peterargue in #5048
• Allow all origins by default on websockets connections by @peterargue in #4954
• Circuit breaker too restrictive by @Guitarheroua in #5010
• gRPC Circuit breaker causes panic by @Guitarheroua in #5012
• Get Block endpoint is missing the system collection by @AndriiDiachuk in #5049
• Handle inconsistent protocol state in access rpc engine by @UlyanaAndrukhiv in #5052
• Get protocol snapshot by block id and block height by @AndriiDiachuk in #4957
• Optimize header ID calculation in get events by @peterargue in #5214
• add config to limit script execution range - v0.33 by @peterargue in #5282

BFT

• ReceiptValidator ensures k receipts committing to the execution result by @durkmurder in #5050
• Epoch setup random source update by @tarakby in #5088

EVM on Flow

• beyond EVM - benchmarking performance part 1 by @ramtinms in #4879
• update EVM integration test to use FVM by @ramtinms in #4878
• EVM: Refactor stdlib to use new approach by @turbolent in #4876
• minor fixes to new EVM contract branch by @ramtinms in #4903
• beyond EVM part 1 - adding types for the EVM by @ramtinms in #4859
• beyond EVM part 2.1 - adding EVM emulator database by @ramtinms in #4925
• beyond EVM part 3 - updating environment for direct event emission, and checking metering capacity by @ramtinms in #4861
• beyond EVM part 2.2 - adding EVM emulator by @ramtinms in #4927
• beyond EVM part 4 - adding EVM handler by @ramtinms in #4864
• beyond EVM part 5 - integrate EVM into FVM by @ramtinms in #4866
• beyond EVM part 6.1 - Implement EVMAddress.deposit by @turbolent in #4889
• beyond EVM part 6.2 - Implement EVM.BridgedAccount.withdraw by @turbolent in #4890
• beyond EVM part 6.3 - Implement EVM.BridgedAccount.deploy by @turbolent in #4897
• beyond EVM part 6.4 - Fix EVM contract tests by @turbolent in #4941
• beyond EVM part 6.5: Improve and test deposit and withdrawal by @turbolent in #4986
• beyond EVM part 7 - adding more evm-based contract types for testing by @ramtinms in #4872
• performant StateDB implementation - part 1 by @ramtinms in #5166
• performant StateDB implementation - part 2 by @ramtinms in #5167
• performant StateDB implementation - part 3 by @ramtinms in #5168
• performant StateDB implementation - part 4 by @ramtinms in #5169
• Updating Geth version by @ramtinms in #5196
• Updateing StateDB createAccount behaviour to match Geth's latest version by @ramtinms in #5212
• Resolve ambiguous import by @sideninja in #5006
• Fix the setup process by @sideninja in #5069
• Emit events as Cadence events by @sideninja in #5090
• Benchmark state transitions by @sideninja in #5032
• Bridged account balance function by @sideninja in #5098
• Handle EVM errors by @sideninja in #5216
• Add EVM transactions to FVM benchmark tests by @janezpodhostnik in #5061
• Setup EVM account Flow vault during bootstrap by @janezpodhostnik in #5070
• Add EVM methods for encodeABI & decodeABI by @m-Peter in #5024
• Report proper computation for EVM.encodeABI & EVM.decodeABI methods by @m-Peter in #5146
• Add the EVM.encodeABIWithSignature and EVM.decodeABIWithSignature functions by @m-Peter in #5189
• Update EVM test by @janezpodhostnik in #5215
• Add storage limit check exception for EVM address by @janezpodhostnik in #5106
• Add feature flag for deploying EVM contract only with ABI functionality - Port by @janezpodhostnik in #5264
• Set the correct testnet/mainnet evm state storage account - port by @janezpodhostnik in #5311

CI

• Update workflows to use setup-go@v4 by @peterargue in #4482
• Add BuildJet CI runners for unit tests, integration tests; cache Docker images by @gomisha in #4906
• Add timeout to setup go step by @peterargue in #5082
• Cleanup integration make targets by @peterargue in #5141
• Use build tag for without-netgo by @peterargue in #5157
• Modifies block rate in VN test to address sealing lagging finalization by @jordanschalm in #4975
• Update BN2 configs by @jordanschalm in #5162
• Update Makefile to include changes to upload and download the values.yml file generated by @sjonpaulbrown in #5177
• Move stale issue bot workflow to worflow dir by @justinbarry in #4116
• Reconcile build automation by @sjonpaulbrown in #5259
• Fix Dockerfile by @sjonpaulbrown in #5261
• Revert change to use semver build for without-netgo images - v0.33 by @peterargue in #5266

Caching

• Adds support for atomic get-with-init and adjust-with-init on backend mempools by @yhassanzadeh13 in #5206

Consensus and Collection

• Make QC Voter more resiliant to access node instability by @peterargue in #4924
• Remove random delays from DKG controller by @tarakby in #4913
• Missing consensus telemetry events from FinalizationConsumer and CommunicatorConsumer by @AlexHentschel in #5020
• Fix epoch event docs by @jordanschalm in #5035

Execution

• Return OutOfRange instead of Internal when account block is not cached by @peterargue in #4917
• Refactor Checker Engine by @zhangchiqing in #5184
• Add transaction ms per computation metrics by @janezpodhostnik in #4615
• UUID partition change - port by @janezpodhostnik in #5287
• Handle cadence ParentErrors - v0.33 by @peterargue in #5317

Cadence

• Refactor event emission code by @janezpodhostnik in #4982
• Add Change Contract Code Migration by @janezpodhostnik in #5191
• Migrate-by-account changes by @janezpodhostnik in #5128
• Deduplicate contract names migration by @janezpodhostnik in #5143
• Migrate payloads by reference by @janezpodhostnik in #5123
• Unify core contracts address definitions by @janezpodhostnik in #5033
• bugfix bench test random by @sideninja in #4833
• Update cadence to v0.42.4 by @SupunS in #4974
• Update cadence to v0.42.5 by @SupunS in #4990
• Update cadence to v0.42.6 by @SupunS in #5093
• Update Cadence to v0.42.7 by @dsainati1 in #5205
• Update to Cadence v0.42.9 by @turbolent in #5360

Ledger

• Add special handling for global register keys by @peterargue in #4942
• Refactor NewRegisterID to accept address by @peterargue in #5144

Networking

• GossipSub Publish Message Validation (Unknown Topic Spam) by @kc1116 in #4720
• Refine command line options for configuring libp2p resource manager constraints by @yhassanzadeh13 in #4929
• Update RPC inspector error log by @kc1116 in #4968
• GossipSub-level Sybil Mitigation by @kc1116 in #4773
• Improves p2p node logging by @yhassanzadeh13 in #4898
• Fixing enforced dial conflict between unicast manager and swarm by @yhassanzadeh13 in #4951
• Optimizing GossipSub RPC Handling Memory Usage Through Asynchronous Subscription Updates by @yhassanzadeh13 in #4988
• Backports PR 4971 by @yhassanzadeh13 in #4972
• Adds Test To Verify Independent RPC ID Computation Method by @yhassanzadeh13 in #5025
• GossipSub Spam Mitigation- Dynamic Decay Speed by @kc1116 in #4891
• Fixing flakey TestGossipSubSpamMitigationIntegration by @yhassanzadeh13 in #5095
• Restores skipped TestGossipSubSpamMitigationIntegration by @yhassanzadeh13 in #5112
• Caching application specific score of GossipSub by @yhassanzadeh13 in #5045
• Stream protection and connection manager configuration by @yhassanzadeh13 in #5115
• Fixing flakey broken promises tests by @yhassanzadeh13 in #5145
• Extending GossipSub Router Tracer Telemetry by @yhassanzadeh13 in #5131
• Chore: fixes package names by @yhassanzadeh13 in #5154
• Revising RPC inspector metrics by @yhassanzadeh13 in #5172
• Configurable GossipSub Peer Scoring Parameters by @kc1116 in #5210
• Implements a silence period for GossipSub peer scoring. by @kc1116 in #5084
• Remove t.Parallel from GossipsubScoring tests by @kc1116 in #4841
• Add leniency to control message misbehaviors on cluster prefixed topics by @kc1116 in #4857
• Ports in 5234 and 5265 on v0.33 by @yhassanzadeh13 in #5274

Node operation improvement

• Make path configs to be usable by container by default by @zhangchiqing in #4885

Observer

• Improve upstream gRPC failovers with AN connection pooling by @UlyanaAndrukhiv in #5011

Storehouse

• Add storehouse interfaces by @zhangchiqing in #4835
• add PayloadToRegister by @zhangchiqing in #4955
• Storehouse in memory regsiter store by @zhangchiqing in #4836
• Add Finalized Reader by @zhangchiqing in #4977
• Implement Register store by @zhangchiqing in #4940
• Unfinalized blocks loader by @zhangchiqing in #5028
• Add Extending block snapshot by @zhangchiqing in #5008
• Update committer by @zhangchiqing in #5029
• Add CreateStorageSnapshot to Execution state by @zhangchiqing in #5031
• Blockend Snapshot by @zhangchiqing in #4985
• Execution State - Update IsBlockExecuted by @zhangchiqing in #5044
• Optimization: Reduce finalized block call by @zhangchiqing in #5053
• Adding flag to enable storehouse by @zhangchiqing in #5054
• Optimize finalized reader to cache last finalized height by @zhangchiqing in #5056
• Fix storage snapshot by @zhangchiqing in #5107
• Fix getting highest executed block ID when storehouse is enabled by @zhangchiqing in #5109
• Add tests for storehouse-backed execution state by @zhangchiqing in #5086
• Improve InMemoryRegisterStore's IsBlockExecuted method by @zhangchiqing in #5125
• Improving logging by @zhangchiqing in #5124
• Use unfinalized loader when storehouse is enabled by @zhangchiqing in #5151
• Add metrics for register store by @zhangchiqing in #5170
• Optimize finalized reader to use block id index by @zhangchiqing in #5175
• Fix dynamic bootstrapping for storehouse by @zhangchiqing in #5165
• make worker count as option for indexing checkpoint by @zhangchiqing in #4884

Testing

• Fix emulator version by @peterargue in #5077

Util

• Update go exp package and SortFunc usage by @tarakby in #5155
• make chunk-data-pack-dir optional for non-execution node by @zhangchiqing in #4896
• Change migration progress log to be more verbose by @janezpodhostnik in #5122

Cryptography

• new BLST-based crypto module:
  • add blst source code by @tarakby in #3851
  • new BLST-based scalar and field element type by @tarakby in #3993
  • E2 BLST-based type by @tarakby in #4230
  • merge master to BLST feature branch by @tarakby in #4264
  • use deterministic randomness in C backend by @tarakby in #4253
  • add BLS cross tests against Go-BLST by @tarakby in #4328
  • BLST-based E1 and G1 types by @tarakby in #4355
  • BLST pairing by @tarakby in #4377
  • unplug relic from the crypto package and the rest of flow-go by @tarakby in #4380
  • multiple improvements of the code base by @tarakby in #4397
  • enable ADX support by default by @tarakby in #4405
  • C formatting and DKG optimization by @tarakby in #4645
  • C code sanitization by @tarakby in #4654
  • misc updates by @tarakby in #4681
  • merge master branch by @tarakby in #4687
  • update BLST version by @tarakby in #4693
  • minor clean up by @tarakby in #4740
  • Merge master into BLST feature branch by @jordanschalm in #4863
  • Code review changes and fix access integration test by @tarakby in #4817
  • Slow down block production in integration tests by @tarakby in #5111
  • BLST-based crypto package by @tarakby in #4358
  • Clean up Flow crypto module import by @tarakby in #5100
• Deprecate flow-go/crypto by @tarakby in #5119
• Revert deleted crypto/hash and crypto/random packages by @tarakby in #5239
• improve shuffle test by @tarakby in #4844
• Backport PR 5254 by @tarakby in #5359

Docs

• Update Responsible Disclosure Link by @jribbink in #4622

New Contributors

• @jribbink made their first contribution in #4622

Full Changelog: v0.32.11-verification-disalbe-invalid-message-delivery-penalty...v0.33.1