Fresh base op-reth sync crashing (critical error)

[CRossel87a]

Describe the bug

With --engine.experimental:
2024-10-22T05:53:42.266572Z INFO Block added to canonical chain number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 peers=4 txs=17 gas=3.75 Mgas gas_throughput=277.54 Mgas/second full=12.5% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=13.514576ms
2024-10-22T05:53:42.271360Z INFO Block added to canonical chain number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f peers=4 txs=7 gas=1.30 Mgas gas_throughput=273.92 Mgas/second full=4.3% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=4.761636ms
2024-10-22T05:53:42.271549Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=103.456µs
2024-10-22T05:53:52.216913Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9:
add_branch key Nibbles("0c05") self.key Nibbles("0d0f")

Without:
2024-10-22T06:00:40.674219Z WARN Failed to insert downloaded block err=Failed to insert block (hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f, number=1985833, parent_hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370): block number is lower than the last finalized block number #1985833
2024-10-22T06:00:40.674275Z WARN Marking block as invalid invalid_number=1985833 invalid_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f err=block number is lower than the last finalized block number #1985833
2024-10-22T06:00:40.674292Z WARN Bad block with hash hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f header=Header { parent_hash: 0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xf654e2cbaaf85bff1751711de985dc2a5a080892cc284d6b1fa5af86a383b480, transactions_root: 0x7949f1cd25f597978932d4d16b3f6e5ecca994e679668c8a4762685dc4232357, receipts_root: 0x53d508521b7f5be34fe673aed3c7d59087e1a2e746fef41050a37a074830acb1, withdrawals_root: None, logs_bloom: 0x00201000000002000000000080000000000008100008000000840008000000800000000020000040000400100000000000000000000000000000000001200000800000240000400000000118000000200000000000402100000000008010000008400000080440080000000000000040010000000a0004100000201000600000000208000000000000120200002800000002120108200000000000400800800002004002004400040c00000000200000010080000800000001080208a00000008000000200000100000000040000080002002080005002000000000201000000001000000000002000000000010411020000000000000040000800000c000000, difficulty: 0, number: 1985833, gas_limit: 30000000, gas_used: 1304308, timestamp: 1690761013, mix_hash: 0x48b5afb9cedd5b522582acf10dff992885b8ac17b6cc56e8ff90f7c9166a5acd, nonce: 0x0000000000000000, base_fee_per_gas: Some(77), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, requests_hash: None, extra_data: 0x }

Steps to reproduce

Normal op-reth run with --full

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

reth-optimism-cli Version: 1.1.0-dev
Commit SHA: f25cceb
Build Timestamp: 2024-10-21T12:39:05.541893344Z
Build Features: asm_keccak,jemalloc,optimism
Build Profile: maxperf

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

Base

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[CRossel87a]

Attempt fix: op-reth stage unwind --datadir /root/base --chain base num-blocks 100

Result:

2024-10-22T06:06:40.547935Z INFO Canonical chain committed number=1985829 hash=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7 elapsed=13.281116ms
2024-10-22T06:06:40.549489Z INFO Block is already canonical, ignoring. block_hash=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7
2024-10-22T06:06:40.549585Z INFO Canonical chain committed number=1985829 hash=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7 elapsed=62.124µs
2024-10-22T06:06:40.549711Z INFO New payload job created id=0x5016d6a1de2cbfce parent=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7
2024-10-22T06:06:40.563416Z INFO Block added to canonical chain number=1985830 hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 peers=2 txs=10 gas=2.11 Mgas gas_throughput=365.67 Mgas/second full=7.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=5.757786ms
2024-10-22T06:06:40.573192Z INFO Canonical chain committed number=1985830 hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 elapsed=9.222479ms
2024-10-22T06:06:40.574463Z INFO Block is already canonical, ignoring. block_hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401
2024-10-22T06:06:40.574524Z INFO Canonical chain committed number=1985830 hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 elapsed=55.134µs
2024-10-22T06:06:40.574635Z INFO New payload job created id=0xdfc5b8e7260edf01 parent=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401
2024-10-22T06:06:40.586913Z INFO Block added to canonical chain number=1985831 hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a peers=2 txs=10 gas=1.20 Mgas gas_throughput=257.46 Mgas/second full=4.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=4.65812ms
2024-10-22T06:06:40.597081Z INFO Canonical chain committed number=1985831 hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a elapsed=9.78652ms
2024-10-22T06:06:40.598732Z INFO Block is already canonical, ignoring. block_hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a
2024-10-22T06:06:40.598806Z INFO Canonical chain committed number=1985831 hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a elapsed=69.675µs
2024-10-22T06:06:40.598943Z INFO New payload job created id=0x13ed0e7d144b3e80 parent=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a
2024-10-22T06:06:40.598984Z INFO Pruner started tip_block_number=1985831
2024-10-22T06:06:40.599151Z INFO Pruner finished tip_block_number=1985831 elapsed=161.153µs stats=[]
2024-10-22T06:06:40.618532Z INFO Block added to canonical chain number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 peers=2 txs=17 gas=3.75 Mgas gas_throughput=477.89 Mgas/second full=12.5% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=7.848649ms
2024-10-22T06:06:40.636331Z INFO Canonical chain committed number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 elapsed=17.335485ms
2024-10-22T06:06:40.638037Z INFO Block is already canonical, ignoring. block_hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370
2024-10-22T06:06:40.638131Z INFO Canonical chain committed number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 elapsed=69.125µs
2024-10-22T06:06:40.638253Z INFO New payload job created id=0x07579ffccfc6a862 parent=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370
2024-10-22T06:06:40.651047Z INFO Block added to canonical chain number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f peers=2 txs=7 gas=1.30 Mgas gas_throughput=264.75 Mgas/second full=4.3% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=4.926644ms
2024-10-22T06:06:40.662012Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=10.589528ms
2024-10-22T06:06:40.663565Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
2024-10-22T06:06:40.663634Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=59.729µs
2024-10-22T06:06:40.663727Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9:
add_branch key Nibbles("0c05") self.key Nibbles("0d0f")
stack backtrace:
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
2024-10-22T06:06:43.532761Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
2024-10-22T06:06:43.532926Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=145.469µs
2024-10-22T06:06:43.533065Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9:
add_branch key Nibbles("0c05") self.key Nibbles("0d0f")
stack backtrace:
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
2024-10-22T06:06:47.786829Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
2024-10-22T06:06:47.786988Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=139.05µs
2024-10-22T06:06:47.787170Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9:
add_branch key Nibbles("0c05") self.key Nibbles("0d0f")
stack backtrace:
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
2024-10-22T06:06:57.785464Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
2024-10-22T06:06:57.785617Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=161.273µs
2024-10-22T06:06:57.785790Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9:
add_branch key Nibbles("0c05") self.key Nibbles("0d0f")
stack backtrace:
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

[CRossel87a]

Trying new fresh run without -full but with engine experimental

[CRossel87a]

Also fails in archive mode same errors

[mattsse]

thanks for flagging, we're currently investigating but at least for us this turned out hard to repro

[CRossel87a]

Which op-node commit are you using?

[mattsse]

also main(ish)

[CRossel87a]

Getting same error with fresh archive run (ensured pruning configs at NULL) and up to date op-node.

[mattsse]

on the same block or different one?

[CRossel87a]

on the same block or different one?

Not sure. But I just tried something different. I restored Base from Publicnode's achieve snapshot and it was performing quite nicely for a day, until:

2024-10-26T14:17:22.359595Z INFO Canonical chain committed number=21581447 hash=0x4965d55f2c917db95bc11c0affb2c47d198de7b215b668abe14b22c7636c5722 elapsed=560.775µs
thread 'reth-rayon-11' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:114:9:
add_leaf key Nibbles("05030e020a090203020406070d07010f09080f0108070f03010808010c0007000403020d00030a07030901030d0c04010f09090701030f0703060a0a0a060503") self.key Nibbles("060e010504000107010b060c000c0906000b07010a070002000d090f06000007070f060a0f0903010a080b0b0f0509000d0a000202030d0a0c0f07050c070a0f")

[luono]

same issue
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:114:9:
add_leaf key Nibbles("0503010f070f040c09000e000f0c0006010b0501050f0f040d030d01010b01050b090d050a000b06010d070209000f0e030d0b030002030f08090b080e000b04") self.key Nibbles("060e0d030001090b04070605060f0d02020a0907040406080f0f0505050a03060c0a0f0205010d09080b05000d010f0702050107040b0908070e08060701040c")
stack backtrace:
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
2024-10-26T15:04:51.468886Z ERROR Critical task pipeline task panicked: add_leaf key Nibbles("0503010f070f040c09000e000f0c0006010b0501050f0f040d030d01010b01050b090d050a000b06010d070209000f0e030d0b030002030f08090b080e000b04") self.key Nibbles("060e0d030001090b04070605060f0d02020a0907040406080f0f0505050a03060c0a0f0205010d09080b05000d010f0702050107040b0908070e08060701040c")
2024-10-26T15:04:51.468950Z ERROR backfill sync task dropped err=channel closed
2024-10-26T15:04:51.468982Z ERROR Fatal error in consensus engine
2024-10-26T15:04:51.469012Z ERROR shutting down due to error
Error: Critical task pipeline task panicked: add_leaf key Nibbles("0503010f070f040c09000e000f0c0006010b0501050f0f040d030d01010b01050b090d050a000b06010d070209000f0e030d0b030002030f08090b080e000b04") self.key Nibbles("060e0d030001090b04070605060f0d02020a0907040406080f0f0505050a03060c0a0f0205010d09080b05000d010f0702050107040b0908070e08060701040c")

[CRossel87a]

same issue thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:114:9: add_leaf key Nibbles("0503010f070f040c09000e000f0c0006010b0501050f0f040d030d01010b01050b090d050a000b06010d070209000f0e030d0b030002030f08090b080e000b04") self.key Nibbles("060e0d030001090b04070605060f0d02020a0907040406080f0f0505050a03060c0a0f0205010d09080b05000d010f0702050107040b0908070e08060701040c") stack backtrace: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. 2024-10-26T15:04:51.468886Z ERROR Critical task pipeline task panicked: add_leaf key Nibbles("0503010f070f040c09000e000f0c0006010b0501050f0f040d030d01010b01050b090d050a000b06010d070209000f0e030d0b030002030f08090b080e000b04") self.key Nibbles("060e0d030001090b04070605060f0d02020a0907040406080f0f0505050a03060c0a0f0205010d09080b05000d010f0702050107040b0908070e08060701040c") 2024-10-26T15:04:51.468950Z ERROR backfill sync task dropped err=channel closed 2024-10-26T15:04:51.468982Z ERROR Fatal error in consensus engine 2024-10-26T15:04:51.469012Z ERROR shutting down due to error Error: Critical task pipeline task panicked: add_leaf key Nibbles("0503010f070f040c09000e000f0c0006010b0501050f0f040d030d01010b01050b090d050a000b06010d070209000f0e030d0b030002030f08090b080e000b04") self.key Nibbles("060e0d030001090b04070605060f0d02020a0907040406080f0f0505050a03060c0a0f0205010d09080b05000d010f0702050107040b0908070e08060701040c")

Yeah, exactly the same. Great. I was worried it was my setup