Chain sync issues with gameofzoneshub-2

[zmanian]

We are seeing issues with full nodes for gameofzoneshub-2.

Iqlusion has a full node halted at 2299.

We have a report of a full node halted at 2328.

Asking for help from the Interchain Berlin team to debug.

Will update scoring timeline based on feed back from Interchain Berlin.

[zmanian]

Also seeing reports of invalid port errors.

[jeelimm]

For invalid port error, the below is our log.

1. We use public sentry node of our own
2. We completed connecting to client and connection, but err(port: invalid port) occurs when we try to create channel.

I[2020-05-11|04:29:49.494] ✔️ [gameofzoneshub-2]@{3356} - msg(0:create_client) hash(A57B3FCBB70A82CD4222AF3DB955966BE98AD72FF6B89B29E806DA1DB22405CF)
I[2020-05-11|04:29:53.849] ✔️ [achain-2]@{337} - msg(0:create_client) hash(8D8643887289ADAD3EEEB6BD591F87036274FDE535ECC2118CDB89FBFE2056D5)
I[2020-05-11|04:29:53.850] ★ Clients created: [gameofzoneshub-2]client(srchclientb) and [achain-2]client(dstaclientb)
I[2020-05-11|04:30:00.862] ✔️ [gameofzoneshub-2]@{3358} - msg(0:connection_open_init) hash(35449806D4CE76E7F29E0C03B81AC93C6C446D4C8AA4A730453783A203304E62)
I[2020-05-11|04:30:06.235] ✘ [gameofzoneshub-2]@{3359} - msg(0:connection_open_init) err(connection: connection already exists)
I[2020-05-11|04:30:18.937] ✔️ [achain-2]@{342} - msg(0:update_client,1:connection_open_try) hash(132AF085F83EBF10BDD4FDF4F35A619428BFB660FFECDBB7732979328A5DB3E1)
I[2020-05-11|04:30:28.969] ✘ [achain-2]@{344} - msg(0:update_client,1:connection_open_try) err(connection: invalid connection)
I[2020-05-11|04:30:34.190] ✘ [gameofzoneshub-2]@{0} - msg(0:update_client,1:connection_open_ack) err(commitment: invalid proof)
I[2020-05-11|04:30:45.635] ✔️ [gameofzoneshub-2]@{3366} - msg(0:update_client,1:connection_open_ack) hash(119723645087BF0ADC66D2F3A578FAF84100CC823FADB632E0380A85AA1AC4C9)
I[2020-05-11|04:30:59.070] ✔️ [achain-2]@{350} - msg(0:update_client,1:connection_open_confirm) hash(2FCE962986132F901D52238DAA7C75026417A479047203F0C65BFD589362BFF3)
I[2020-05-11|04:30:59.070] ★ Connection created: [gameofzoneshub-2]client{srchclientb}conn{srchconnectionb} -> [achain-2]client{dstaclientb}conn{dstaconnectionb}
I[2020-05-11|04:31:02.361] ✘ [gameofzoneshub-2]@{3369} - msg(0:channel_open_init) err(port: invalid port)
I[2020-05-11|04:31:13.431] ✘ [gameofzoneshub-2]@{3371} - msg(0:channel_open_init) err(port: invalid port)
I[2020-05-11|04:31:24.349] ✘ [gameofzoneshub-2]@{3373} - msg(0:channel_open_init) err(port: invalid port)

[jhernandezb]

Error syncing a full node with peers listed in the repo.

6ed008bf3a2ad341d84391bf47ea46e75a87e35e@35.233.155.199:26656
7cb9cbba21fdc3b004f098c116e5e2c2ac77ddfb@34.83.218.4:26656

name: gaia
server_name: gaiad
client_name: gaiacli
version: 0.0.0-196-gbde5e9c
commit: bde5e9c003256cfb6417d3441889bafd14089e0b
build_tags: netgo,ledger
go: go version go1.14.2 linux/amd64

May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: I[2020-05-11|05:14:22.554] Executed block                               module=state height=2327 validTxs=0 invalidTxs=0
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: I[2020-05-11|05:14:22.563] Committed state                              module=state height=2327 txs=0 appHash=0373BF4BFD56FE62BD721860CB2EB4BEABBF1D185E78BD151630CBDAF6FB7E1E
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: I[2020-05-11|05:14:22.595] Executed block                               module=state height=2328 validTxs=1 invalidTxs=0
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: I[2020-05-11|05:14:22.604] Committed state                              module=state height=2328 txs=1 appHash=AF3FC83C4A2B42E69B5B15FBCA178DF0C64C7ABF853B9D15CD6CF6268EB040BC
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: panic: Failed to process committed block (2329:2994DE3B79D078C815D712828F4F28DD9C3D63B81E5F1FB8817E062BD22970D5): wrong Block.Header.AppHash.  Expected AF3FC83C4A2B42E69B5B15FBCA178DF0C64C7ABF853B9D15CD6CF6268EB040BC, got 3E629FF1789D17C0C09706AC1E52345B67D222BEFF965FA31DE18CB53BDDF4EC
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: goroutine 699 [running]:
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc0003a11e0)
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]:         github.com/tendermint/tendermint@v0.33.4/blockchain/v0/reactor.go:349 +0x12ed
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]: created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
May 11 05:14:22 ip-172-31-3-25 gaiad[14921]:         github.com/tendermint/tendermint@v0.33.4/blockchain/v0/reactor.go:118 +0x84

May 11 05:14:32 ip-172-31-3-25 systemd[1]: Stopped gaiad.
May 11 05:14:32 ip-172-31-3-25 systemd[1]: Started gaiad.
May 11 05:14:32 ip-172-31-3-25 gaiad[15038]: I[2020-05-11|05:14:32.502] starting ABCI with Tendermint                module=main
May 11 05:14:32 ip-172-31-3-25 gaiad[15038]: ERROR: error during handshake: error on replay: wrong Block.Header.AppHash.  Expected AF3FC83C4A2B42E69B5B15FBCA178DF0C64C7ABF853B9D15CD6CF6268EB040BC, got 3E629FF1789D17C0C09706AC1E52345B67D222BEFF965FA31DE18CB53BDDF4EC

[sriney-staked]

Same error as @jhernandezb , submitted data directly to @zmanian

[zhangyelong]

I can confirm this issue is related to cosmos/relayer#224, it seems the validators of gameofzoneshub-2 have already lost the capabilities for creating channels, try following steps and you will get height 2328 passed:

gaiad unsafe-reset-all
gaiad start

# stop gaiad before height 2328 and start again
gaiad start

Note this is NOT the solution to get gameofzoneshub-2 working, it's just an evidence to show that the validators have lost the capabilities, and your node will lose the capabilities after restarting too, that is why we can pass height 2328 after restarting the node

[Thunnini]

@zhangyelong I also solved this invalid app hash error in this way like you. But, there seems to be a bug in which the node does not load the state properly after restarting. They lost capabilities and the "transfer" port state. So, I can't make the channel like @jeelimm.

[asmodat]

gaiad unsafe-reset-all
gaiad start

# stop gaiad before height 2328 and start again
gaiad start

Does NOT fix the connectivity issue and also getting invalid port here on the rly

[zhangyelong]

gaiad unsafe-reset-all
gaiad start

# stop gaiad before height 2328 and start again
gaiad start

Does NOT fix the connectivity issue and also getting invalid port here on the rly

Yes, I have said that it is not the solution, it's just an evidence :-) The network has lost the capabilities for creating channels.

[iammelea]

new nodes cant sync. halted on 2329

one node sync before is

still working

can't start GoZ in this conditions.

[jjangg96]

gaiad start --halt-height=2000
gaiad start

[joe-bowman]

gaiad start --halt-height=2000
gaiad start

This might get past the point on hub; but the fact it does is a huge red flag itself. Don't rely on this. The chain should be deterministic. It is broken.

[stakezone]

I have the same invalid port error as above when trying to create a channel.

I[2020-05-11|10:34:56.427] ✘ [gameofzongshub-2]@{5989} - msg(0:update_client,1:channel_open_try) err(port: invalid port)
{"height":"5989","txhash":"568970A886106B6FADB0531DC9075D777C3F691E053BF7321C8CA3CE609273CD","codespace":"port","code":4,"raw_lo
g":"failed to execute message; message index: 1: could not retrieve module from portID: invalid port","gas_wanted":"8000000","ga
s_used":"96734"}