Not possible to returning from 4 days old backup

[calganaygun]

My node went offline and I backed up all working dir 4 days ago. I just tried to use this backup. But after the unlocking node, I see error logs on the docker logs. It is not possible to proceed on the UI of the HUB.

Logs:

{"level":"info","msg":"Launching LN Backend: LDK","time":"2024-10-24T08:13:02Z"}
{"level":"info","msg":"Started LDK node. Syncing wallet...","nodeId":"XXXXXX","status":{"IsRunning":true,"IsListening":true,"CurrentBestBlock":{"BlockHash":"00000000000000000002a14ca968a2448132772eaa47a0f381195c0ac1451662","Height":866527},"LatestWalletSyncTimestamp":null,"LatestOnchainWalletSyncTimestamp":null,"LatestFeeRateCacheUpdateTimestamp":1729757583,"LatestRgsSnapshotTimestamp":null,"LatestNodeAnnouncementBroadcastTimestamp":null},"time":"2024-10-24T08:13:03Z"}
{"error":"oauth2: \"invalid_grant\" \"The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client\"","level":"error","msg":"Failed to refresh existing token","time":"2024-10-24T08:13:04Z"}

And then, oauth2 error repeated so many times.

Also I noticed I got an error about lightning channel:

We have fallen behind - we have received proof that if we broadcast our counterparty is going to claim all our funds.
This implies you have restarted with lost ChannelMonitor and ChannelManager state, the first of which is a violation of the LDK chain::Watch requirements.
More specifically, this means you have a bug in your implementation that can cause loss of funds, or you are running with an old backup, which is unsafe.
If you have restored from an old backup and wish to force-close channels and return to operation, you should start up, call

Is there a way to force-close this channel via CLI on HUB or UI?

[rolznz]

Hi,

You restored an old backup and afterwards some lightning channel state changed. That is why LDK is preventing the node to start to avoid being penalized for using outdated chain data.

We are in the progress of building a dedicated tool to restore from a static channel backup (see #53). I will let you know as soon as it's ready for testing.

By the way, what was the reason for using a backup? did you lose access to your hub data directory?

[calganaygun]

@rolznz Thanks for the response, I am waiting to test it. Before the tool, is there any hacky way to accomplish and start the node, you can suggest? Node is important to me.

And, yes, the data directory has vanished between my homelab upgrades.

[rolznz]

@calganaygun Please note, this is experimental, and by not waiting for the tool you may be at risk of loss of funds. I didn't test this yet. Best to wait for me to confirm it works, unless you really need to proceed

You could possibly run another Alby Hub on a different device / working directory by doing an advanced setup -> importing only the seed phrase, and then connect to the peers you originally had channels with. LDK should tell the counterparty to force close the channels, because on the second Alby Hub instance it will have no knowledge of these channels, and will close unknown channels the counterparty says exist. You would then close this Alby Hub, and open your normal Alby Hub (the one you restored from the backup). The node should then start OK (since the channels were closed) and it should at some point then be able to sweep the funds from the channel, because it has the original channel monitors.

[rolznz]

Possibly another option is we ask the counterparty to close the channel for you. If it was a channel with one of the LSPs, you could contact support@getalby.com with your node ID and the LSP(s) you used and we can ask them to force close it. Then you should be able to use your Alby Hub that was restored from the backup.

[calganaygun]

Great suggestion thanks, I have also thought about directly adding the reading static channel files and passing the current nil element on

hub/service/start.go

Line 185 in 70ac096

lnClient, err = ldk.NewLDKService(ctx, svc.cfg, svc.eventPublisher, Mnemonic, LDKWorkdir, svc.cfg.GetEnv().LDKNetwork, nil, false)

. To run once in this specific case and then return to the official image of the hub.

But looks like, It is more convenient to ask Olympus to close the channel. I am sending a support request. Thanks for the help

[rolznz]

@calganaygun great!

Regarding the code: we did some initial testing with that code to ensure SCB recovery worked, but it requires running in development mode. We're migrating that code to this new tool also to simplify the LDK code in Alby Hub.

[rolznz]

Good news - funds were recovered after asking LSPs to close the channels.

I will keep this issue open until the backup restoration tool is available.