Making bitcoind and electrsd production-grade?

[MaxFangX]

The bitcoind / electrsd crates have been enormously helpful for our integration testing, and we (Lexe) think they could be very useful in production as well.

Our interest in a production-grade bitcoind/electrs Rust harnesses: Due to the headaches of managing bash scripts / config files / systemd service files and whatnot, we actually decided to write our production (as well as testnet, dev, regtest etc) orchestration code in pure Rust, so that we could handle all of our argument shuttling in a strongly-typed language, resulting in a sharp reduction in deployment errors. Cleaning up bitcoind and electrsd to be "production-grade" software would allow us to extend our current paradigm to the deployment of these services as well, and allows bitcoind / electrsd arguments to be programmatically configurable in production. We also have an internal framework which centralizes things like CLI arguments, config files, and environment variables, so exposing Rust hooks into a bitcoind/electrs harness allows us to reuse this system as well.

So, I wanted to start a discussion on this. Some questions:

• Is anyone else interested in this?
• What is the current readiness of the bitcoind / electrsd crates from a "production" standpoint, and what work would need to be done to accomplish this?
• Is it better to pursue this development goal under this repository, or should interested parties add these capabilities in a permanent fork which does not have the intention of being merged back upstream? (Similar to the romanz/Blockstream electrsd fork situation). In other words, are the requirements of a "production-ready" vs integration test-oriented bitcoind/electrs harness so different that it would be better to develop them in separate repositories? (My hunch is this is not so).

Especially interested in your thoughts @RCasatta.

[RCasatta]

The bitcoind / electrsd crates have been enormously helpful for our integration testing, and we (Lexe) think they could be very useful in production as well.

Great to know!

Is anyone else interested in this?

Not that I am aware of

Is it better to pursue this development goal under this repository, or should interested parties add these capabilities in a permanent fork which does not have the intention of being merged back upstream?

The only functional change I see to achieve this is to make the network configurable, which I would accept. But I am not sure I get the entire picture of using this out of the testing framework.
I would hope there isn't a need to fork so we can keep a bigger user base.

What is the current readiness of the bitcoind / electrsd crates from a "production" standpoint, and what work would need to be done to accomplish this?

I think there is some work to accomplish this:

• API stabilization (would be great some help from @Kixunil)
• More test/coverage
• bigger user base
• At least another maintainer (@apoelstra do you have room for another one?), possibly moving the repo in the rust-bitcoin

Also electrsd is a lagging a little bit behind usually and it also has the issue of lacking binary released "officially", at the moment I am building the binary and I don't think it would be acceptable for production. Maybe @romanz has something to say on this.

[MaxFangX]

• API stabilization (would be great some help from @Kixunil)

This is not a big deal for us. We're happy to contribute to the 'productionizing' of these crates and use them without API stabilization guarantees. Although, if we change the wording of the readme and other documents to express that this crate is actually ready for production use, people will probably expect that API-breaking changes (before 1.0.0) come with a minor version bump, but it seems you stick to this already. If you do accept to productionize however I think it is important to remove the "integration testing" phrasing throughout the docs to make it clear to contributors that security and reliability should be taken more seriously than would be for simple test util code.

• More test/coverage

Happy to contribute here too.

Reading through the code it seems like you've already done a good job of avoiding random .unwrap()s and handling errors properly. So at least that component looks done or close to it. 👍

Another thing to think about is an async interface. All the rest of our Rust code is async, so integrating with this crate comes with a bit of a mismatch. Thankfully, this harness code is pretty high-level, rather than something that needs to be very efficient or performant, so we can probably get around it with spawn_blocking somewhere in our orchestration logic. Will report back if we run into significant problems with this approach.

[apoelstra]

Sure, I'm happy to be a second maintainer of this crate.

But @MaxFangX I'd encourage you to take a quick look at whether Nix would be a better fit than Rust for this sort of thing. Nix is is dynamically typed but it is strongly typed, unlike Bash, and can provide determinisism/reproducibility guarantees that Rust can't. (That blog post describes doing integration tests for Python and isn't directly applicable to rust-bitcoin. It's just somewhat accessible. But you can take a look at the nix-bitcoin project to see what's possible for bitcoind.)

[MaxFangX]

Thanks @apoelstra, I'll look into it!

[Kixunil]

I currently focus my efforts on the bitcoin crate and its dependencies. I could help with this after that one is (at least partially) stabilized - because it's required anyway. bitcoind depends on bitcoincore-rpc which depends on bitcoin and these dependencies are visible in the API. Expect this to take loong time.

Speaking about alternatives there's also lnpbp-testkit which is packaged in Cryptoanarchy Debian Repository (both are my projects) which allows you spin up LND as well. So all that "headaches of managing bash scripts / config files / systemd service files and whatnot" should turn into apt install bitcoin-regtest and similar. IDK what exactly you're doing so I have no idea if this is useful for you.

[romanz]

Also electrsd is a lagging a little bit behind usually and it also has the issue of lacking binary released "officially", at the moment I am building the binary and I don't think it would be acceptable for production. Maybe @romanz has something to say on this.

I am not sure if there is a best-practice method to reproducibly build electrs (given that it depends on the RocksDB)...