RPC support for CometBFT 0.38 (#1317)

* rpc: 0.38 support for /block_results endpoint

In the v0_37+ dialect helper, support the new finalize_block_events
field if it is present. Also default the begin_block_events and
end_block_events fields to None if they are not present.

* rpc: add app_hash field to /block_results response

* tendermint: restore Deserialize on some types

For abci::Event and abci::response::CheckTx, restore the Deserialize
implementation on domain types themselves.

The event serialization format as used in 0.37+ will be established as
the "canonical" serialization.

* tendermint: serde impls for ExecTxResult

* rpc: support more changes in CometBFT 0.38

Add the new fields and types without static differentiation for 0.38.x.

Use the domain type ExecTxResult in deserializing the endpoint response types in the current (i.e. 0.37+) dialect for these endpoints:
- /block_results
- /broadcast_tx_commit
- /tx
- /tx_commit

The Deserialize impls are restored on the response types,
corresponding to the JSON schema for the current RPC dialect.

Simplify the 0.34 dialect support for these responses, no longer using
generics.

In endpoint::tx_commit::Response, rename the deliver_tx field to
tx_result to correspond to the current JSON schema. The Deserialize
impl falls back to deliver_tx to also be able to parse 0.37 responses.

* rpc: recognize 0.38 in CompatMode::from_version

* Fix deserialization of RPC results

Some serde attributes were not migrated onto the ExecTxResult
domain type when it has been tasked with deserializing the latest
RPC dialect.
On abci::response::CheckTx, add serde(default) on fields that are
no longer present in CometBFT 0.38.

* rpc: 0.37 compat on /block_results response

If the finalize_block_events is not present, default to an empty vector.
Conversely, skip the field from serializing if it's set to an
empty vector.

* Add kvstore fixtures for 0.38

* Derive serde impls for abci responses

BeginBlock, EndBlock, FinalizeBlock are used to deserialize
RPC responses.

* rpc: adapt serialization of Event for 0.38

Now that event data can have new fields, distinguish this with
a new NewBlock enum variant, renaming the old one to LegacyNewBlock.

The dialect serialization helpers for subscription events are
separated to v0_34::DialectEvent, that does not recognize the
0.38 fields at all and parses the event attributes as base64-encoded,
and latest::DialectEvent, which has provisions for 0.38 fields if they
are present, and in this case converts to the NewBlock variant.

* base64 for app_hash in JSON of FinalizeBlock

* rpc: version-alias latest event serde helpers

For consistency, define event::v0_37 and event::v0_38, aliasing
the public definitions in event::latest.

* rpc: split ser/de helpers for Event

The Serialize impl needs to be implemented differently for
0.37 and 0.38 in order to emit old or new fields for NewBlock.
The Deserialize impl, however, flexibly handles both formats to avoid
introducing another dialect where most other helpers would be
identical. This means that Event gets two different helpers for each
protocol version: DeEvent for Deserialize and SerEvent for Serialize.

* rpc: 0.37 serialization for /broadcast_tx_commit

Provide v0_37::DialectResponse as a way to serialize responses
in the 0.37 JSON format.

.changelog/unreleased/breaking-changes/1317-cometbft-rpc-0.38.md

@@ -0,0 +1,24 @@
+- `[tendermint-rpc]`  Changes to support the RPC protocol in CometBFT 0.38
+  ([\#1317](https://github.com/informalsystems/tendermint-rs/pull/1317)):
+  * Add `finalize_block_results` and `app_hash` fields to
+    `endpoint::block_results::Response`.
+  * The `deliver_tx` field is renamed to `tx_result` in
+    `endpoint::broadcast::tx_commit::Response`.
+  * The `tx_result` field type changed to `ExecTxResult` in
+    `endpoint::tx::Response`.
+  * The `event::EventData::NewBlock` variant is renamed to `LegacyNewBlock`.
+    The new `NewBlock` variant only carries fields relevant since CometBFT 0.38.
+  * Removed `event::DialectEvent`, replaced with non-generic serialization
+    helpers in `event::{v0_34, v0_37, v0_38}`. The `Deserialize` helpers in
+    the latter two modules are aliased from common types that can support both
+    fields added in CometBFT 0.38, `block_id` and `result_finalize_block`,
+    as well as the fields present 0.37. Likewise for `DialectEventData`
+    and other event data structure types.
+  * Changed some of the serialization dialect helpers to only be
+    used by the 0.34 dialect and remove generics. The current dialect's
+    seralization is switched to the serde impls on the domain types in
+    `tendermint`.
+- `[tendermint]` Changes to support the RPC protocol in CometBFT 0.38
+  ([\#1317](https://github.com/informalsystems/tendermint-rs/pull/1317)):
+  * Due to some attribute changes, the format emitted by `Serialize` is
+    changed for `abci::response` types `CheckTx` and `FinalizeBlock`.

.changelog/unreleased/improvements/1317-cometbft-rpc-0.38.md

@@ -0,0 +1,9 @@
+- `[tendermint-rpc]` Support for CometBFT 0.38
+  ([\#1317](https://github.com/informalsystems/tendermint-rs/pull/1317)):
+  * `Deserialize` implementations on `abci::Event`, `abci::EventAttribute`
+    that correspond to the current RPC serialization.
+  * Domain types under `abci::response` also get `Deserialize` implementations
+    corresponding to the current RPC serialization.
+  * `Serialize`, `Deserialize` implementations on `abci::types::ExecTxResult`
+    corresponding to the current RPC serialization.
+  * Added the `apphash_base64` serializer module.

rpc/src/client/bin/main.rs

@@ -7,8 +7,7 @@ use structopt::StructOpt;
 use tendermint::Hash;
 use tendermint_rpc::{
     client::CompatMode,
-    dialect::{Dialect, LatestDialect},
-    event::DialectEvent,
+    event::{self, Event, EventData},
     query::Query,
     Client, Error, HttpClient, Order, Paging, Scheme, Subscription, SubscriptionClient, Url,
     WebSocketClient,
@@ -377,6 +376,8 @@ where
                 .map_err(Error::serde)?
         },
         ClientRequest::BroadcastTxCommit { tx } => {
+            // NOTE: this prints out the response in the 0.38+ format,
+            // regardless of the actual protocol version.
             serde_json::to_string_pretty(&client.broadcast_tx_commit(tx).await?)
                 .map_err(Error::serde)?
         },
@@ -504,8 +505,7 @@ async fn recv_events_with_timeout(
                     }
                 };
                 let event = result?;
-                let event: DialectEvent<<LatestDialect as Dialect>::Event> = event.into();
-                println!("{}", serde_json::to_string_pretty(&event).map_err(Error::serde)?);
+                print_event(event)?;
                 event_count += 1;
                 if let Some(me) = max_events {
                     if event_count >= (me as u64) {
@@ -526,11 +526,7 @@ async fn recv_events(mut subs: Subscription, max_events: Option<u32>) -> Result<
     let mut event_count = 0u64;
     while let Some(result) = subs.next().await {
         let event = result?;
-        let event: DialectEvent<<LatestDialect as Dialect>::Event> = event.into();
-        println!(
-            "{}",
-            serde_json::to_string_pretty(&event).map_err(Error::serde)?
-        );
+        print_event(event)?;
         event_count += 1;
         if let Some(me) = max_events {
             if event_count >= (me as u64) {
@@ -542,3 +538,21 @@ async fn recv_events(mut subs: Subscription, max_events: Option<u32>) -> Result<
     info!("The server terminated the subscription");
     Ok(())
 }
+
+fn print_event(event: Event) -> Result<(), Error> {
+    let json = match &event.data {
+        EventData::LegacyNewBlock { .. } => {
+            // Print the old field structure in case the event was received
+            // from a pre-0.38 node. This is the only instance where the
+            // structure of the dumped event data currently differs.
+            let ser_event: event::v0_37::SerEvent = event.into();
+            serde_json::to_string_pretty(&ser_event).map_err(Error::serde)?
+        },
+        _ => {
+            let ser_event: event::v0_38::SerEvent = event.into();
+            serde_json::to_string_pretty(&ser_event).map_err(Error::serde)?
+        },
+    };
+    println!("{}", json);
+    Ok(())
+}

rpc/src/client/compat.rs

@@ -49,6 +49,7 @@ impl CompatMode {
         match (version.major, version.minor) {
             (0, 34) => Ok(CompatMode::V0_34),
             (0, 37) => Ok(CompatMode::V0_37),
+            (0, 38) => Ok(CompatMode::V0_37),
             _ => Err(Error::unsupported_tendermint_version(version.to_string())),
         }
     }
@@ -88,7 +89,11 @@ mod tests {
             CompatMode::from_version(parse_version("v0.37.0")).unwrap(),
             CompatMode::V0_37
         );
-        let res = CompatMode::from_version(parse_version("v0.38.0"));
+        assert_eq!(
+            CompatMode::from_version(parse_version("v0.38.0")).unwrap(),
+            CompatMode::V0_37
+        );
+        let res = CompatMode::from_version(parse_version("v0.39.0"));
         assert!(res.is_err());
         let res = CompatMode::from_version(parse_version("v1.0.0"));
         assert!(res.is_err());

rpc/src/client/transport/mock.rs

@@ -264,12 +264,10 @@ mod test {
 
     mod v0_34 {
         use super::*;
-        use crate::dialect::v0_34::Event as RpcEvent;
-        use crate::event::DialectEvent;
+        use crate::event::v0_34::DeEvent;
 
         async fn read_event(name: &str) -> Event {
-            let msg = DialectEvent::<RpcEvent>::from_string(read_json_fixture("v0_34", name).await)
-                .unwrap();
+            let msg = DeEvent::from_string(read_json_fixture("v0_34", name).await).unwrap();
             msg.into()
         }
 
@@ -334,12 +332,10 @@ mod test {
 
     mod v0_37 {
         use super::*;
-        use crate::dialect::v0_37::Event as RpcEvent;
-        use crate::event::DialectEvent;
+        use crate::event::v0_37::DeEvent;
 
         async fn read_event(name: &str) -> Event {
-            let msg = DialectEvent::<RpcEvent>::from_string(read_json_fixture("v0_37", name).await)
-                .unwrap();
+            let msg = DeEvent::from_string(read_json_fixture("v0_37", name).await).unwrap();
             msg.into()
         }
 

rpc/src/client/transport/router.rs

@@ -144,7 +144,7 @@ mod test {
     use super::*;
     use crate::{
         client::sync::{unbounded, ChannelRx},
-        event::{Event, WrappedEvent},
+        event::Event,
         utils::uuid_str,
     };
 
@@ -180,16 +180,15 @@ mod test {
 
     mod v0_34 {
         use super::*;
-        use crate::dialect::v0_34::Event as RpcEvent;
+
+        type WrappedEvent = crate::response::Wrapper<crate::event::v0_34::DialectEvent>;
 
         async fn read_event(name: &str) -> Event {
-            serde_json::from_str::<WrappedEvent<RpcEvent>>(
-                read_json_fixture("v0_34", name).await.as_str(),
-            )
-            .unwrap()
-            .into_result()
-            .unwrap()
-            .into()
+            serde_json::from_str::<WrappedEvent>(read_json_fixture("v0_34", name).await.as_str())
+                .unwrap()
+                .into_result()
+                .unwrap()
+                .into()
         }
 
         #[tokio::test]
@@ -229,16 +228,15 @@ mod test {
 
     mod v0_37 {
         use super::*;
-        use crate::dialect::v0_37::Event as RpcEvent;
+
+        type WrappedEvent = crate::response::Wrapper<crate::event::v0_37::DeEvent>;
 
         async fn read_event(name: &str) -> Event {
-            serde_json::from_str::<WrappedEvent<RpcEvent>>(
-                read_json_fixture("v0_37", name).await.as_str(),
-            )
-            .unwrap()
-            .into_result()
-            .unwrap()
-            .into()
+            serde_json::from_str::<WrappedEvent>(read_json_fixture("v0_37", name).await.as_str())
+                .unwrap()
+                .into_result()
+                .unwrap()
+                .into()
         }
 
         #[tokio::test]