kgo: record all records and validate leader epoch monotonicity

[nvartolomei]

https://github.com/redpanda-data/kgo-verifier/pull/58/commits

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.2.x
• v24.1.x
• v23.3.x

Release Notes

• none

[vbotbuildovich]

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56825#0192a076-cdc7-478f-94b9-0ba402665ce3:

"rptest.tests.shard_placement_test.ShardPlacementTest.test_node_join"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type_and_url_style=.CloudStorageType.S3.1.virtual_host.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56825#0192a076-cdc5-4485-b2a6-934cfb6d7a6f:

"rptest.tests.shard_placement_test.ShardPlacementTest.test_manual_rebalance"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56825#0192a07a-f5b7-4af5-a800-b56aaf948bab:

"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shard_placement_test.ShardPlacementTest.test_node_join"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56825#0192a07a-f5b6-48bb-a3ef-60a97e1caa23:

"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.ABS"

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a076-cdc7-478f-94b9-0ba402665ce3
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a076-cdc1-4c16-81b2-9ad23254320f
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a076-cdc3-4897-a3dd-f3e6d1daff4a
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a076-cdc5-4485-b2a6-934cfb6d7a6f
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a07a-f5b7-4af5-a800-b56aaf948bab
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a07a-f5b6-48bb-a3ef-60a97e1caa23
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a07a-f5b2-4bb4-85a8-ae9455245948
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56825#0192a07a-f5b4-440e-baeb-1058d4621943

[vbotbuildovich]

Retry command for Build#56825

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_node_join
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"test_case":{"name":"(TS_Read == True, TS_TxRangeMaterialized == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_upgrade
tests/rptest/tests/consumer_offsets_consistency_test.py::ConsumerOffsetsConsistencyTest.test_flipping_leadership
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_manual_rebalance
tests/rptest/tests/upgrade_test.py::UpgradeBackToBackTest.test_upgrade_with_all_workloads@{"single_upgrade":false}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy@{"cloud_storage_type":1,"short_retention":true}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy@{"cloud_storage_type":2,"short_retention":true}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy@{"cloud_storage_type":2,"short_retention":false}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy@{"cloud_storage_type":1,"short_retention":false}
tests/rptest/tests/workload_upgrade_runner_test.py::RedpandaUpgradeTest.test_workloads_through_releases@{"cloud_storage_type":1}

[andrewhsu]

i suggest this PR block on merge of redpanda-data/kgo-verifier#58

[andrewhsu]

fab9660 points to a git commit in unmerged PR redpanda-data/kgo-verifier#58. it would be best to point to the merge commit after that PR merges to main to ensure the git lineage tracks back to latest of what exists on main branch