How is Bigtable compatibility tested?

[li-boxuan]

I noticed https://docs.janusgraph.org/master/changelog/ has a BigTable compatibility matrix. https://github.com/JanusGraph/janusgraph/releases/tag/v0.5.2 claims BigTable compatibility is tested. I am wondering how that is tested. AFAIK, BigTable is not open-sourced and is only available on the Google Cloud Platform.

[porunov]

Pinging @mbrukman for this question

[farodin91]

We could use the bigtable emulator to run tests against bigtable: https://github.com/testcontainers/testcontainers-java/blob/f76970f570fd1f1dcd0a6087162de8a3e7733d29/modules/gcloud/src/main/java/org/testcontainers/containers/BigtableEmulatorContainer.java

[porunov]

That's great idea @farodin91 !

[mbrukman]

There's also an alternative https://github.com/steveniemitz/littletable which runs a custom written emulator (written in Scala) in-process with the JVM rather than running a few external processes (gcloud is in Python, the Cloud Bigtable emulator itself is in Go); the developer who wrote it works at Twitter and this is what they use, and they said it's lower overhead without the additional processes.

However, the way to make sure it actually works with the Cloud Bigtable service is to spin up an actual Cloud Bigtable instance in GCP and test a few operations with it (at this point manually, since we haven't automated it), which is how @kolea2, @igorbernstein2, and I have periodically tested it.

It would be good to create a way to run automated tests against an in-memory emulator as well as a real Cloud Bigtable service (if credentials and connection parameters are provided), which would make it possible to run the full test suite for every release with a single command.

[steveniemitz]

👋 littletable author here. @mbrukman pointed me this way. We use littletable pretty extensively for integration testing our TSDB built on top of Bigtable. As mentioned, not having to worry about a bunch of external, non-jvm deps was the main driver for writing it. Getting that stuff working in our CI environment would have been a huge pain.

If you decide to try it out, feel free to reach out to me, or open issues in the repo if you run into anything.