Build out some integration with Cloud Bigtable

[jgeewax]

We want to make users of Happybase "just work", so...some ideas:

Option 1: A wrapped import module

from gcloud.bigtable import happybase

Option 2: Some sort of monkey patching?

from gcloud import bigtable
import happybase
happybase = bigtable.monkey_patch(happybase)

[jgeewax]

/cc @maxluebbe

[jgeewax]

/cc @dhermes @tseaver : Max might have some spare cycles to work on this, figured you guys should chat.

[dhermes]

Relevant docs:
https://happybase.readthedocs.org/en/latest/
https://cloud.google.com/bigtable/docs/

[tseaver]

ISTM that monkey-patching the happybase third-party module would be a poor choice. I think we need to think of this as "write a happybase API emulation layer" instead (which may or may not itself even use the real happybase code).

[dhermes]

Can you elaborate? Why is it a bad idea? (I have yet to look at the happybase code/docs.)

[tseaver]

I think our goal is that they should be able to port scripts easily (ideally, just changing import happytable to from gcloud.bigtable import happytable and adjusting the connection parameters). I don't see interfering with the actual happytable module's internals as a useful way to approach that goal. For instance, users of gcloud.bigtable might need to interact with a real happybase/Thrift backend (e.g., to move or compare data between it and the bigtable dataset).

[jgeewax]

I agree that there may be times where someone would want to talk to Cloud Bigtable and HBase at the same time, so monkey-patching happybase sounds like we'd screw over those multi-tenant situations.

Maybe there's a way to from gcloud import bigtable and then happybase_compatible_client = bigtable.get_happybase_client() which uses the real happybase code, but swaps out a connection of sorts at the lower level...

[tseaver]

The gcloud.bigtable docs say:

Cloud Bigtable works with Hadoop 2.4 and higher and is interface compatible with Apache HBase 1.0.

If that means that it exports the same REST API as HBase / Thrift, then maybe we don't have to do much to enable happybase, beyond ensuring that we set the connection up with gcloud-relevant credentials.

[jgeewax]

This is the interesting piece. It is not wire-protocol compatible. That is -- it doesn't speak Thrift (AFAIK). Instead, they made the Java code compatible, and swapped out the thing that does serialization and sending of requests underneath.

I think @maxluebbe can comment more to confirm.

[dhermes]

Yeah that one is important. I was under the impression from @jgeewax previous comments that the web APIs were the same / close cousins.

[brianolson]

The Java "HBase" client to Bigtable appears to be an emulation layer. A Python client should be built on the same protobuf rpc interface as that:
https://github.com/GoogleCloudPlatform/cloud-bigtable-client/tree/master/bigtable-protos/src/main/proto/google

I'd imagine that Python client having a native mode that just handles the proto-rpc; and then there could be a happybase emulation layer on top of that.

[maxluebbe]

Sorry have been OOO. Some clarifications.

1. Additional work item is modernizing Happybase to 1.0 HBase API.
2. Cloud Bigtable does not support Thrift/Rest interfaces. Part of the work that needs to be done here is decoupling the thrift parts of happybase from the interface provided to the user, so that we can drop in gRPC in place of Thrift.

[jgeewax]

gRPC or Protobuf over HTTP 1.0?

gRPC is not possible right now given the installation process (we need to keep our install to pip install gcloud and can't ask people to do a bunch of Makefile madness to use our library).