UX: Design and implement a configuration file format

[spenczar]

For background and motivation, see #82 (comment).

Description

As a user, I want to configure the hop client without needing magic invocations like "sasl.mechanism=SCRAM-SHA-512".

I want a configuration language which supports comments.

I want a configuration language which is easy readable by humans.

I want a configuration language which is easily diffed in pull requests.

I want a configuration language which supports validation without execution.

I want to configure my username and password in one place for hop-client.

I want to be able to upgrade from an old configuration file to a new one easily.

Definition of Done

Support a new configuration file format for publishing and subscribing with hop.

[myNameIsPatrick]

Based on the spec laid out by @spenczar in #82 (comment), I am proposing the following minimal configuration format using TOML:

[auth]
username = "username"
password = "..."

I'm a bit hesitant whether we should expose the following properties as well, but it may provide some flexibility while doing development and while all the security details are not set in stone yet (although please correct me if I'm wrong on this):

[auth]
...
protocol = "SASL_SSL"
mechanism = "SCRAM-SHA-512"
ssl_ca_location = "/path/to/cacert"

I went along with TOML rather than YAML, partly due to the smaller spec, partly because it's likely the configuration will be mostly flat and may look nicer, and partly due to python's adoption of TOML for project management (see https://www.python.org/dev/peps/pep-0518/).

Any comments/suggestions are welcomed.

[dpetravick]

can the config file support multiple usernames?

[myNameIsPatrick]

@dpetravick, the config format certainly can but the current spec doesn't allow for that. It would be nice to support, however. I'd have to think of a clean way of adding that in that looks sensible.

[spenczar]

What would multiple usernames mean? I don't see how you could connect to the Kafka broker as multiple users at the same time.

[myNameIsPatrick]

I had interpreted that to be supporting multiple user profiles that you can switch between at runtime (with some default).

[spenczar]

I think we should default to you-aint-gonna-need-it; that adds a lot of complexity to the config format and to the code, and I don't see the benefit. I could be missing it, though.

[shereenElSayed]

I am wondering if there is a way we can have the Kafka server URL in the config instead of rewriting it over and over again in each connection. It can be used as by default value if it is not passed in the hop-client call. Maybe we can have the same with the topics. I am not sure if people will keep changing the topics a lot or not but if the topics are already limited and well defined, they can just add the URL and topics in the configs.

[myNameIsPatrick]

Given that there are no objections/concerns behind the configuration file format or current key/value pairs proposed, I say we move forward with a minimal implementation in #89 which will provide similar functionality as what was present before. We can iterate through any additions such as 'multiple usernames' and 'default broker URL' here and propose that in a future PR.