Skip to content

Tutorial: using hop client with the SCiMMA Hopskotch server

Jump to bottom
Bryce Cousins edited this page Oct 5, 2021 · 13 revisions

This tutorial demonstrates how to use the hop-client to connect to the SCiMMA Hopskotch messaging network. This tutorial focuses on accessing the production cloud instance of Hopskotch; notes on how to use the development instance or a local kafka server are included at the end.

Account Setup

To connect to a Hopskotch server, a Hopskotch user account and credentials are needed.

Sign up for an account at https://my.hop.scimma.org/hopauth. Follow the directions to create an account (or go directly to https://registry.scimma.org/registry/co_petitions/start/coef:127). Select your corresponding institution from the dropdown (under Google) and click Begin to start the account setup process. Once this is completed, your account can be approved.

For more information, see the Hopskotch Authenticator docs for details on account management.

Installing hop-client

Once your account has been approved, you will be able to generate credentials to access the SCiMMA Hopskotch server. But, it is recommended that you first install hop-client before generating credentials. This will simplify the authentication configuration.

When installing hop-client, a virtual environment is recommended to manage packages. You must have either pip or conda:

  • pip (requires python3-venv): 
    python3 -m venv hop-venv
source hop-venv/bin/activate
pip install pip --upgrade
pip install hop-client
  • conda: 
    conda create --name hop-venv python=3.8
conda activate hop-venv
conda install hop-client

Verify your installation by checking hop --version:

hop --version
  hop version 0.4

The hop-client also allows you to interact with the Hopskotch server network; see hop.SCiMMA for more information.

Credential Creation

After hop-client is successfully installed, you are ready to generate Hopskotch credentials and store them with hop-client (follow the steps from the Hopskotch Authenticator for help):

hop auth add
Username: <paste-your-username>
Password: <paste-your-password>    # note that this will not show your password; just press enter after you paste it
Hostname (may be empty): kafka.scimma.org

The configuration file that holds the credentials can be found via:

hop auth locate

Using hop-client

Accessing Hopskotch topics

Once you have your credentials, you will be able to read from public topics, such as GCN Circulars or the Hopskotch system heartbeat:

$ hop subscribe --persist kafka://kafka.scimma.org/sys.heartbeat
$ hop subscribe --persist kafka://kafka.scimma.org/gcn.circular

A full list of topics can be found via hop list-topics kafka://kafka.scimma.org. To access topics in other groups, a SCiMMA group admin must add you to the corresponding group.

Once you are in the group, you must then add topic permissions to your Hopskotch account for the new topics before you can access them:

  • Go to the Hopskotch auth website to Edit your Active Credentials
  • Go to Add Permission, select the topic name (for example, test.topic: All) from the dropdown, then click Add permission. Do this for all topics you want to have access to.
  • See the scimma-admin documentation for more help with this process.

These changes may take up to a minute to take effect.

Sending messages

The hop publish function can be used to send JSON-formatted messages to topics. Different message formats, such as VOEvent or GCNCircular, can be specified via --format <name>. New message formats can be created via hop.plugin as in the Message Formats documentation.

For example, to send the file detector-observation.gcn that is formatted as a GCN Circular to <your.topic>:

hop publish --format GCN kafka://kafka.scimma.org/<your.topic> detector-observation.gcn

Receiving messages

The hop subscribe function can be used to read messages incoming messages (as well as archival messages) from Hopskotch topics.

For example, to indefinitely read messages from the sys.heartbeat:

hop subscribe --persist kafka://kafka.scimma.org/sys.heartbeat

To read all messages from the gcn.circular topic (warning: this contains many messages and may fill your terminal):

hop subscribe kafka://kafka.scimma.org/gcn.circular -s EARLIEST

Miscellaneous

Using the development Hopskotch instance

There are two instances of Hopskotch: production (prod) and development (dev). Prod is more stable, but may not have the most recent additions to Hopskotch. To use dev instead, you will simply need to sign up for a dev Hopskotch account. Note that accounts, credentials, and topics are not shared between prod and dev, so you cannot access information in the other instance.

To create an account on the dev instance, log in at https://hop.scimma.org/IAM/.

The management of credentials on dev can be done at

The remaining steps are identical to the above prod tutorial, but you must add your dev credential to hop-client separately (note the different Hostname):

hop auth add
Username: <paste-your-username>
Password: <paste-your-password>    # note that this will not show your password; just press enter after you paste it
Hostname (may be empty): dev.hop.scimma.org

Note that the topics on dev are accessed via kafka://dev.hop.scimma.org (not kafka://kafka.scimma.org as in prod).

Using a local kafka server instead of Hopskotch

The hop-client can be used with a local (or self-hosted) kafka server instead of the cloud-based Hopskotch. This has the advantage of easy testing since Hopskotch accounts are not required. The scimma-server-container is one method for this. This method requires Docker to use the server container.

Once you are prepared, start the container. No topics will exist on the server, so publish test messages to generate topics test.topic1 and test.topic2:

sudo docker run -p 9092:9092 -dit --rm=true --name=scimma-server --hostname localhost scimma/server:latest --noSecurity
sleep 20    # required to allow the server to start
echo '{"content": "initial message"}' | hop publish -f BLOB test.topic1 --no-auth
echo '{"content": "initial message"}' | hop publish -f BLOB test.topic2 --no-auth

Resources

Source code

hop-client package repositories

SCiMMA webpages

Troubleshooting

If you have set up an account and stored your credentials correctly, but are having issues, try methods in this section. You can also gather verbose logging from the hop-client by using the debug flag via hop -d ....

Errors involving messages remain unsent or Broker transport failure

Errors like this suggest that your network may be blocking outbound connections on port 9092.

Test to see if this port is open on your network: (this will quickly print P0 if the port is open; if not, it will hang for a minute and then print 1) echo 'test' | nc -w 10 kafka.scimma.org 9092; echo $?

Request Group Access (if needed)

Once your account has been approved, you will have the SCiMMA Active Collaborator status, which should automatically grant you access to Hopskotch as a kafkaUser.

If you do not have kafkaUser status, you must add yourself to the Hopskotch user group inside COManage:

Clone this wiki locally