Skip to content

Tutorial: using hop client with the SCiMMA Hopskotch server

Jump to bottom
Donald Petravick edited this page Sep 1, 2023 · 13 revisions

This tutorial demonstrates how to use the hop-client to connect to the SCiMMA Hopskotch messaging network. This tutorial focuses on accessing the production cloud instance of Hopskotch; notes on how to use the development instance or a local kafka server are included at the end.

Account Setup

To connect to a Hopskotch server, a Hopskotch user account and credentials are needed.

Sign up for an account at https://my.hop.scimma.org/hopauth. Follow the "login" button. Choose an identity provider from Google, ORCID, or an institution you belong to from the dropdown list. Make a note of the identity provider you have used, and click LogOn. Review the Acceptable Use Policy, and click Accept if you agree. You will we redirected to a SCiMMA Auth page. For now, just log out, we will come back to this this application later in this tutorial.

For more information, see the Hopskotch Authenticator docs for details on account management.

Installing hop-client

The next step is to install hop-client. Hop-client is a shell tool that supports interaction with Hopscotch. When installing hop-client, a virtual environment is recommended to manage packages. You must have either pip or conda:

  • pip (requires python3-venv): 
    python3 -m venv hop-venv
source hop-venv/bin/activate
pip install pip --upgrade
pip install hop-client
  • conda: 
    conda create --name hop-venv python=3.8
conda activate hop-venv
conda install -c conda-forge hop-client

Verify your installation via hop --version or hop --help.

The hop-client also allows you to interact with the Hopskotch server network; see hop.SCiMMA for more information.

Credential Creation

After hop-client is successfully installed, you are ready to generate Hopskotch credentials and store them with hop-client (follow the steps from the Hopskotch Authenticator for help):

hop auth add
Username: <paste-your-username>
Password: <paste-your-password>    # note that this will not show your password; just press enter after you paste it
Hostname (may be empty): kafka.scimma.org

The configuration file that holds the credentials can be found via:

hop auth locate

Using hop-client

Accessing Hopskotch topics

Once you have your credentials, you will be able to read from public topics, such as GCN Circulars or the Hopskotch system heartbeat (enter Ctrl-C to exit the commands):

$ hop subscribe kafka://kafka.scimma.org/sys.heartbeat
$ hop subscribe kafka://kafka.scimma.org/gcn.circular

A full list of topics can be found via hop list-topics kafka://kafka.scimma.org. To access topics in other groups, a SCiMMA group admin must add you to the corresponding group.

Once you are in the group, you must then add topic permissions to your Hopskotch account for the new topics before you can access them:

  • Go to the Hopskotch auth website to Edit your Active Credentials
  • Go to Add Permission, select the topic name (for example, test.topic: All) from the dropdown, then click Add permission. Do this for all topics you want to have access to.
  • See the scimma-admin documentation for more help with this process.

These changes may take up to a minute to take effect.

Sending messages

The hop publish function can be used to send JSON-formatted messages to topics. Different message formats, such as VOEvent or GCNCircular, can be specified via --format <name>. New message formats can be created via hop.plugin as in the Message Formats documentation.

For example, to send the file detector-observation.gcn that is formatted as a GCN Circular to <your.topic>:

hop publish --format GCN kafka://kafka.scimma.org/<your.topic> detector-observation.gcn

Receiving messages

The hop subscribe function can be used to read messages incoming messages (as well as archival messages) from Hopskotch topics.

For example, to indefinitely read messages from the sys.heartbeat:

hop subscribe kafka://kafka.scimma.org/sys.heartbeat

To read all messages from the gcn.circular topic (warning: this contains many messages and may fill your terminal):

hop subscribe kafka://kafka.scimma.org/gcn.circular -s EARLIEST

Miscellaneous

Using a local kafka server instead of Hopskotch

The hop-client can be used with a local (or self-hosted) kafka server instead of the cloud-based Hopskotch. This has the advantage of easy testing since Hopskotch accounts are not required. The scimma-server-container is one method for this. This method requires Docker to use the server container.

Once you are prepared, start the container. No topics will exist on the server, so publish test messages to generate topics test.topic1 and test.topic2:

sudo docker run -p 9092:9092 -dit --rm=true --name=scimma-server --hostname localhost scimma/server:latest --noSecurity
sleep 20    # required to allow the server to start
echo '{"content": "initial message"}' | hop publish -f BLOB test.topic1 --no-auth
echo '{"content": "initial message"}' | hop publish -f BLOB test.topic2 --no-auth

Resources

Source code

hop-client package repositories

SCiMMA webpages

Troubleshooting

If you have set up an account and stored your credentials correctly, but are having issues, try methods in this section. You can also gather verbose logging from the hop-client by using the debug flag via hop -d ....

Errors involving messages remain unsent or Broker transport failure

Errors like this suggest that your network may be blocking outbound connections on port 9092.

Test to see if this port is open on your network: (this will quickly print P0 if the port is open; if not, it will hang for a minute and then print 1) echo 'test' | nc -w 10 kafka.scimma.org 9092; echo $?

Clone this wiki locally