Adding pika authentication and sleep time to queue consumer #221

lmarinve · 2024-02-05T20:12:04Z

sleep time and pika authentication added to: sdx_controller/messaging/rpc_queue_consumer.py

congwang09

Hey Luis, got some comments here.

congwang09 · 2024-02-06T02:56:17Z

Dockerfile

 WORKDIR /usr/src/app

+<<<<<<< HEAD


This should be from the conflict between the two branches. We should fix this.

congwang09 · 2024-02-06T02:57:21Z

sdx_controller/messaging/rpc_queue_consumer.py

 # subscribe to the corresponding queue
 SUB_QUEUE = os.environ.get("SUB_QUEUE")
-
+MQ_SRVC = os.environ.get("MQ_SRVC")


If there are new environment variables added, they should be added to the env.template, so we can have examples when deploying this.

Hi @congwang09, @sajith, MQ_SRVC is a docker-compose variable that receives the existing .env variable RABBITMQ_DEFAULT_HOST, which has the docker-compose service name rabbitmq3

congwang09 · 2024-02-06T02:58:19Z

sdx_controller/messaging/rpc_queue_consumer.py

 logger = logging.getLogger(__name__)


 class RpcConsumer(object):
    def __init__(self, thread_queue, exchange_name, topology_manager):
        self.logger = logging.getLogger(__name__)
+        SLEEP_TIME = 5


The constant should be defined on top of the file. Also it'll be nice to also set it as an env var, with a default value. This makes it configurable.

Why do we a need to add a delay anyway?

I suspect this is to wait for the RabbitMQ service to start up, in the case of RabbitMQ and SDX Controller spawned by the same compose file. If that is the case, we should be using depends_on in the compose file.

Hi @sajith, docker depends on is only responsible for managing the status of Docker services, not the internal services running inside the container, such as RabbitMQ. To avoid repetition, I plan to add a new variable to the .env file, as we will use this value multiple times in sdx lc and controller.

I do not understand. The message queue would not be running in the same container. What is the delay for?

congwang09 · 2024-02-06T02:58:52Z

sdx_controller/messaging/rpc_queue_consumer.py

@@ -38,9 +46,17 @@ def on_request(self, ch, method, props, message_body):
        response = message_body
        self._thread_queue.put(message_body)

+        self.logger = logging.getLogger(__name__)
+        SLEEP_TIME = 5


This has been set in L25.

congwang09 · 2024-02-06T02:59:54Z

sdx_controller/messaging/rpc_queue_consumer.py

@@ -38,9 +46,17 @@ def on_request(self, ch, method, props, message_body):
        response = message_body
        self._thread_queue.put(message_body)

+        self.logger = logging.getLogger(__name__)


This logger has been set in L24.

congwang09 · 2024-02-06T03:00:48Z

sdx_controller/messaging/rpc_queue_consumer.py

        self.connection = pika.BlockingConnection(
-            pika.ConnectionParameters(host=MQ_HOST)
+            pika.ConnectionParameters(MQ_SRVC, 5672, "/", credentials)


It would be nice to have the MQ port number configurable, as an env var.

Sure, I will add this as well. we need to update sdx lc and controller with this change

sajith

The way I have understood, we can achieve password auth with something like this:

MQ_USER = os.getenv("MQ_USER") or "guest"
MQ_PASS = os.getenv("MQ_PASS") or "guest"

creds = pika.PlainCredentials(username=MQ_USER, password=MQ_PASS)
params = pika.ConnectionParameters(host=MQ_HOST, credentials=creds)

That is the general idea. Adding a sleep time is probably unnecessary, and if there is a reason, we should document it.

Also please be sure to run the tests in your local development setup. How to do this is documented in the README file of this repository. If something is unclear, we should address that.

sajith · 2024-02-06T13:36:59Z

Dockerfile

-FROM python:3.9-slim-bullseye
-
-RUN apt-get update \
-    && apt-get -y upgrade \
-    && apt-get install -y --no-install-recommends gcc python3-dev git \
-    && apt-get clean \
-    && rm -rf /var/lib/apt/lists/* \
-    && mkdir -p /usr/src/app
+FROM flask-base

+RUN mkdir -p /usr/src/app
 WORKDIR /usr/src/app

+<<<<<<< HEAD
+COPY ./container-sdx-controller /usr/src/app
+=======
 WORKDIR /usr/src/app
 COPY . /usr/src/app
+>>>>>>> main


These changes to Dockerfile are not necessary, and they are not within the scope of this PR. They contain errors too, with git merge markers. Please revert the changes.

There is a different Dockerfile that needs to be used while running in compose mode. To avoid affecting the original code, I moved the file to a different directory. There was an error on Github, and although I did a merge, the file was not updated. Therefore, I will update it manually.

This happens when there's conflict between your branch and your merged branch. You need to resolve the conflict between the branches.

sajith · 2024-02-06T13:38:33Z

sdx_controller/messaging/rpc_queue_consumer.py

 # subscribe to the corresponding queue
 SUB_QUEUE = os.environ.get("SUB_QUEUE")
-
+MQ_SRVC = os.environ.get("MQ_SRVC")


It looks like MQ_HOST was renamed to MQ_SRVC, but not with good reason. Everywhere else (environment files, tests, CI setup) we use MQ_HOST.

MQ_SRVC has the Docker service name value and MQ_HOST has the IP address. I suspect the test error is caused because adding authentication is not tested, and MQ_HOST is not used anymore.

Please continue to use MQ_HOST, and please make just the necessary changes to make password authentication work.

No problem! So, password authentication is currently working fine. I'll go ahead and bring back MQ_HOST, and if the test still doesn't go as planned, I'm happy to keep working on it. But I'd really appreciate your help in figuring out what's causing the issue and finding a solution to make the deployment happen. Thanks so much!

I'm looking into the test failures.

I think the problem is that not all environment variables are set when running the tests: especially MQ_SRVC (which should be MQ_HOST) in this case. It will also fail if MQ_USER or MQ_PASS is not set.

It would be nice to have some checks to see that all configuration items are set, but that is a separate issue (see #27) from this. We should definitely use some defaults in this PR -- like using RabbitMQ default username/password of guest/guest when no username and password environment variable is set, as I've stated in another review comment.

So: it looks like addressing the test failures should be straightforward: rename MQ_SRVC back to MQ_HOST, and use defaults for MQ_USER and MQ_PASS. PRs are tested against an RabbitMQ service that uses the default username and password.

In the meanwhile, could you set up a local development environment and try running the tests yourself?

You probably want to reset your PR branch, and cherry-pick the relevant commits to the one relevant file, like so:

$ git switch luisdev $ git fetch origin $ git reset --hard origin/main $ git cherry-pick 9ba4b92 c0d3459

Now, rename MQ_SRVC back to MQ_HOST, and use user/auth defaults. Also remove the time.sleep() calls. We should not add unnecessary latency, unless there is a good reason. And then update this PR with:

$ git push <remote> luisdev --force-with-lease

Obviously replace <remote> with whatever remote name that you are using.

coveralls · 2024-02-07T04:52:06Z

Pull Request Test Coverage Report for Build 7810077489

-37 of 79 (53.16%) changed or added relevant lines in 16 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.4%) to 37.787%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
swagger_server/controllers/connection_controller.py	2	4	50.0%
swagger_server/handlers/connection_handler.py	5	12	41.67%
swagger_server/main.py	0	28	0.0%

Totals
Change from base Build 7757392363:	-0.4%
Covered Lines:	516
Relevant Lines:	1222

💛 - Coveralls

sajith · 2024-03-28T14:12:58Z

Superseded by #223 and #250. Let us close this now.

Ubuntu and others added 6 commits June 19, 2023 19:44

worker process

4c40d0f

adding mq worker

6d38b46

Merge branch 'main' into validation

3b338b8

adding credentials to rabbitmq #197

9ba4b92

adding credentials to rabbitmq #197

6a05a47

adding authentication to rabbit mq

a6fc177

lmarinve requested review from YufengXin and sajith February 5, 2024 20:12

adding authentication to rabbit mq #221

c0d3459

congwang09 reviewed Feb 6, 2024

View reviewed changes

sajith requested changes Feb 6, 2024

View reviewed changes

lmarinve added 7 commits February 6, 2024 20:11

Adding pika authentication and sleep time to queue consumer #221

2c79d7f

adding authentication to rabbit mq #221

572f407

adding authentication to rabbit mq #221

44fbcf4

adding authentication to rabbit mq #221

d5a8947

adding authentication to rabbit mq #221

909e1bc

adding authentication to rabbit mq #221

0029574

adding credentials to rabbit mq #101

e57ca47

sajith closed this Mar 28, 2024

sajith deleted the luisdev branch March 28, 2024 14:13

lmarinve restored the luisdev branch April 1, 2024 19:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding pika authentication and sleep time to queue consumer #221

Adding pika authentication and sleep time to queue consumer #221

lmarinve commented Feb 5, 2024

congwang09 left a comment

congwang09 Feb 6, 2024 •

edited

Loading

congwang09 Feb 6, 2024

lmarinve Feb 6, 2024

congwang09 Feb 6, 2024

sajith Feb 6, 2024

lmarinve Feb 6, 2024

sajith Feb 6, 2024

congwang09 Feb 6, 2024

congwang09 Feb 6, 2024

congwang09 Feb 6, 2024

lmarinve Feb 6, 2024

sajith left a comment •

edited

Loading

sajith Feb 6, 2024

lmarinve Feb 6, 2024

congwang09 Feb 6, 2024

sajith Feb 6, 2024

lmarinve Feb 6, 2024

sajith Feb 6, 2024

lmarinve Feb 6, 2024

sajith Feb 6, 2024

sajith Feb 6, 2024 •

edited

Loading

sajith Feb 6, 2024

coveralls commented Feb 7, 2024

sajith commented Mar 28, 2024

Adding pika authentication and sleep time to queue consumer #221

Adding pika authentication and sleep time to queue consumer #221

Conversation

lmarinve commented Feb 5, 2024

congwang09 left a comment

Choose a reason for hiding this comment

congwang09 Feb 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sajith left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sajith Feb 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Feb 7, 2024

Pull Request Test Coverage Report for Build 7810077489

💛 - Coveralls

sajith commented Mar 28, 2024

congwang09 Feb 6, 2024 •

edited

Loading

sajith left a comment •

edited

Loading

sajith Feb 6, 2024 •

edited

Loading