Do minInstances and maxInstances work?

[iyannsch]

After discussing session- and instance-pre-warming with @JonasHelming , I'd like to raise the question in this forum: Do the minInstances and maxInstances parameters in Kubernetes currently take effect? After setting the minInstances for an app to 2, I cannot verify any pod spawning.

Will the operator consider those values and spawn x instances before they are needed? How are those instances personalized then and at which time?

Thanks for your help
Yannik

[jfaltermeier]

We prototyped eagerly starting sessions as part of our initial Theia Cloud implementation. However, we currently only support the lazy start option, which does not honor minInstances, though it does respect the maxInstances parameter. This lazy start handler checks the current number of sessions before creating a deployment for new ones.

The development of the eager start option has been stalled because attaching a volume to an already running pod requires a restart, which negates the benefits of starting it early. I think this is/was a Docker limitation. So we need to find a solution for this before the eager start mode can be enabled again.

During the prototype phase, when a new session was requested by a user, we selected one of the unassigned running deployments and made changes to the ConfigMap that configures the OAuth2Proxy for this deployment. The proxy would pick up these changes, configuring itself to accept connections for the session's user. Before that, all connections were declined.

[iyannsch]

Thanks for you answer @jfaltermeier!
Could you elaborate a bit more on why you'd need to attach a volume during runtime and not eagerly during startup of the (yet un-personalized) container? Is it related to access rights?
If yes: I though they are checked purely outside of the actual container (on ingress/service level).

Looking forward to your response 👍 In case you got some pointers on where I can look/experiment around with, I'd also appreciate it

[jfaltermeier]

The challenge we're encountering is that Workspaces in TheiaCloud, serving as models for persistent storage, are user-specific. When a user initiates a session with their Workspace A, they expect to access the same data they interacted with during their previous session with Workspace A. Consequently, we cannot identify which storage to attach until the user logs in and requests a session for a specific workspace.

Here are some relevant code pointers:

We have two eager start handlers in the current codebase: EagerStartAppDefinitionAddedHandler and EagerStartSessionHandler.
Generally, handlers are bound via methods in AbstractTheiaCloudOperatorModule such as bindWorkspaceHandler(), bindAppDefinitionHandler(), and bindSessionHandler(). These handlers react on changes of the Theia Cloud custom resources.
Launching the operator with the --eagerStart option enables this mode.
The helm values allow setting operator.eagerStart to true. However, our documentation currently says the only supported value is false, and we have not tested this feature in recent releases. Thus, its functionality may be uncertain.
The EagerStartSessionHandler currently does not attempt to attach storage. You can compare this with LazySessionHandler to see how it handles storage attachments.

[JonasHelming]

Just to clarify: In a use case, where the user would start with an empty workspace or with a predefined content, there wouldn't be an issue with prewarming, correct? @jfaltermeier

[jfaltermeier]

Using a fresh, empty workspace should be possible; however, it would need adjustments concerning ownership of workspaces and cleanup processes, as this setup would be more like one-time use workspaces then.

Operating without any persistent storage (i.e., relying solely on in-memory storage) could lead to data loss if a Pod is terminated+restarted by Kubernetes.

Besides that additional new features introduced since our initial tests with the eager start, such as providing environment variables with sessions, may also be challenging. Unix processes often require a restart to recognize changes in environment variables, and we would have to check if changing the env of a Deployment is propagated to its Pod and does not require a restart.

Furthermore, the user activity monitoring, which shuts down idle pods, would need adaptions. This mechanism must be adapted to recognize and respond appropriately under the eager start conditions.