Skip to content

Commit

Permalink
update handling of $JUPYTERHUB_SERVICE_URL when defined
Browse files Browse the repository at this point in the history
JupyterHub 2.0 uses $JUPYTERHUB_SERVICE_URL instead of `--port` CLI argument
  • Loading branch information
minrk committed Mar 8, 2024
1 parent 3f2ad16 commit 050ae92
Showing 1 changed file with 10 additions and 7 deletions.
17 changes: 10 additions & 7 deletions batchspawner/singleuser.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import sys
from runpy import run_path
from shutil import which
from urllib.parse import urlparse
from urllib.parse import urlparse, urlunparse

import requests
from jupyterhub.services.auth import HubAuth
Expand Down Expand Up @@ -35,12 +35,15 @@ def main(argv=None):

# Read the env var JUPYTERHUB_SERVICE_URL and replace port in the URL
# with free port that we found here
url = urlparse(os.environ.get("JUPYTERHUB_SERVICE_URL", ""))
# Updated URL. We are effectively passing the port arg via env var
if url.hostname:
os.environ["JUPYTERHUB_SERVICE_URL"] = (
f"{url.scheme}://{url.hostname}:{port}{url.path}"
)
# JUPYTERHUB_SERVICE_URL is added in JupyterHub 2.0
service_url_env = os.environ.get("JUPYTERHUB_SERVICE_URL", "")
if service_url_env:
url = urlparse(os.environ["JUPYTERHUB_SERVICE_URL"])
url = url._replace(netloc=f"{url.hostname}:{port}")
os.environ["JUPYTERHUB_SERVICE_URL"] = urlunparse(url)
else:
# JupyterHub < 2.0 specifies port on the command-line
sys.argv.append(f"--port={port}")

cmd_path = which(sys.argv[1])
sys.argv = sys.argv[1:]
Expand Down

1 comment on commit 050ae92

@wixaw
Copy link

@wixaw wixaw commented on 050ae92 Mar 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hello, I'm not sure that this works, because when I run the job the ports between the slurm node and the jupyter server seem to be different

log hub :

11:33:48 root@r9jupyter:jupyterhub# jupyterhub -f /etc/jupyterhub/jupyterhub_config.py --debug
[D 2024-03-19 11:33:56.944 JupyterHub application:908] Looking for /etc/jupyterhub/jupyterhub_config in /etc/jupyterhub
[D 2024-03-19 11:33:56.953 JupyterHub application:929] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[W 2024-03-19 11:33:56.963 JupyterHub app:674] Both bind_url and ip/port/base_url have been configured. JupyterHub.ip, JupyterHub.port, JupyterHub.base_url are deprecated in JupyterHub 0.9, please use JupyterHub.bind_url instead.
[I 2024-03-19 11:33:56.963 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2024-03-19 11:33:56.963 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.0.2
[I 2024-03-19 11:33:56.963 JupyterHub app:2889] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2024-03-19 11:33:56.963 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[D 2024-03-19 11:33:56.967 JupyterHub app:2816] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2024-03-19 11:33:56.967 JupyterHub app:1664] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[D 2024-03-19 11:33:56.967 JupyterHub app:1833] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2024-03-19 11:33:57.362 JupyterHub orm:984] Stamping empty database with alembic revision 0eee8c825d24
[I 2024-03-19 11:33:57.364 alembic.runtime.migration migration:216] Context impl SQLiteImpl.
[I 2024-03-19 11:33:57.364 alembic.runtime.migration migration:219] Will assume non-transactional DDL.
[I 2024-03-19 11:33:57.989 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2024-03-19 11:33:57.989 JupyterHub app:2104] Loading roles into database
[I 2024-03-19 11:33:58.031 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[D 2024-03-19 11:33:58.035 JupyterHub app:2343] Purging expired APITokens
[D 2024-03-19 11:33:58.037 JupyterHub app:2343] Purging expired OAuthCodes
[D 2024-03-19 11:33:58.038 JupyterHub app:2179] Loading role assignments from config
[D 2024-03-19 11:33:58.045 JupyterHub app:2502] Initializing spawners
[D 2024-03-19 11:33:58.046 JupyterHub app:2633] Loaded users:

[I 2024-03-19 11:33:58.046 JupyterHub app:2928] Initialized 0 spawners in 0.002 seconds
[I 2024-03-19 11:33:58.050 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2024-03-19 11:33:58.051 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2024-03-19 11:33:58.051 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
[W 2024-03-19 11:33:58.052 JupyterHub proxy:746] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2024-03-19 11:33:58.052 JupyterHub proxy:750] Starting proxy @ http://r9jupyter:8000/
[D 2024-03-19 11:33:58.052 JupyterHub proxy:751] Proxy cmd: ['configurable-http-proxy', '--ip', 'r9jupyter', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://r9jupyter:8081/hub/error', '--log-level', 'info']
[D 2024-03-19 11:33:58.056 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
11:33:58.291 [ConfigProxy] info: Proxying http://r9jupyter:8000 to (no default)
11:33:58.293 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-03-19 11:33:58.506 JupyterHub proxy:787] Proxy started and appears to be up
[D 2024-03-19 11:33:58.509 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
11:33:58.525 [ConfigProxy] info: 200 GET /api/routes
[I 2024-03-19 11:33:58.526 JupyterHub app:3178] Hub API listening on http://r9jupyter:8081/hub/
[D 2024-03-19 11:33:58.526 JupyterHub proxy:389] Fetching routes to check
[D 2024-03-19 11:33:58.526 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
11:33:58.529 [ConfigProxy] info: 200 GET /api/routes
[D 2024-03-19 11:33:58.530 JupyterHub proxy:392] Checking routes
[I 2024-03-19 11:33:58.530 JupyterHub proxy:477] Adding route for Hub: / => http://r9jupyter:8081
[D 2024-03-19 11:33:58.530 JupyterHub proxy:880] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
11:33:58.533 [ConfigProxy] info: Adding route / -> http://r9jupyter:8081
11:33:58.534 [ConfigProxy] info: Route added / -> http://r9jupyter:8081
11:33:58.534 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-03-19 11:33:58.535 JupyterHub app:3245] JupyterHub is now running at http://r9jupyter:8000/
[D 2024-03-19 11:33:58.536 JupyterHub app:2852] It took 1.595 seconds for the Hub to start
[W 2024-03-19 11:34:03.763 JupyterHub base:422] Invalid cookie token
[I 2024-03-19 11:34:03.764 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@x.x.4.106) 8.65ms
[W 2024-03-19 11:34:03.776 JupyterHub base:422] Invalid cookie token
[I 2024-03-19 11:34:03.801 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2F (@x.x.4.106) 26.50ms
[D 2024-03-19 11:34:04.238 JupyterHub log:191] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@x.x.4.106) 3.42ms
[W 2024-03-19 11:34:06.954 JupyterHub base:422] Invalid cookie token
[D 2024-03-19 11:34:07.071 JupyterHub roles:281] Assigning default role to User wvincent
[I 2024-03-19 11:34:07.079 JupyterHub roles:238] Adding role user for User: wvincent
[D 2024-03-19 11:34:07.332 JupyterHub roles:281] Assigning default role to User wvincent
[W 2024-03-19 11:34:07.343 JupyterHub base:422] Invalid cookie token
[D 2024-03-19 11:34:07.343 JupyterHub base:591] Setting cookie for wvincent: jupyterhub-hub-login
[D 2024-03-19 11:34:07.343 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2024-03-19 11:34:07.344 JupyterHub base:837] User logged in: wvincent
[I 2024-03-19 11:34:07.344 JupyterHub log:191] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (wvincent@x.x.4.106) 393.19ms
[D 2024-03-19 11:34:07.357 JupyterHub base:299] Recording first activity for <User(wvincent 0/1 running)>
[D 2024-03-19 11:34:07.468 JupyterHub user:431] Creating <class 'wrapspawner.wrapspawner.ProfilesSpawner'> for wvincent:
[W 2024-03-19 11:34:07.471 JupyterHub spawner:170]
    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.

    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.

[I 2024-03-19 11:34:07.475 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (wvincent@x.x.4.106) 118.97ms
[D 2024-03-19 11:34:07.488 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2024-03-19 11:34:07.488 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2024-03-19 11:34:07.488 JupyterHub pages:209] Serving options form for wvincent
[I 2024-03-19 11:34:07.494 JupyterHub log:191] 200 GET /hub/spawn (wvincent@x.x.4.106) 7.69ms
[D 2024-03-19 11:34:08.581 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2024-03-19 11:34:08.581 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2024-03-19 11:34:08.582 JupyterHub pages:257] Triggering spawn with supplied form options for wvincent
[D 2024-03-19 11:34:08.582 JupyterHub base:961] Initiating spawn for wvincent
[D 2024-03-19 11:34:08.583 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-03-19 11:34:08.583 JupyterHub base:970] 0 active servers
[I 2024-03-19 11:34:08.854 JupyterHub provider:659] Creating oauth client jupyterhub-user-wvincent
[D 2024-03-19 11:34:09.042 JupyterHub user:794] Calling Spawner.start for wvincent
[I 2024-03-19 11:34:09.047 JupyterHub batchspawner:281] Spawner script options: {'account': '', 'cluster': '', 'epilogue': '', 'gres': '', 'homedir': '/home/wvincent', 'host': 'r9jupyter.domain.fr:443', 'keepvars': 'PATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_ROOT_DIR,JUPYTERHUB_DEFAULT_URL,JUPYTERHUB_DEBUG,USER,HOME,SHELL', 'keepvars_extra': '', 'memory': '', 'ngpus': '', 'nprocs': '', 'options': '', 'partition': '24CPUNodes', 'prologue': '', 'qos': '', 'queue': '', 'reservation': '', 'runtime': '12:00:00', 'srun': 'srun', 'username': 'wvincent', 'cmd': 'batchspawner-singleuser jupyterhub-singleuser --debug', 'profile': '24CPUNodes'}
[I 2024-03-19 11:34:09.047 JupyterHub batchspawner:282] Spawner submitting command: sudo -E -u wvincent sbatch --parsable
[D 2024-03-19 11:34:09.047 JupyterHub batchspawner:283] Spawner submitting script:
    #!/bin/bash
    #SBATCH --time=12:00:00
    #SBATCH --output=/home/wvincent/jupyterlab-%j.output
    #SBATCH --error=/home/wvincent/jupyterlab-%j.error
    #SBATCH --job-name=jupyterlab
    #SBATCH --export=PATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_ROOT_DIR,JUPYTERHUB_DEFAULT_URL,JUPYTERHUB_DEBUG,USER,HOME,SHELL
    #SBATCH --mail-user=william.vincent@domain.fr
    #SBATCH --ntasks=1
    #SBATCH --partition=24CPUNodes
    #SBATCH --chdir=/home/wvincent

    set -x

    # jupyter-singleuser anticipates that environment will be dropped during sudo, however
    # it is retained by batchspawner. The XDG_RUNTIME_DIR variable must be unset to force a
    # fallback, otherwise a permissions error occurs when starting the notebook.
    # https://github.com/jupyter/notebook/issues/1318
    export SINGULARITYENV_JUPYTERHUB_API_TOKEN=$JUPYTERHUB_API_TOKEN
    export SINGULARITYENV_XDG_RUNTIME_DIR=$HOME/.singularity-jupyter-run
    export SINGULARITYENV_CONTAINER_PATH=/apps/containerCollections/CUDA12/pytorch2-will.sif

    #env

    echo "# debug"

    srun singularity run --env JUPYTER_PORT=0 $SINGULARITYENV_CONTAINER_PATH env
    echo "# debug 2 "
    srun singularity exec --env JUPYTER_PORT=0 --bind $PWD:/run/user $SINGULARITYENV_CONTAINER_PATH batchspawner-singleuser jupyterhub-singleuser --debug > jupyter.debug 2>&1
    echo "# fin"

[D 2024-03-19 11:34:09.048 JupyterHub batchspawner:284] Spawner submitting environment: {'PATH': '/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin', 'LANG': 'en_US.UTF-8', 'JUPYTERHUB_API_TOKEN': '95c28a59792a4ec38755d2cffb156df2', 'JPY_API_TOKEN': '95c28a59792a4ec38755d2cffb156df2', 'JUPYTERHUB_CLIENT_ID': 'jupyterhub-user-wvincent', 'JUPYTERHUB_HOST': '', 'JUPYTERHUB_OAUTH_CALLBACK_URL': '/user/wvincent/oauth_callback', 'JUPYTERHUB_OAUTH_SCOPES': '["access:servers!server=wvincent/", "access:servers!user=wvincent"]', 'JUPYTERHUB_OAUTH_ACCESS_SCOPES': '["access:servers!server=wvincent/", "access:servers!user=wvincent"]', 'JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES': '[]', 'JUPYTERHUB_USER': 'wvincent', 'JUPYTERHUB_SERVER_NAME': '', 'JUPYTERHUB_API_URL': 'http://r9jupyter:8081/hub/api', 'JUPYTERHUB_ACTIVITY_URL': 'http://r9jupyter:8081/hub/api/users/wvincent/activity', 'JUPYTERHUB_BASE_URL': '/', 'JUPYTERHUB_SERVICE_PREFIX': '/user/wvincent/', 'JUPYTERHUB_SERVICE_URL': 'http://0.0.0.0:0/user/wvincent/', 'JUPYTERHUB_ROOT_DIR': '~', 'JUPYTERHUB_DEFAULT_URL': '/lab', 'JUPYTERHUB_DEBUG': '1', 'USER': 'wvincent', 'HOME': '/home/wvincent', 'SHELL': '/bin/bash'}
[I 2024-03-19 11:34:09.103 JupyterHub batchspawner:287] Job submitted. output: 123
[D 2024-03-19 11:34:09.104 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:09.166 JupyterHub batchspawner:428] Job 123 still pending
[W 2024-03-19 11:34:09.585 JupyterHub base:183] Rolling back dirty objects IdentitySet([<Server(:0)>])
[I 2024-03-19 11:34:09.591 JupyterHub log:191] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/wvincent?_xsrf=[secret] (wvincent@x.x.4.106) 1009.71ms
[D 2024-03-19 11:34:09.601 JupyterHub scopes:877] Checking access to /hub/spawn-pending/wvincent via scope servers
[D 2024-03-19 11:34:09.601 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/wvincent via servers
[I 2024-03-19 11:34:09.603 JupyterHub pages:398] wvincent is pending spawn
[I 2024-03-19 11:34:09.606 JupyterHub log:191] 200 GET /hub/spawn-pending/wvincent?_xsrf=[secret] (wvincent@x.x.4.106) 7.87ms
[D 2024-03-19 11:34:09.669 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:09.751 JupyterHub scopes:877] Checking access to /hub/api/users/wvincent/server/progress via scope read:servers
[D 2024-03-19 11:34:09.752 JupyterHub scopes:690] Argument-based access to /hub/api/users/wvincent/server/progress via read:servers
[D 2024-03-19 11:34:10.242 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:10.804 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:11.374 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:11.946 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:12.534 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:13.098 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[D 2024-03-19 11:34:13.145 JupyterHub base:299] Recording first activity for <APIToken('95c2...', user='wvincent', client_id='jupyterhub')>
[I 2024-03-19 11:34:13.260 JupyterHub log:191] 200 POST /hub/api/batchspawner (wvincent@x.x.102.71) 122.89ms
[I 2024-03-19 11:34:13.263 JupyterHub batchspawner:462] Notebook server job 123 started at r9nc-24-1:56621
[D 2024-03-19 11:34:13.381 JupyterHub spawner:1384] Polling subprocess every 30s
[I 2024-03-19 11:34:16.494 JupyterHub log:191] 200 GET /hub/api (@x.x.102.71) 0.88ms
[D 2024-03-19 11:34:16.509 JupyterHub scopes:877] Checking access to /hub/api/users/wvincent/activity via scope users:activity
[D 2024-03-19 11:34:16.509 JupyterHub scopes:690] Argument-based access to /hub/api/users/wvincent/activity via users:activity
[D 2024-03-19 11:34:16.512 JupyterHub users:874] Activity for user wvincent: 2024-03-19T10:34:14.098227Z
[D 2024-03-19 11:34:16.513 JupyterHub users:892] Activity on server wvincent/: 2024-03-19T10:34:14.098227Z
[I 2024-03-19 11:34:16.624 JupyterHub log:191] 200 POST /hub/api/users/wvincent/activity (wvincent@x.x.102.71) 119.30ms
[D 2024-03-19 11:34:18.584 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u wvincent squeue -h -j 123 -o '%T %B'
[W 2024-03-19 11:34:18.693 JupyterHub base:1122] User wvincent is slow to become responsive (timeout=10)
[D 2024-03-19 11:34:18.694 JupyterHub base:1127] Expecting server for wvincent at: http://r9jupyter:0/user/wvincent/

port on slurm node :

LISTEN    0         4096               0.0.0.0:40003              0.0.0.0:*        users:(("srun",pid=3618,fd=3),("srun",pid=3617,fd=3))
LISTEN    0         4096               0.0.0.0:35393              0.0.0.0:*        users:(("srun",pid=3617,fd=11))
LISTEN    0         4096               0.0.0.0:45881              0.0.0.0:*        users:(("srun",pid=3617,fd=8))
LISTEN    0         128                0.0.0.0:43769              0.0.0.0:*        users:(("batchspawner-si",pid=3643,fd=10))

Please sign in to comment.