Bug 249936 - www/py-notebook: not working in jail, "Kernel starting, please wait..."
Summary: www/py-notebook: not working in jail, "Kernel starting, please wait..."
Status: Closed Not A Bug
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Many People
Assignee: Danilo G. Baio
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-09-27 12:36 UTC by Leonardo M. Ramé
Modified: 2020-11-07 19:46 UTC (History)
1 user (show)

See Also:
dbaio: maintainer-feedback+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Leonardo M. Ramé 2020-09-27 12:36:22 UTC
Hi, I'm trying to run py37-notebook-6.0.3_2 on a FreeBSD 12.1 Jail. I've been trying to make it work since more than a week ago without success.

I installed it first using `pkg install` and when I found this problem I tried with `pip3 install ....`, but both methods ends with the same problem.

After installing and making some changes in the config file `/home/jupyter/.jupyter/jupyter_notebook_config.py` I can open it from a web browser. The changes are these:

c.NotebookApp.ip ='*'
c.NotebookApp.open_browser = False
c.NotebookApp.notebook_dir  = '/tmp'
c.NotebookApp.port = 8888

I can point from another machine using `http://192.168.0.111:8888`, and get this:

So everything seems to be working...until I try do execute a simple calc like `1+1` or `print("hello world")`. When I do that and press <kbd>ctrl+r</kbd> there's a `Kernel starting, please wait...`, and after a while, in the server's console there's this:

[I 16:23:09.145 NotebookApp] Creating new notebook in 
/usr/local/lib/python3.7/site-packages/jupyter_client/connect.py:164: RuntimeWarning: Failed to set sticky bit on '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-67e1b082-cc4e-4b25-b5a5-a38d13fd717f.json': [Errno 79] Inappropriate file type or format: '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-67e1b082-cc4e-4b25-b5a5-a38d13fd717f.json'
Probably not a big deal, but runtime files may be cleaned up periodically.
  RuntimeWarning,
[I 16:23:10.143 NotebookApp] Kernel started: 67e1b082-cc4e-4b25-b5a5-a38d13fd717f
[W 16:23:10.192 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20200926161340 (192.168.0.110) 0.99ms referer=http://192.168.0.111:8888/notebooks/Untitled4.ipynb?kernel_name=python3
/usr/local/lib/python3.7/site-packages/jupyter_client/connect.py:164: RuntimeWarning: Failed to set sticky bit on '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-67e1b082-cc4e-4b25-b5a5-a38d13fd717f.json': [Errno 79] Inappropriate file type or format: '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-67e1b082-cc4e-4b25-b5a5-a38d13fd717f.json'
Probably not a big deal, but runtime files may be cleaned up periodically.
  RuntimeWarning,
[I 16:23:17.731 NotebookApp] Saving file at /Untitled3.ipynb
[W 16:23:31.238 NotebookApp] Replacing stale connection: 67e1b082-cc4e-4b25-b5a5-a38d13fd717f:f20bd23a2dc0450c8f9db79d7eb82a33
[W 16:23:54.305 NotebookApp] Replacing stale connection: 67e1b082-cc4e-4b25-b5a5-a38d13fd717f:f20bd23a2dc0450c8f9db79d7eb82a33
[W 16:24:10.227 NotebookApp] Timeout waiting for kernel_info reply from 67e1b082-cc4e-4b25-b5a5-a38d13fd717f
[I 16:24:10.234 NotebookApp] Starting buffering for 67e1b082-cc4e-4b25-b5a5-a38d13fd717f:f20bd23a2dc0450c8f9db79d7eb82a33
[I 16:24:10.235 NotebookApp] Restoring connection for 67e1b082-cc4e-4b25-b5a5-a38d13fd717f:f20bd23a2dc0450c8f9db79d7eb82a33


[jupyter@jupyterlab ~]$ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.0.3
qtconsole        : not installed
ipython          : 7.16.1
ipykernel        : 5.3.0
jupyter client   : 6.1.3
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : not installed
nbformat         : 5.0.7
traitlets        : 4.3.3

By searching on the web I found others have a similar issue with `tornado 6`, but this installation is running `py37-tornado-5.1.1` which is installed automatically by `pkg install notebook`.

Can anyone help me with this?
Comment 1 Danilo G. Baio freebsd_committer freebsd_triage 2020-09-27 13:35:19 UTC
Hi.

It seems you are using packages from the quarterly branch.

Could you try the latest repo? It has some updates on jupyter-notebook, ipython and traitlets.

More details here:
https://www.freebsd.org/doc/handbook/pkgng-intro.html#quarterly-latest-branch

If the issue still occurs, we'll need to investigate more.

And the quarterly repo will be updated soon, finishing this quarter in a few days.
Comment 2 Leonardo M. Ramé 2020-09-27 21:27:56 UTC
Hi Danilo, I created a new jail, with the latest repo, which containts py37-notebook vestion 6.1.4, instead of 6.0.3_2 and the problem is still there:

[I 18:23:42.510 NotebookApp] Serving notebooks from local directory: /tmp
[I 18:23:42.510 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 18:23:42.510 NotebookApp] http://jupyterlab:8888/
[I 18:23:42.510 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 18:23:46.730 NotebookApp] 404 GET /notebooks/Untitled.ipynb?kernel_name=python3 (192.168.0.110): No such file or directory: Untitled.ipynb
[W 18:23:46.750 NotebookApp] 404 GET /notebooks/Untitled.ipynb?kernel_name=python3 (192.168.0.110) 20.28ms referer=http://192.168.0.111:8888/tree?
[I 18:23:52.054 NotebookApp] Creating new notebook in 
/usr/local/lib/python3.7/site-packages/jupyter_client/connect.py:164: RuntimeWarning: Failed to set sticky bit on '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-b4649028-a681-47b3-83fb-0d1c22625c6f.json': [Errno 79] Inappropriate file type or format: '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-b4649028-a681-47b3-83fb-0d1c22625c6f.json'
Probably not a big deal, but runtime files may be cleaned up periodically.
  RuntimeWarning,
[I 18:23:53.134 NotebookApp] Kernel started: b4649028-a681-47b3-83fb-0d1c22625c6f, name: python3
[W 18:23:53.136 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20200927182342 (192.168.0.110) 0.89ms referer=http://192.168.0.111:8888/notebooks/Untitled.ipynb?kernel_name=python3
/usr/local/lib/python3.7/site-packages/jupyter_client/connect.py:164: RuntimeWarning: Failed to set sticky bit on '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-b4649028-a681-47b3-83fb-0d1c22625c6f.json': [Errno 79] Inappropriate file type or format: '/usr/home/jupyter/.local/share/jupyter/runtime/kernel-b4649028-a681-47b3-83fb-0d1c22625c6f.json'
Probably not a big deal, but runtime files may be cleaned up periodically.
  RuntimeWarning,
Comment 3 Danilo G. Baio freebsd_committer freebsd_triage 2020-09-27 22:08:53 UTC
Make a test, remove that config and start it directly (from jupyter homedir).

// Use the IP address from your lan

$ jupyter notebook --ip=192.168.15.231
[I 18:53:06.714 NotebookApp] Serving notebooks from local directory: /usr/home/notebook
[I 18:53:06.714 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 18:53:06.714 NotebookApp] http://192.168.15.231:8888/?token=7f91b4e60e6afead5020e4a9c40228598bb08afa5c32d2e7
[I 18:53:06.714 NotebookApp]  or http://127.0.0.1:8888/?token=7f91b4e60e6afead5020e4a9c40228598bb08afa5c32d2e7
[I 18:53:06.714 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 18:53:06.718 NotebookApp] No web browser found: could not locate runnable browser.
[C 18:53:06.718 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///usr/home/notebook/.local/share/jupyter/runtime/nbserver-92277-open.html
    Or copy and paste one of these URLs:
        http://192.168.15.231:8888/?token=7f91b4e60e6afead5020e4a9c40228598bb08afa5c32d2e7
     or http://127.0.0.1:8888/?token=7f91b4e60e6afead5020e4a9c40228598bb08afa5c32d2e7

[I 18:53:24.928 NotebookApp] 302 GET /?token=7f91b4e60e6afead5020e4a9c40228598bb08afa5c32d2e7 (192.168.X.X) 1.64ms

I've created a new jail and it seems fine.
Comment 4 Leonardo M. Ramé 2020-09-27 22:37:03 UTC
No, same issue:

$ jupyter notebook --ip=192.168.1.102 
[I 19:32:22.764 NotebookApp] Serving notebooks from local directory: /usr/home/jupyter
[I 19:32:22.764 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 19:32:22.764 NotebookApp] http://192.168.1.102:8888/
[I 19:32:22.764 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 19:32:22.767 NotebookApp] No web browser found: could not locate runnable browser.
[W 19:32:24.910 NotebookApp] 404 GET /api/kernels/12feb62d-d6f6-40ba-941b-baccd16304b1/channels?session_id=555d9586b20746e984b2f63194384499 (192.168.0.110): Kernel does not exist: 12feb62d-d6f6-40ba-941b-baccd16304b1
[W 19:32:24.930 NotebookApp] 404 GET /api/kernels/12feb62d-d6f6-40ba-941b-baccd16304b1/channels?session_id=555d9586b20746e984b2f63194384499 (192.168.0.110) 21.16ms referer=None

Important: my network is as this:

1. Client: 192.168.0.110
2. Host: 192.168.0.111
3. Jail: 192.168.1.102

Host routes port 8888 to jail using a pf rule.

My /etc/pf.conf:

# vim: set ft=pf
# /etc/pf.conf

#Declare the interfaces, Public IP, private subnet,
EXT_IF0 = "re0"
EXT_IF1 = "lo1"

IP_PUB="192.168.0.111"
NET_JAIL="192.168.1.0/24"
LAN_IP="192.168.0.1"
PSQL_JAIL_IP="192.168.1.101"
JUPYTERLAB_JAIL_IP="192.168.1.102"
nat pass on $EXT_IF1 from $NET_JAIL to any -> $LAN_IP
nat pass on $EXT_IF0 from $NET_JAIL to any -> $IP_PUB 
# PostgreSql Jail
rdr on $EXT_IF0 proto tcp from any to $IP_PUB port 5432 -> $PSQL_JAIL_IP
rdr on lo0 proto tcp from any to 127.0.0.1 port 5432 -> $PSQL_JAIL_IP
# JupyterLab Jail
rdr on $EXT_IF0 proto tcp from any to $IP_PUB port 8888 -> $JUPYTERLAB_JAIL_
Comment 5 Danilo G. Baio freebsd_committer freebsd_triage 2020-09-27 22:51:24 UTC
(In reply to Leonardo M. Ramé from comment #4)

so it seems network related...
BTW, using VNET here, it simplifies a lot.
Comment 6 Danilo G. Baio freebsd_committer freebsd_triage 2020-10-31 23:46:42 UTC
Hi Leonardo, I'm closing this as it is related with network/jail.
Regards.
Comment 7 Leonardo M. Ramé 2020-11-07 19:46:08 UTC
Hi Danilo, I write just to let you know that I created a VNET jail and JupyterLab works without issues there, so, as you pointed out, the problem is networking related.