Jupyter-server-proxy setup for rstudio

JupyterLab
2

There’s a dedicated plugin for rstudio (that wraps jupyter-server-proxy):

Though note this bug with RStudio 1.4 which sounds like the problem you’re seeing: