New ask Hacker News story: JupyterHub in Kubernetes
JupyterHub in Kubernetes
2 by spicyramen | 2 comments on Hacker News.
Current scenario: Our company has been using Jupyter Lab Notebooks in their local workstations and we have a shared VM with GPUs in Google Cloud with GPUs for larger training jobs which we power on when needed. Background: Most of the experiments in Notebooks are very different, some data scientists use Pytorch, while others use TF. Sagemaker or Colab or Cloud Notebooks don't offer vertical scaling. For larger jobs (hours, days) the solution proposed by Google Cloud is to use AI Platform Training, but this requires for us to rewrite Notebooks. An we don't want to rewrite code in Notebook to dispatch it to a different system. One of the ideas we started looking is to use Notebooks in K8s using JupyterHub. Does anyone uses JupyerHub in production? We already have a dedicated k8s cluster and the expertise to manage it so I can think JupyerHub can be a good solution where each notebook runs in a pod that can be scale up/down Any pointers or comments about the experience are appreciated
2 by spicyramen | 2 comments on Hacker News.
Current scenario: Our company has been using Jupyter Lab Notebooks in their local workstations and we have a shared VM with GPUs in Google Cloud with GPUs for larger training jobs which we power on when needed. Background: Most of the experiments in Notebooks are very different, some data scientists use Pytorch, while others use TF. Sagemaker or Colab or Cloud Notebooks don't offer vertical scaling. For larger jobs (hours, days) the solution proposed by Google Cloud is to use AI Platform Training, but this requires for us to rewrite Notebooks. An we don't want to rewrite code in Notebook to dispatch it to a different system. One of the ideas we started looking is to use Notebooks in K8s using JupyterHub. Does anyone uses JupyerHub in production? We already have a dedicated k8s cluster and the expertise to manage it so I can think JupyerHub can be a good solution where each notebook runs in a pod that can be scale up/down Any pointers or comments about the experience are appreciated
Comments
Post a Comment