Kernels (Architecture)

Pre-requisites

• Please read Terminology

Overview

The core functionalities offered by the Kernels component are

• Kernel discovery and launching. More details in general overview and code architecture
  • Discovery. Responsible for finding Jupyter kernel specs on user's machine, connecting to remote jupyter servers and turning Python Environments into Jupyter specs.
  • Kernel launcher. Launching the selected Jupyter kernel with proper environment.
• Execution. Handling the messages between the extension and Jupyter kernel.
• Raw vs Jupyter kernels. There are two types of connections we create.

How does it work end to end

Below graphs is a bird view of where kernels are loaded from, how they are launched and what's being exposed once kernels are launched and connected:

• The kernel component can list all kernels found from multiple sources. The source of kernels limits how a kernel can be launched.
  • Globally installed kernel specs are started as separate processes by default and the extension will connect to it through zeromq.
    • If zeromq is not available/enabled, then we would attempt to launch jupyter notebook|lab and connect to the kernels through Jupyter Server API.
  • Jupyter servers created outside of the extension can be connected and communicated through Jupyter Server API.
  • Lastly, Python environments can be used as "kernel" (see details), we would install ipykernel to the Python environment and start the kernel process.
    • There might be Jupyter kernel specs installed in Python environments, they can be started like the globally installed kernel specs
    • Before starting kernels from Python environments, we need to activate the environment first to ensure the kernel process have all the required environment variables loaded
• Once the kernels are started, we can communicate with the kernels through the same library and protocol (Jupyter Messaging Protocol). The message handling component is also exposed publicly to allow features to be built on top, including
  • Update notebook cell outputs through NotebookController
  • Handle IPyWidgets
  • Handle Debugging
  • Exposed to 3rd party extensions