Use AI models in terminal, VSCode and Obsidian locally, free and without a connection

Table of contents

• Use AI models in terminal, VSCode and Obsidian locally, free and without a connection
  • Table of contents
  • What is this?
  • Requirements
  • Installation
    • Run local api in docker
    • Fetch the models
    • Restart the api
    • Make your first request
  • Usage
    • In your terminal
      • Terminal Requirements
      • Install zsh_local_ai plugin
      • Using oh-my-zsh edit ~/.zshrc
      • Reload zsh
    • In VS Code
      • VS Code Requirements
      • Install Vscode and Continue Plugin
      • Configure extension
    • In your notes or markdown files
      • Notes Requirements
      • Install obsidian and text generator plugin
      • Setup text generator
  • License

What is this?

This is a simple api that runs locally and can be used to run ai models locally without internet connection and for free. It uses Hugging Face transformers library to load the models and FastAPI to serve the api. It also uses Docker to run the api in a container.

Requirements

• Docker
• Docker-compose
• Make

Installation

Run local api in docker

Wont demonize the process, so you can see the logs (Also debug mode is on)

make start

Fetch the models

This will take a while to download the models for the first time. Wait until this is done before making a request.

make get_models

Restart the api

Give it a few seconds to stop the container and start it again.

make restart

Make your first request

That will take a while to load the model for the first time

make test

Usage

In your terminal

Terminal Requirements

• Zsh
• Oh-my-zsh
• Python3

Install zsh_local_ai plugin

cp -r zsh-local-ai ~/.oh-my-zsh/custom/plugins/

Using oh-my-zsh edit ~/.zshrc

Where plugins=() add zsh-local-ai to the list Add a key binding to create_completion function

......
plugins=(....... zsh-local-ai .......)

bindkey '^X' create_completion

Reload zsh

source ~/.zshrc

Start typing the command or a comment and press ctrl+x to get the completion. Then press enter to run the command.

In VS Code

VS Code Requirements

• VS Code
• Continue - Claude, CodeLlama, GPT-4, and more extension

Install Vscode and Continue Plugin

Download and install VSCode, then from the extensions search and install the Continue plugin.

Configure extension

• At you vscode Toolbar click on the new Continue icon, then click on the gear icon at the bottom right to open the settings.

• In the models array add the following first

{
  "title": "Monday AI",
  "provider": "openai",
  "model": "gpt4all-j",
  "apiBase": "http://localhost:8085/v1/"
},

• Replace the "tabAutocompleteModel" Object with the following

  "tabAutocompleteModel": {
    "title": "Tab Autocomplete Model",
    "provider": "openai",
    "model": "gpt4all-j",
    "apiBase": "http://localhost:8085/v1/"
  },
  "tabAutocompleteOptions": {
    "useCopyBuffer": false,
    "maxPromptTokens": 400,
    "prefixPercentage": 0.5,
    "debounceDelay": 2000
  },

Save the config and now you can chat with Monday in the continue chat and use it as autocompletion..... it is slow and depends on your setup

In your notes or markdown files

Notes Requirements

• Obsidian
• Text Generator plugin

Install obsidian and text generator plugin

You can find obsidian here. In the settings, go to community plugins and install text generator.

Setup text generator

• In the settings under community plugins settings, select text generator. Then in LLM Providers select Custom and add the url of the local api. This will be http://localhost:8085/v1/chat/completions.

• Toggle advance mode and in Header section remove the bearer token header. Headers should look like that:

{
    "Content-Type": "application/json",
}

• In the Variables section, change the model to be gpt4all-j

• Lastly, enable CORS Bypass.

Now you can use the text generator plugin to generate text in your notes.

License

MIT