Open AI alternative

[avirajkhare00]

Hi guys. I don't have too much knowledge on LLMs. Is it possible to have own LLM running inside machine and replace OpenAI part with Llama or something else. I am noob in this area. Any references would be extremely helpful.

[srbhr]

Hi @avirajkhare00 the way you can integrate LLMs with Swirl is by using their Metapipe. But if you want alternative to OpenAI. Then I suggest that we can work on integrating Langchain with Swirl's results. Langchain can work with a lot of LLMs and other AI stuff altogether.
So, you can have a lot of LLMs like LLaMa 2, Vicuna, Falcon, etc.

[erikspears]

Hello @avirajkhare00 , and thanks for opening up a Discussion with us! I'd like to understand your question a bit better before we try to answer in more detail. When you say "OpenAI", are you referring to our integration with ChatGPT as a SearchProvider? Or were you asking about another aspect of LLM usage in the product in general? Thanks!

[avirajkhare00]

I would like to take an example of BERT
We do have powerful local machines(laptop, desktop) that can run this LLM.
Since it can run offline without the need of internet connection, how can swirl search fit in, in this case scenario?

[sid-swirl]

Hi @avirajkhare00 ... you'll need to create a new Swirl object that loads the BERT model. It should expose a similarity function, which should then be called by the CosineRelevancyPostResultProcessor.

Follow swirl/spacy.py as a starting point.

For example, you could make a swirl/bert.py that does something like this:

from transformers import pipeline, AutoModel, AutoTokenizer

import os
os.environ["OMP_NUM_THREADS"] = "1"

import torch

class BERT_model:
    def __init__(self, ner_model_name='dbmdz/bert-large-cased-finetuned-conll03-english', model_name='bert-base-multilingual-cased'):
        self.model = AutoModel.from_pretrained(model_name)
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)

    def sim(self, sentence1, sentence2):
        # Tokenize and encode sentences
        inputs1 = self.tokenizer(sentence1, return_tensors='pt', truncation=True, padding=True)
        inputs2 = self.tokenizer(sentence2, return_tensors='pt', truncation=True, padding=True)

        # Get embeddings
        with torch.no_grad():
            embeddings1 = self.model(**inputs1).last_hidden_state.mean(dim=1)
            embeddings2 = self.model(**inputs2).last_hidden_state.mean(dim=1)

        # Compute similarity (cosine similarity is often used for this)
        similarity = torch.nn.functional.cosine_similarity(embeddings1, embeddings2).item()
        
        return similarity

Then you would copy and create a newCosineRelevancyResultProcessor that calls BERT_model.sim() instead of spacy similarity. Note that I have not tried running this in Celery, there might be issues with memory etc, since the object may be large. Maybe start with a smaller model.

Ping me if you need any help with this.

[avirajkhare00]

Is it possible to give list of smaller models?

[sid-swirl]

This has a couple - https://huggingface.co/amine/bert-base-5lang-cased

What requirements do you have for languages? Cross-language similarity etc?

[DenisBY]

Is there any progress with using Swirl with, i.e., Ollama?

[sid-swirl]

Hi @DenisBY ! SWIRL Enterprise supports 10+ AIs including Ollama. Request a demo here: https://swirlaiconnect.com/schedule-a-demo