mini-recall

A small and simple attempt to implement the Microsoft's new Recall feature.

This application has four main components:

• A python service that takes a screenshot of the PC every few seconds and stores in a folder.
• A python service that processes the screenshots by fetching the image embeddings and storing them in a vector database - qdrant
• A FastAPI api that serves endpoints to query the vector database and fetch the most related images for a given text query.
• A simple frontend that talks to the api for querying the images using text and displays the results.

This project uses the CLIP-ViT-g-14 model and open_clip for image and text embeddings. Check server.py to see how the embeddings are generated.

The results are not perfect and the model is not fine-tuned. The main goal of this project is to understand the working of the Recall feature and to build a simple prototype. As the type of images are all screenshots from a computer, the results are not very accurate. But are considerably good. The model can be fine-tuned on a dataset of screenshots to improve the results.

🚧 This project is a work in progress and is not yet complete.

Installation

Python Services

Creating a virtual environment is recommended.

git clone
cd mini-recall
pip install -r requirements.txt

Frontend

cd frontend
npm install

Qdrant

Follow the instructions in the qdrant documentation to setup qdrant.

Usage

Python Services

python capture_service.py
python processing_service.py
python server.py

Frontend

cd frontend
npm start

License

This project is licensed under the MIT License - see the LICENSE file for details.