Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support MinIO for file storage #365

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Support MinIO for file storage #365

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
662d796
feat: integrate MinIO for file storage
minai621 Oct 4, 2024
6ccd8d8
feat: Add storage module for file upload
minai621 Oct 12, 2024
5627a97
refactor: storage module to remove unnecessary credentials case by aw…
minai621 Oct 12, 2024
0825028
refactor: storage module update condition check logic to use file_upl…
minai621 Oct 14, 2024
cb403cd
feat: set default configuration values for Minio
minai621 Oct 16, 2024
322201c
refactor: update storage module to remove default value for AWS_REGION
minai621 Oct 16, 2024
a1f132b
refactor: remove unused StorageType enum
minai621 Oct 16, 2024
0d22ee0
refactor: update .env.development to set default values for Minio
minai621 Oct 16, 2024
47f41d1
Merge branch 'main' into support-minio
minai621 Nov 5, 2024
d38b9d2
feat: enhance file upload configuration for MinIO support
minai621 Nov 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions backend/.env.development
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,14 @@ LANGCHAIN_PROJECT=your_langsmith_project_name_here


# FILE_UPLOAD: Whether to enable file upload to storage
# Set to true if file upload is required.
# If set to false, AWS_S3_BUCKET_NAME is not required.
# Set to true if file upload is required.
FILE_UPLOAD=false
minai621 marked this conversation as resolved.
Show resolved Hide resolved
# AWS_S3_BUCKET_NAME: S3 Bucket name
# S3 or MinIO configuration
BUCKET_TYPE="S3 or MINIO"
minai621 marked this conversation as resolved.
Show resolved Hide resolved
# This is the name of the S3 Bucket
AWS_S3_BUCKET_NAME="your_s3_bucket_name"
BUCKET_NAME="your_bucket_name"
AWS_REGION="your_aws_region"
MINIO_ENDPOINT="your_minio_endpoint"
MINIO_ACCESS_KEY="your_minio_access_key"
MINIO_SECRET_KEY="your_minio_secret_key"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comments for these variables.
The format of the comments should be consistent with the others.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we have to set default value.
Note that users should be able to run CodePair with minimal configuration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devleejb
Regarding the default values, I think not having them won't cause any issues. As noted in the implementation, if the MINIO_ENDPOINT environment variable is not provided, the MinIO client won't be initialized or used. This ensures that MinIO is only executed when all required variables are properly configured.

For reference, please check the storage.module.ts file, where the MinIO client is only created when the necessary endpoint is provided.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand. However, I want the default upload options to be configured for MinIO. Also, I want the CodePair project to run only with the following commands:

docker-compose -f ./backend/docker/docker-compose-full.yml up -d
pnpm i
pnpm frontend dev

Therefore, we should set default values (This is just for the local environment. For production, we will provide other guidelines).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devleejb
sorry, I don't check this comment.
Minio buckets need to be created manually either via the web interface after the server starts or through a shell script.
In that case, how should the initial value for the bucket name be set?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then, can we create bucket using entrypoint in docker-compose?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devleejb
I checked that it's possible through the entry point, so I updated the implementation in this pull request.

28 changes: 27 additions & 1 deletion backend/docker/docker-compose-full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,21 @@ services:
LANGCHAIN_API_KEY: "your_langsmith_api_key_here"
LANGCHAIN_PROJECT: "your_langsmith_project_name_here"
FILE_UPLOAD: false
AWS_S3_BUCKET_NAME: "your_s3_bucket_name"
BUCKET_TYPE: "S3 or MINIO"
minai621 marked this conversation as resolved.
Show resolved Hide resolved
BUCKET_NAME: "your_s3_bucket_name_here"
MINIO_ENDPOINT: "your_minio_endpoint_here"
MINIO_ACCESS_KEY: "your_minio_access_key_here"
MINIO_SECRET_KEY: "your_minio_secret_key_here"
ports:
- "3000:3000"
depends_on:
- mongo
- minio
restart: unless-stopped
links:
- "mongo:mongo"
- "yorkie:yorkie"
- "minio:minio"

yorkie:
image: "yorkieteam/yorkie:latest"
Expand Down Expand Up @@ -59,3 +65,23 @@ services:
interval: 5s
timeout: 2s
retries: 20

minio:
image: minio/minio
ports:
- "9000:9000"
- "9001:9001"
volumes:
- minio_data:/data
environment:
MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
minai621 marked this conversation as resolved.
Show resolved Hide resolved
command: server --console-address ":9001" /data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3

volumes:
minio_data:
32 changes: 28 additions & 4 deletions backend/src/files/files.service.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { GetObjectCommand, PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { GetObjectCommand, PutObjectCommand, S3Client, S3ClientConfig } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import {
BadRequestException,
Expand All @@ -13,6 +13,7 @@ import * as htmlPdf from "html-pdf-node";
import * as MarkdownIt from "markdown-it";
import { PrismaService } from "src/db/prisma.service";
import { generateRandomKey } from "src/utils/functions/random-string";
import { StorageType } from "src/utils/types/storage.type";
import { CreateUploadPresignedUrlResponse } from "./types/create-upload-url-response.type";
import { ExportFileRequestBody, ExportFileResponse } from "./types/export-file.type";

Expand All @@ -25,7 +26,7 @@ export class FilesService {
private configService: ConfigService,
private prismaService: PrismaService
) {
this.s3Client = new S3Client();
this.s3Client = new S3Client(this.getStorageConfig());
minai621 marked this conversation as resolved.
Show resolved Hide resolved
this.markdown = new MarkdownIt({
html: true,
breaks: true,
Expand Down Expand Up @@ -55,7 +56,7 @@ export class FilesService {

const fileKey = `${workspace.slug}-${generateRandomKey()}.${contentType.split("/")[1]}`;
const command = new PutObjectCommand({
Bucket: this.configService.get("AWS_S3_BUCKET_NAME"),
Bucket: this.configService.get("BUCKET_NAME"),
minai621 marked this conversation as resolved.
Show resolved Hide resolved
Key: fileKey,
StorageClass: "INTELLIGENT_TIERING",
ContentType: contentType,
Expand All @@ -70,7 +71,7 @@ export class FilesService {
async createDownloadPresignedUrl(fileKey: string) {
try {
const command = new GetObjectCommand({
Bucket: this.configService.get("AWS_S3_BUCKET_NAME"),
Bucket: this.configService.get("BUCKET_NAME"),
Key: fileKey,
});
return getSignedUrl(this.s3Client, command, { expiresIn: 3600 });
Expand Down Expand Up @@ -125,4 +126,27 @@ export class FilesService {
fileName: `${fileName}.pdf`,
};
}

private getStorageConfig = (): S3ClientConfig => {
const bucketType: StorageType = this.configService.get("BUCKET_TYPE") || "S3";
const region = this.configService.get("AWS_REGION") || "us-east-1";
if (bucketType === "MINIO") {
const endpoint = this.configService.get("MINIO_ENDPOINT");
const accessKeyId = this.configService.get("MINIO_ACCESS_KEY");
const secretAccessKey = this.configService.get("MINIO_SECRET_KEY");
return {
region,
endpoint,
forcePathStyle: true,
credentials: {
accessKeyId,
secretAccessKey,
},
};
}

return {
region,
};
};
devleejb marked this conversation as resolved.
Show resolved Hide resolved
}
1 change: 1 addition & 0 deletions backend/src/utils/types/storage.type.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export type StorageType = "S3" | "MINIO";