Added new seedfarmer module for Question and Answering Gen AI constru…

…ct (#81)

* seedformer module for qaRagUsingAppSync

* Added deployspec and integrated llm-rag modules in deployment yaml

* Added IAC for Question and Answering module

* removed unwanted comments from the code

* added READ ME and other minor corrections to paths

* Added change description to CHANGELOG.md

* Updated repository README.md with qna-rag module details

* Corrected type in repository readme

* updated urlib version

* Added tests and upgdated read me

* removed print statement and added open search security group as input parameter

* updated example to include open search security group id

* Fixed unit test failure

* Moved qna-rag manifest to examples

* Updated Q&A Rag ingestion module to include ingestion modules as well

* deleted storage yaml from examples

* updated readme and added mypy validation fixes

* Updated read me to include ingestion construct and add existing S3 bucket as optional input to RAG module

* Added Optional type for input asset bucket

* removed the unnecessary comment

* Addressed review comments - Updated IDF open search module version, Added Port as configurable parameter

---------

Co-authored-by: kukushking <[email protected]>

CHANGELOG.md

@@ -38,6 +38,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - added `sagemaker-model-package-promote-pipeline` module.
 - added `sagemaker-hugging-face-endpoint` module
 - added `hf_import_models` template to import hugging face models
+- added `qna-rag` module
 
 ### **Changed**
 

README.md

@@ -42,10 +42,11 @@ See deployment steps in the [Deployment Guide](DEPLOYMENT.md).
 
 ### FMOps Modules
 
-| Type                                                                                                            | Description                                                     |
-|-----------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
-| [SageMaker JumpStart Foundation Model Endpoint Module](modules/fmops/sagemaker-jumpstart-fm-endpoint/README.md) | Creates an endpoint for a SageMaker JumpStart Foundation Model. |
-| [SageMaker Hugging Face Foundation Model Endpoint Module](modules/fmops/sagemaker-hugging-face-endpoint/README.md) | Creates an endpoint for a SageMaker Hugging Face Foundation Model. |
+| Type                                                                                                               | Description                                                                                                                             |
+|--------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
+| [SageMaker JumpStart Foundation Model Endpoint Module](modules/fmops/sagemaker-jumpstart-fm-endpoint/README.md)    | Creates an endpoint for a SageMaker JumpStart Foundation Model.                                                                         |
+| [SageMaker Hugging Face Foundation Model Endpoint Module](modules/fmops/sagemaker-hugging-face-endpoint/README.md) | Creates an endpoint for a SageMaker Hugging Face Foundation Model.                                                                      |
+| [AppSync Knowledge Base Ingestion and Question and Answering RAG Module](modules/fmops/qna-rag/README.md)          | Creates an Graphql endpoint for ingestion of data and and use ingested as knowledge base for a Question and Answering model using RAG.  |
 
 
 ### MWAA Modules

examples/manifests/deployment.yaml

@@ -14,6 +14,7 @@ groups:
     path: manifests/sagemaker-model-package-group-modules.yaml
   - name: promote-models
     path: manifests/sagemaker-model-package-promote-pipeline-modules.yaml
+
 targetAccountMappings:
   - alias: primary
     accountId:

manifests/fmops-qna-rag/deployment.yaml

@@ -0,0 +1,20 @@
+name: mlops-qna-rag
+toolchainRegion: us-east-1
+forceDependencyRedeploy: true
+groups:
+  - name: networking
+    path: manifests/fmops-qna-rag/networking-modules.yaml
+  - name: storage
+    path: manifests/fmops-qna-rag/storage-modules.yaml
+  - name: qna-rag
+    path: manifests/fmops-qna-rag/qna-rag-modules.yaml
+
+targetAccountMappings:
+  - alias: primary
+    accountId:
+      valueFrom:
+        envVariable: PRIMARY_ACCOUNT
+    default: true
+    regionMappings:
+      - region: us-east-1
+        default: true

manifests/fmops-qna-rag/networking-modules.yaml

@@ -0,0 +1,6 @@
+name: networking
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.3.0&depth=1
+targetAccount: primary
+parameters:
+  - name: internet-accessible
+    value: True

manifests/fmops-qna-rag/qna-rag-modules.yaml

@@ -0,0 +1,24 @@
+name: qna-rag
+path: modules/fmops/qna-rag
+parameters:
+  - name: cognito-pool-id
+    #Replace below value with valid congnito pool id
+    value: us-east-1_XXXXX
+  - name: os-domain-endpoint
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: opensearch
+        key: OpenSearchDomainEndpoint
+  - name: os-security-group-id
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: opensearch
+        key: OpenSearchSecurityGroupId
+  - name: vpc-id
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: VpcId

manifests/fmops-qna-rag/storage-modules.yaml

@@ -0,0 +1,22 @@
+---
+name: opensearch
+path: git::https://github.com/awslabs/idf-modules.git//modules/storage/opensearch?ref=release/1.7.0&depth=1
+targetAccount: primary
+targetRegion: us-east-1
+parameters:
+  - name: encryption-type
+    value: SSE
+  - name: retention-type
+    value: RETAIN
+  - name: vpc-id
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: VpcId
+  - name: private-subnet-ids
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: PrivateSubnetIds

manifests/qna-rag-modules.yaml

@@ -0,0 +1,24 @@
+name: qna-rag
+path: modules/fmops/qna-rag
+parameters:
+  - name: cognito-pool-id
+    #Replace below value with valid congnito pool id
+    value: us-east-1_XXXXX
+  - name: os-domain-endpoint
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: opensearch
+        key: OpenSearchDomainEndpoint
+  - name: os-security-group-id
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: opensearch
+        key: OpenSearchSecurityGroupId
+  - name: vpc-id
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: VpcId

manifests/storage-modules.yaml

@@ -51,3 +51,25 @@ parameters:
     value: 30
   - name: removal-policy
     value: DESTROY
+---
+name: opensearch
+path: git::https://github.com/awslabs/idf-modules.git//modules/storage/opensearch?ref=release/1.7.0&depth=1
+targetAccount: primary
+targetRegion: us-east-1
+parameters:
+  - name: encryption-type
+    value: SSE
+  - name: retention-type
+    value: RETAIN
+  - name: vpc-id
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: VpcId
+  - name: private-subnet-ids
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: PrivateSubnetIds

manifests/uber-deployment.yaml

@@ -22,6 +22,8 @@ groups:
     path: manifests/mwaa-modules.yaml
   - name: mwaa-dags
     path: manifests/mwaa-dag-modules.yaml
+  - name: qna-rag
+    path: manifests/qna-rag-modules.yaml
 targetAccountMappings:
   - alias: primary
     accountId: