From bd1642f9ef807699f6a534d48a5b501dbb2828c5 Mon Sep 17 00:00:00 2001 From: Eli Nudler Date: Tue, 3 Oct 2023 21:56:18 +0300 Subject: [PATCH] Deployed e6a25ff with MkDocs version: 1.5.3 --- CNAME | 1 - loading/index.html | 1 + search/search_index.json | 2 +- sitemap.xml.gz | Bin 127 -> 127 bytes 4 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 CNAME diff --git a/CNAME b/CNAME deleted file mode 100644 index 5f0fb85..0000000 --- a/CNAME +++ /dev/null @@ -1 +0,0 @@ -docs.gpohackathon.com \ No newline at end of file diff --git a/loading/index.html b/loading/index.html index 1c27584..c7d5b03 100644 --- a/loading/index.html +++ b/loading/index.html @@ -575,6 +575,7 @@

Loading the files from SFTP to ja

After a successful response, those file that we loaded will exist in Jarvis, and we will be able to ask question about them. More on that on the next chapter.

+

supported file types: the supported file types are: pdf, csv, docx, xlsx, pptx, md, tsx, txt

collection: By choosing different collection in the API you can isolate different docs, you will be able to specify the collection on your prompts

diff --git a/search/search_index.json b/search/search_index.json index a84c02e..f77bf4e 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Welcome to Jarvis Docs","text":"

Welcome to Jarvis documentation. Jarvis is an abstraction layer, built for GPO 2023 Gen AI Hackathon.

Jarvis mission is to create an easy-to-use API that expose different foundation models offered by AWS Bedrock, To be consumed easily and efficiently to enable the hackathon teams focus on their solution rather than implementing common patterns in the Gen AI world.

Jarvis supports 3 types of functionalities:

  1. Zero shot prompts that are passed to the foundation model by your choice.
  2. Ability to send prompts on private GPO data that is uploaded prior to the prompt (RAG Pattern).
  3. Ability to have a chat conversation with the foundation model where Jarvis will persist the chat history for you.

In addition to that, Jarvis supports the upload of private GPO data that will be stored safely on the GPO AWS account and will be enabled to receive queries (RAG + Chat)

More information about these APIs and how to upload private data can be found in this documentation site.

"},{"location":"RAG/","title":"RAG","text":""},{"location":"RAG/#rag-pattern","title":"RAG Pattern","text":"

While zero-shot is nice and can give you a good head start, the biggest benefit of using Jarvis on top of Bedrock is Jarvis' ability to use embeddings and the RAG pattern.

So far what we used zero shot part to ask a simple question, but we can further its capabilities by providing a text document in the prompt.

As you can see this is very limited functionality because you will have to find the relevant document by your self and the document size is also limited to the max tokens the FM supports.

"},{"location":"RAG/#rag-retrieval-augmented-generation","title":"RAG - Retrieval-Augmented Generation","text":"

\"RAG\" in the context of AI often refers to the \"Retrieval-Augmented Generation\" model. This is a hybrid model that combines retrieval and generation techniques to improve the performance of natural language processing tasks.

The traditional approach to language models involves training them to generate text from scratch based on the input they receive. However, sometimes it's more effective to combine generation with retrieval of existing information. This is where RAG models come in.

RAG models consist of two main components:

Retrieval Component: This part of the model retrieves relevant information or passages from a large dataset of text. It's like having a search engine that retrieves relevant snippets of text based on a query.

Generation Component: Once the relevant information is retrieved, the generation component takes over. It generates the final output, incorporating the retrieved information to create coherent and contextually relevant responses.

RAG models are particularly useful for tasks that require a combination of knowledge retrieval and creative generation, such as question answering, content summarization, and more. They leverage the strengths of both techniques to produce high-quality outputs.

Let's go over what we see here and start with phase 2

"},{"location":"RAG/#uploading-files-2","title":"Uploading files (2)","text":"

Jarvis supports uploading files and taking care of everything that happens. Jarvis provides the abilities to upload files from SFTP, confluence and webpages. When you upload files through Jarvis API, Jarvis will do the following: 1. Get the file (SFTP/Confluence/Webpages) 2. Split the docs into chunks 3. Invoke Amazon titan embeddings model to create a vector from the doc chunk 4. Store the vector created by the Amazon titan embeddings model inside Postgres with pgvector support

SPANISH SUPPORTED: Amazon titan embeddings model support working on the spanish language as well, which means you can upload docs in spanish and ask question about them in spanish

"},{"location":"RAG/#prompt-with-rag-13","title":"Prompt with RAG (1+3)","text":"

When jarvis is being asked a question using the prompt API, Jarvis will create a vector from the question and will find for us the most similar vectors in the Postgres DB. only then Jarvis will invoke bedrock model together with the docs it found as the context.

For more information about RAG and vectors jump to GPO confluence

"},{"location":"api/","title":"API Reference","text":""},{"location":"api/#api-reference","title":"API Reference","text":"

Jarvis API

"},{"location":"bedrock/","title":"Bedrock","text":""},{"location":"bedrock/#bedrock","title":"Bedrock","text":"

Bedrock is a new service that as of writing this line (25.9.23) is still in beta stage and not available to AWS customers.

Bedrock is a product by AWS that accelerates development of generative AI applications using FMs through an API, without managing infrastructure. Currently, Bedrock provides access to foundation model from Anthropic, AI21, AWS Titan family and Stability AI. Bedrock also enables us to privately customize FMs using our organization's data.

In the hackathon, we chose to expose those foundation models through Jarvis.

Model Name Max Tokens Model Id Notes Classification Claude v2 12k anthropic.claude-v2 Anthropic's most powerful model, which excels at a wide range of tasks from sophisticated dialogue and creative content generation to detailed instruction following. Text generation, Conversational Claude Instant v1.1 9k anthropic.claude-instant-v1 A faster and cheaper yet still very capable model, which can handle a range of tasks including casual dialogue, text analysis, summarization, and document question-answering. Text generation, Conversational Jurassic-2 Mid 8k ai21.j2-mid Jurassic-2 Mid is AI21\u2019s mid-sized model, carefully designed to strike the right balance between exceptional quality and affordability. Jurassic-2 Mid can be applied to any language comprehension or generation task including question answering, summarization, long-form copy generation, advanced information extraction and many others. Text, Classification, Insert/edit, Math Jurassic-2 Ultra 8k Jurassic-2 Ultra Jurassic-2 Ultra is AI21\u2019s most powerful model offering exceptional quality. Apply Jurassic-2 Ultra to complex tasks that require advanced text generation and comprehension. Popular use cases include question answering, summarization, long-form copy generation, advanced information extraction, and more. Text, Classification, Insert/edit Titan Text Large 8k amazon.titan-tg1-large Titan Text is a generative large language model (LLM) for tasks such as summarization, text generation (for example, creating a blog post), classification, open-ended Q&A, and information extraction. Text generation, Code generation, Instruction following

As you can see, each model has different token size and they fit a different usecase; you may need to play with your hackathon product to find the right model for the job. When using the API, you will be able to change the FM that is in use by playing with the model_id.

NOTE: 1 token is equal to 3/4 of a word or 4 chars .

"},{"location":"bedrock/#why-bedrock-and-not-chatgpt","title":"Why Bedrock and not chatGPT?","text":"

One of the biggest advantages of bedrock over other foundation models as chatGPT or Bard is that Bedrock allows us to access to multiple foundation models as noted above. Moreover, the most critical aspect that gave Bedrock winning edge is that it allows us to use our private data as context to the foundation model. More info on this can be found in the RAG part.

"},{"location":"chat/","title":"Chat","text":""},{"location":"chat/#chat-rag","title":"Chat & RAG","text":"

In the preceding section, we covered prompts with RAG.

Now, let's explore how the Jarvis API can be utilized for chatting with RAG in this section.

NOTE: Chat is supported for anthropic.claude-v2 model only

Through the website loader, we uploaded two distinct blog posts. One of these posts dived into debeizum, a CDC tool that is widely used in GPO engineering. The FM are stateless and don't know anything about the previous question we send them. Jarvis adds this functionality by persisting the previous questions and passing them to the model once again on every new question being asked.

The structure of the chat object is as follows:

 \"chat\": {\n    \"conversation_id\": \"21389712387\", // The conversation_id to based on when asking the question\n    \"k_history\": 10, // The k last history messages (request&answer) to send to the model as part of the context\n    \"return_history\": true, // Return the history messages used for answer from the LLM in the response\n    \"rephrase_question\": false // Whether to use LLM to rephrase the question based on the history before trying to retrieve documents\n}\n

Let's start our chat about that article that was written after the FM finished its training

curl --location --request POST 'https://api.paymentsos.com/hackathon-ai/chat' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 5000,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n     },\n    \"rag\": {\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\",\n        \"return_source_documents\": false\n      },\n    \"chat\": {\n        \"conversation_id\": \"222222222222\",\n        \"k_history\": 10,\n        \"return_history\": true\n      }\n    }'\n

And the result will be:

{\n  \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n  \"result\": \" Based on the context provided, there are a few potential reasons a Debezium connector can lose its binlog offset:\\n\\n- The connector was down for an extended period of time, during which the MySQL binlog rotated. When the connector comes back up, it tries to resume from its last recorded offset, but that offset no longer exists in the current binlog.\\n\\n- The connector's configuration does not have heartbeat messages enabled. Heartbeat messages periodically store the connector's latest offset, protecting against offset loss on connector restart.\\n\\n- There is a misconfiguration or bug that causes the connector to not properly store its latest offset in Kafka. This can lead to the offset being lost when the connector is restarted.\\n\\n- If Kafka offsets topic retention is too short, offset messages can be deleted before the connector has a chance to resume from them.\\n\\nSo in summary, the main reasons are binlog rotation during an outage, lack of heartbeat messages, improper offset storage, and short retention of offset messages. Enabling heartbeat messages and configuring appropriate retention is key to avoiding offset loss.\",\n  \"source_documents\": null,\n  \"chat_history\": []\n}\n

Now, let's ask another question using what we learned from the first one:

curl --location --request POST 'https://api.paymentsos.com/hackathon-ai/chat' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n  \"question\": \"How can i solve the first potential reason?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 5000,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\",\n        \"return_source_documents\": false\n    },\n    \"chat\": {\n        \"conversation_id\": \"222222222222\",\n        \"k_history\": 10,\n        \"return_history\": true\n}'\n

And the answer is:

{\n  \"question\": \"How can i solve the first potential reasons?\",\n  \"result\": \" Based on the context provided, here are a few potential ways to solve the issue of a Debezium connector losing its binlog offset due to binlog rotation:\\n\\n1. Increase the binlog retention period in MySQL to be longer than the maximum expected downtime. This will ensure the binlog containing the connector's last offset is still available when the connector restarts.\\n\\n2. Configure the connector to also capture changes from a heartbeat table. This will update the connector's offset regularly even if there are no changes to the main table being captured. \\n\\n3. When restarting the connector after downtime, manually provide the last known good offset the connector had before going down. This can be obtained from the Kafka offsets topic.\\n\\n4. Use MySQL GTIDs to track replication positions instead of binlog offsets. GTIDs are persisted in MySQL and not subject to rotation.\\n\\n5. Set up MySQL replication from the primary to a secondary and have the connector read the binlogs from the secondary instead. This prevents binlog rotation from affecting the connector.\\n\\n6. Increase the frequency of connector offset flushing to Kafka to minimize potential data loss in case of failure.\\n\\nThe key is to ensure the connector's last known offset is still available or can be reconstructed after any downtime event. Let me know if you need any clarification or have additional questions!\",\n  \"source_documents\": null,\n  \"chat_history\": [\n    {\n      \"content\": \"What are the reasons a debezium connector loses its binlog offset?\",\n      \"additional_kwargs\": {},\n      \"example\": false\n    },\n    {\n      \"content\": \" Based on the context provided, there are a few potential reasons a Debezium connector can lose its binlog offset:\\n\\n- The connector was down for an extended period of time, during which the MySQL binlog rotated. When the connector comes back up, it tries to resume from its last recorded offset, but that offset no longer exists in the current binlog.\\n\\n- The connector's configuration does not have heartbeat messages enabled. Heartbeat messages periodically store the connector's latest offset, protecting against offset loss on connector restart.\\n\\n- There is a misconfiguration or bug that causes the connector to not properly store its latest offset in Kafka. This can lead to the offset being lost when the connector is restarted.\\n\\n- If Kafka offsets topic retention is too short, offset messages can be deleted before the connector has a chance to resume from them.\\n\\nSo in summary, the main reasons are binlog rotation during an outage, lack of heartbeat messages, improper offset storage, and short retention of offset messages. Enabling heartbeat messages and configuring appropriate retention is key to avoiding offset loss.\",\n      \"additional_kwargs\": {},\n      \"example\": false\n    }\n  ]\n}\n

We observe that the chat history, which was utilized to feed the LLM, returns in the response under \"chat_history.\"

\"chat_history\": [\n{\n    \"content\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"additional_kwargs\": {},\n    \"example\": false\n},\n{\n    \"content\": \" Based on the context provided, there are a few potential reasons a Debezium connector can lose its binlog offset:\\n\\n- The connector was down for an extended period of time, during which the MySQL binlog rotated. When the connector comes back up, it tries to resume from its last recorded offset, but that offset no longer exists in the current binlog.\\n\\n- The connector's configuration does not have heartbeat messages enabled. Heartbeat messages periodically store the connector's latest offset, protecting against offset loss on connector restart.\\n\\n- There is a misconfiguration or bug that causes the connector to not properly store its latest offset in Kafka. This can lead to the offset being lost when the connector is restarted.\\n\\n- If Kafka offsets topic retention is too short, offset messages can be deleted before the connector has a chance to resume from them.\\n\\nSo in summary, the main reasons are binlog rotation during an outage, lack of heartbeat messages, improper offset storage, and short retention of offset messages. Enabling heartbeat messages and configuring appropriate retention is key to avoiding offset loss.\",\n    \"additional_kwargs\": {},\n    \"example\": false\n}\n]\n
"},{"location":"creds/","title":"Onboarding your team","text":"

Follow the next steps in order to gain access to Jarvis API and the SFTP folder for uploading sensitive or company related data to power up your AI game.

"},{"location":"creds/#prerequisites","title":"Prerequisites","text":""},{"location":"creds/#keybase-username","title":"Keybase username","text":"

Obtain a valid Keybase username. We will share the credentials to Jarvis securely with this user.

"},{"location":"creds/#ssh-keys","title":"SSH keys","text":""},{"location":"creds/#team-name","title":"Team name","text":"

Solve one of programming hardest challenges, and decide on a name to represent your team in the competition! :)

"},{"location":"creds/#signing-up","title":"Signing up","text":"
  1. Go to: Gen AI Hackathon 2023 Teams in OneDrive.
  2. Create a new folder, name it with your team name.
  3. Upload to your new folder the ssh public key.
  4. Create keybase.txt file and add the Keybase username to the file.
  5. You can follow the convention in the example team folder.
"},{"location":"creds/#sharing-credentials","title":"Sharing credentials","text":"

Our magic behind the scenes will setup your access privileges, and the keybase username will be sent the following details: 1. app-id & private-key values used for authentication with Jarvis API. 2. SFTP username - used together with your generated RSA private key to connect to SFTP and upload files.

Good luck :)

"},{"location":"loading/","title":"Loading Documents","text":"

Jarvis supports loading documents from 3 data sources: 1. SFTP 2. Confluence 3. Webpages

"},{"location":"loading/#sftp","title":"SFTP","text":"

Your team will not only receive a private-key and app-id for the API, but it will also receive a set of credentials to use the SFTP server. Private key Username Password Address

Once you connect to the sftp, you will have your own folder where you can upload your files. A recommended SFTP client for mac can be Cyberduck

in this example we uploaded multiple files to the sftp folder

NOTE: At this point in time, those file are not yet known to Jarvis, and not stored in Postgres as vectors. They are only stored in the SFTP.

"},{"location":"loading/#loading-the-files-from-sftp-to-jarvis","title":"Loading the files from SFTP to jarvis","text":"

Jarvis is connected to the sftp and is able via API to get a trigger and upload specific file from the SFTP to Postgres. Prior to this, jarvis will call Amazon titan embedding model and create a vector as we learned in the RAG part. To do that:

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"sftp\",\n        \"files\": [\"mastercard.docx\"]\n    },\n    \"collection\": \"hackathon\"\n}'\n

After a successful response, those file that we loaded will exist in Jarvis, and we will be able to ask question about them. More on that on the next chapter.

collection: By choosing different collection in the API you can isolate different docs, you will be able to specify the collection on your prompts

"},{"location":"loading/#confluence","title":"Confluence","text":"

Jarvis is also directly integrated to our confluence (https://gpo-engineering.atlassian.net/wiki) You can use Jarvis API to stream confluence pages and space in one API call.

in this example we will publish the whole space but limit to 20 pages

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"confluence\",\n        \"space\": \"HUB\",\n        \"max_pages\": 20\n    },\n    \"collection\": \"confluence\"\n}'\n

and in this example we will choose to load only specific page ids from confluence

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"confluence\",\n        \"space\": \"HUB\",\n        \"page_ids\": [\"114327634\", \"830275595\"]\n    },\n    \"collection\": \"confluence\"\n}'\n

NOTE: Uploading multiple pages from confluence can take some time, and you may get a timeout, we suggest starting with limited number of page_ids. At any case, if there is a timeout from the client, Jarvis will continue uploading the files.

"},{"location":"loading/#webpages","title":"Webpages","text":"

Uploading a webpage is simple as that: 

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replamce' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"web\",\n        \"urls\": [\"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\", \"https://medium.com/payu-engineering/strengthen-the-platforms-security-with-capture-the-flag-aafc4ee12c65\"]\n    },\n    \"collection\": \"website-hackathon\"\n}'\n
"},{"location":"privacy/","title":"Privacy","text":""},{"location":"privacy/#data-privacy","title":"Data privacy","text":"

if you decide to allow teams to build models based on real personal data stored by PayU you must consider this two points.

"},{"location":"privacy/#preparation-of-training-data","title":"Preparation of training data","text":"

Any data science activities should not be done on raw data but on data which was deidentified (or even anonymized) as far as possible (e.g. by removing names, surnames, addresses, hashing of card data, etc.). If a team intends to use any real user data, the scope of the data used and the manner of deidentification should be agreed between you and the relevant team. As we discussed, there is small number of teams that plans to work on PayU personal data to create their models, so the consultation should not be too engaging.

"},{"location":"privacy/#safe-environment-for-cooperation","title":"Safe environment for cooperation","text":"

Hackathon data science activities should be done in a dedicated safe environment allowing participants to share information and discuss ideas. The tools used by participants should ensure that any information is secure, accessed only by verified users and has appropriate controls limiting export of data. Shortly after the event any personal data should be deleted.

Also, please remember that any AI projects should also be assessed from a fairness and ethics perspective, especially with respect to bias control, transparency, explainability, etc. However, given that these models will not be used in production but removed after the hackathon, in this case we can resign from such additional assessments.

"},{"location":"prompt-context/","title":"Prompts with RAG Pattern","text":""},{"location":"prompt-context/#using-rag","title":"Using RAG","text":"

We discussed how to load Jarvis with documents from different sources in the previous chapter. Let's see how we can use Jarvis API to ask questions about these documents.

Using the website loader, we uploaded two different blog posts. One of the blog posts discussed debeizum, a CDC tool used in GPO engineering.

Let's ask a question about that article that was written after the FM finished its training

The API using RAG is similar to the previous used API with zero shot, the difference is the addition of the rag object.

\n{\n \"return_source_documents\": true, //if true response will contain the docs that the answer is based on\n \"search_type\": \"similarity\", // see API reference for the options.\n \"k\": 5, // how many vectors should be retrieved from the db.\n \"collection\": \"hackathon\" //on which collection of documents look for the vectors - this must be the same collection where doc was loaded.\n}\n
curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: a4b90687-616f-4173-9774-af3cc6bd86bc' \\\n--header 'app-id: com.payu.3ds-latam-brazil' \\\n--header 'x-zooz-app-name: com.payu.3ds-latam-brazil' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 200,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"return_source_documents\": true,\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\"\n    }\n}'\n

This will be our response

{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"result\": \" Here are some common reasons a Debezium connector can lose its binlog offset:\\n\\n- MySQL server restart - This causes the binlog to be rotated, so if the connector misses this event, it will be looking for the old binlog position that no longer exists.\\n\\n- Connector restart - If the connector goes down uncleanly (crash, kill -9, etc), it may not have time to commit the latest offset to Kafka. Upon restart, it will resume from the last committed offset, losing any uncommitted transactions.\\n\\n- Unclean connector shutdown - Similar to above, if the connector is killed or crashes, any uncommitted offsets are lost.\\n\\n- Heartbeats disabled - The heartbeat feature periodically sends dummy events to Kafka to update the offset. If disabled, long periods of inactivity can cause the connector to lose its place.\\n\\n- Kafka retention issues - If Kafka purges messages or offsets stored in __consumer_offsets before the\",\n    \"source_documents\": [\n        {\n            \"page_content\": \"was among the early adopters of Debezium, which led us to face several challenges throughout the process. However, over time, we grew more confident and gained a deeper understanding of the necessary steps we had to make in order to improve our Debezium implementation.We encountered a few minor issues, often originating from misconfigured connectors. However, the most important topic we\u201a\u00c4\u00f4ll discuss is when a connector loses its binlog offset.As mentioned earlier, every connector can be\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"was among the early adopters of Debezium, which led us to face several challenges throughout the process. However, over time, we grew more confident and gained a deeper understanding of the necessary steps we had to make in order to improve our Debezium implementation.We encountered a few minor issues, often originating from misconfigured connectors. However, the most important topic we\u201a\u00c4\u00f4ll discuss is when a connector loses its binlog offset.As mentioned earlier, every connector can be\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"from two tables \u201a\u00c4\u00ee the original table but also changes of the debezium_heartbeat table.With heartbeat set to occur once an hour, we can be certain that the connector\u201a\u00c4\u00f4s offset will remain up-to-date \u201a\u00c4\u00ee This is due to the connector capturing changes from both tables, effectively storing the latest offset out of both within the binlog.Our connector configuration now looks as such in v2.3.0.{   \\\"name\\\":\\\"debezium-packages-connector-sandbox\\\",\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"from two tables \u201a\u00c4\u00ee the original table but also changes of the debezium_heartbeat table.With heartbeat set to occur once an hour, we can be certain that the connector\u201a\u00c4\u00f4s offset will remain up-to-date \u201a\u00c4\u00ee This is due to the connector capturing changes from both tables, effectively storing the latest offset out of both within the binlog.Our connector configuration now looks as such in v2.3.0.{   \\\"name\\\":\\\"debezium-packages-connector-sandbox\\\",\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"it debezium_connect_offsets.As seen in the Kafka UI screenshot below, within the specific topic dedicated to Debezium offsets, we can see that the most recent offset message for the packages_sandbox connector is listed as 1435109 (pos).Offset message of \u201a\u00c4\u00fapackages\u201a\u00c4\u00f9 connector, storing its latest recorded offsetIt is crucial to understand the following:A binlog contains changes from all the tables within MySQL, not only from the tables we\u201a\u00c4\u00f4ve configured the connector to capture changes\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        }\n    ]\n}\n

as mentioned above, we will get the source_documeants in the response to help us debug why we got the answer we got and an answer to our question.

"},{"location":"prompt-context/#prompt-template","title":"Prompt Template","text":"

One of the features many FM supports is sending instructions to the model to answer your questions. Most of you saw chatGPT prompts that asks for a brief answer as a pirate. While we can send these instructions to the FM we are working with, this instructions will go to RAG pattern as well as they are part of the questions and will affect the vectors that are found.

To solve this we are happy to introduce the prompt_template:

see this request for example:

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"prompt_template\": \"Answer the following questions as best you can, but speaking as a pirate might speak. Use lots of \\\"Arg\\\"s.\\nBegin\\n\\nHuman: {question}\\n\\nContext: {context}\\n\\nAssistant:\",\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 200,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"return_source_documents\": true,\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\"\n    }\n}'\n

And the response:

{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"result\": \" Arr matey, there be a few reasons a Debezium connector could be losin' its place in the binlog:\\n\\nFirst, if the MySQL server be restarted, that wipes the binlog clean and the connector won't know where it left off. So you gotta be careful not to restart the server if ye can help it, else the connector will start over from the beginning.\\n\\nSecond, network issues can cause the connector to lose its connection. If it can't keep talkin' to the MySQL server, it may miss some binlog events and lose track of the latest position. \\n\\nAnd third, sometimes the connector offset gets out o' sync if Debezium's heartbeat don't match up with the actual binlog position. That's why it be important to keep an eye on both and make sure they line up proper.\\n\\nBut don't be despairin'! There be ways to make sure your connector don't lose\",\n}\n

And if we take it more seriously, lets try this: 

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: a4b90687-616f-4173-9774-af3cc6bd86bc' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"prompt_template\": \"Answer the  briefly in bullets and as you were a sales person.\\nBegin\\n\\nHuman: {question}\\n\\nContext: {context}\\n\\nAssistant:\",\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 1000,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"return_source_documents\": true,\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\"\n    }\n}'\n

we will get this

{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"result\": \" Here are some common reasons a Debezium connector can lose its binlog offset:\\n\\n- MySQL server is restarted - This causes the binlog to be rotated, so the connector loses its place\\n\\n- Connector is stopped for too long - If the connector is down for longer than the MySQL binary log expiration period, the logs it needs may be deleted\\n\\n- Heartbeat configuration issue - If heartbeats are not configured properly, the connector may not be updating its offset frequently enough\\n\\n- Kafka retention issues - If Kafka is not retaining the offset topic long enough, offset messages may be deleted \\n\\n- Connector configuration error - An error in the connector configuration, such as the database or table whitelist, can cause offset issues\\n\\n- MySQL GTID mode not enabled - Debezium requires GTID mode to accurately track binlog positions \\n\\n- DB snapshot followed by purge - If a snapshot is followed by a BINLOG PURGE, the connector's recorded offset may be purged\\n\\n- DB admin intervention - An admin may have manually purged logs, inadvertently deleting offsets \\n\\n- Kafka Connect rebalance - Offsets may not be committed properly during a Connect cluster rebalance\\n\\nSo in summary - restarting MySQL, long connector downtime, Kafka retention, Connect errors, and admin actions on the db can all lead to lost offsets. Proper connector setup and config can mitigate many issues.\",}\n
"},{"location":"simple/","title":"Zero shot prompt","text":""},{"location":"simple/#zero-shot-prompt","title":"Zero shot prompt","text":"

Zero-Shot Prompting: If you\u2019ve interacted with an LLM-powered chatbot before, you\u2019ve likely already used zero-shot prompting unwittingly. Zero-shot prompting entails relying solely on an LLM\u2019s pre-trained information to answer a given user prompt.

To create your first request to Jarvis you will need a private-key and an app-id that will be provided by the hackathon team. This are your authentication keys, and they should be used by your team and remain private.

An example of a simple zero shot prompt to jarvis looks like this:

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: replace-me' \\\n--header 'app-id: replace-me' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"question\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 1000,\n        \"temperature\": 1,\n        \"top_k\": 250,\n        \"top_p\": 0.999\n    }\n}'\n

and this is the answer you will get:

{\n    \"query\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"answer\": \" Python is currently one of the most popular programming languages.\"\n}\n

Now lets go over all the params we had in this http request question - This is the prompt that will be sent to the FM stop_sequences - A set of chars that will stop the response from the FM like \\n for new line * name - This is the model id from the - Bedrock supported models max_tokens - The maximum tokens that will be in the answer - (1 token equal to 3/4 words) temperature - Read here top_k - Read here top_p - Read here

Playing with the request body: different models and different params like temperature, top_k and top_p will give you different answers, play with them and find the right tool for the job.

As an example, lets ask FM from AI21 the same question

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: replace-me' \\\n--header 'app-id: replace-me' \\\n--header 'Content-Type: application/json' \\\n--data '{\n\"question\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"ai21.j2-mid\",\n        \"max_tokens\": 1000,\n        \"temperature\": 1,\n        \"top_k\": 250,\n        \"top_p\": 0.999\n    }\n}'\n

and this is the answer you will get:

{\n    \"query\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"answer\": \" According to recent surveys, JavaScript is currently the most popular programming language among developers.\"\n}\n

As you can see different models trained and tuned on different sets of data can return various responses.

"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Welcome to Jarvis Docs","text":"

Welcome to Jarvis documentation. Jarvis is an abstraction layer, built for GPO 2023 Gen AI Hackathon.

Jarvis mission is to create an easy-to-use API that expose different foundation models offered by AWS Bedrock, To be consumed easily and efficiently to enable the hackathon teams focus on their solution rather than implementing common patterns in the Gen AI world.

Jarvis supports 3 types of functionalities:

  1. Zero shot prompts that are passed to the foundation model by your choice.
  2. Ability to send prompts on private GPO data that is uploaded prior to the prompt (RAG Pattern).
  3. Ability to have a chat conversation with the foundation model where Jarvis will persist the chat history for you.

In addition to that, Jarvis supports the upload of private GPO data that will be stored safely on the GPO AWS account and will be enabled to receive queries (RAG + Chat)

More information about these APIs and how to upload private data can be found in this documentation site.

"},{"location":"RAG/","title":"RAG","text":""},{"location":"RAG/#rag-pattern","title":"RAG Pattern","text":"

While zero-shot is nice and can give you a good head start, the biggest benefit of using Jarvis on top of Bedrock is Jarvis' ability to use embeddings and the RAG pattern.

So far what we used zero shot part to ask a simple question, but we can further its capabilities by providing a text document in the prompt.

As you can see this is very limited functionality because you will have to find the relevant document by your self and the document size is also limited to the max tokens the FM supports.

"},{"location":"RAG/#rag-retrieval-augmented-generation","title":"RAG - Retrieval-Augmented Generation","text":"

\"RAG\" in the context of AI often refers to the \"Retrieval-Augmented Generation\" model. This is a hybrid model that combines retrieval and generation techniques to improve the performance of natural language processing tasks.

The traditional approach to language models involves training them to generate text from scratch based on the input they receive. However, sometimes it's more effective to combine generation with retrieval of existing information. This is where RAG models come in.

RAG models consist of two main components:

Retrieval Component: This part of the model retrieves relevant information or passages from a large dataset of text. It's like having a search engine that retrieves relevant snippets of text based on a query.

Generation Component: Once the relevant information is retrieved, the generation component takes over. It generates the final output, incorporating the retrieved information to create coherent and contextually relevant responses.

RAG models are particularly useful for tasks that require a combination of knowledge retrieval and creative generation, such as question answering, content summarization, and more. They leverage the strengths of both techniques to produce high-quality outputs.

Let's go over what we see here and start with phase 2

"},{"location":"RAG/#uploading-files-2","title":"Uploading files (2)","text":"

Jarvis supports uploading files and taking care of everything that happens. Jarvis provides the abilities to upload files from SFTP, confluence and webpages. When you upload files through Jarvis API, Jarvis will do the following: 1. Get the file (SFTP/Confluence/Webpages) 2. Split the docs into chunks 3. Invoke Amazon titan embeddings model to create a vector from the doc chunk 4. Store the vector created by the Amazon titan embeddings model inside Postgres with pgvector support

SPANISH SUPPORTED: Amazon titan embeddings model support working on the spanish language as well, which means you can upload docs in spanish and ask question about them in spanish

"},{"location":"RAG/#prompt-with-rag-13","title":"Prompt with RAG (1+3)","text":"

When jarvis is being asked a question using the prompt API, Jarvis will create a vector from the question and will find for us the most similar vectors in the Postgres DB. only then Jarvis will invoke bedrock model together with the docs it found as the context.

For more information about RAG and vectors jump to GPO confluence

"},{"location":"api/","title":"API Reference","text":""},{"location":"api/#api-reference","title":"API Reference","text":"

Jarvis API

"},{"location":"bedrock/","title":"Bedrock","text":""},{"location":"bedrock/#bedrock","title":"Bedrock","text":"

Bedrock is a new service that as of writing this line (25.9.23) is still in beta stage and not available to AWS customers.

Bedrock is a product by AWS that accelerates development of generative AI applications using FMs through an API, without managing infrastructure. Currently, Bedrock provides access to foundation model from Anthropic, AI21, AWS Titan family and Stability AI. Bedrock also enables us to privately customize FMs using our organization's data.

In the hackathon, we chose to expose those foundation models through Jarvis.

Model Name Max Tokens Model Id Notes Classification Claude v2 12k anthropic.claude-v2 Anthropic's most powerful model, which excels at a wide range of tasks from sophisticated dialogue and creative content generation to detailed instruction following. Text generation, Conversational Claude Instant v1.1 9k anthropic.claude-instant-v1 A faster and cheaper yet still very capable model, which can handle a range of tasks including casual dialogue, text analysis, summarization, and document question-answering. Text generation, Conversational Jurassic-2 Mid 8k ai21.j2-mid Jurassic-2 Mid is AI21\u2019s mid-sized model, carefully designed to strike the right balance between exceptional quality and affordability. Jurassic-2 Mid can be applied to any language comprehension or generation task including question answering, summarization, long-form copy generation, advanced information extraction and many others. Text, Classification, Insert/edit, Math Jurassic-2 Ultra 8k Jurassic-2 Ultra Jurassic-2 Ultra is AI21\u2019s most powerful model offering exceptional quality. Apply Jurassic-2 Ultra to complex tasks that require advanced text generation and comprehension. Popular use cases include question answering, summarization, long-form copy generation, advanced information extraction, and more. Text, Classification, Insert/edit Titan Text Large 8k amazon.titan-tg1-large Titan Text is a generative large language model (LLM) for tasks such as summarization, text generation (for example, creating a blog post), classification, open-ended Q&A, and information extraction. Text generation, Code generation, Instruction following

As you can see, each model has different token size and they fit a different usecase; you may need to play with your hackathon product to find the right model for the job. When using the API, you will be able to change the FM that is in use by playing with the model_id.

NOTE: 1 token is equal to 3/4 of a word or 4 chars .

"},{"location":"bedrock/#why-bedrock-and-not-chatgpt","title":"Why Bedrock and not chatGPT?","text":"

One of the biggest advantages of bedrock over other foundation models as chatGPT or Bard is that Bedrock allows us to access to multiple foundation models as noted above. Moreover, the most critical aspect that gave Bedrock winning edge is that it allows us to use our private data as context to the foundation model. More info on this can be found in the RAG part.

"},{"location":"chat/","title":"Chat","text":""},{"location":"chat/#chat-rag","title":"Chat & RAG","text":"

In the preceding section, we covered prompts with RAG.

Now, let's explore how the Jarvis API can be utilized for chatting with RAG in this section.

NOTE: Chat is supported for anthropic.claude-v2 model only

Through the website loader, we uploaded two distinct blog posts. One of these posts dived into debeizum, a CDC tool that is widely used in GPO engineering. The FM are stateless and don't know anything about the previous question we send them. Jarvis adds this functionality by persisting the previous questions and passing them to the model once again on every new question being asked.

The structure of the chat object is as follows:

 \"chat\": {\n    \"conversation_id\": \"21389712387\", // The conversation_id to based on when asking the question\n    \"k_history\": 10, // The k last history messages (request&answer) to send to the model as part of the context\n    \"return_history\": true, // Return the history messages used for answer from the LLM in the response\n    \"rephrase_question\": false // Whether to use LLM to rephrase the question based on the history before trying to retrieve documents\n}\n

Let's start our chat about that article that was written after the FM finished its training

curl --location --request POST 'https://api.paymentsos.com/hackathon-ai/chat' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 5000,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n     },\n    \"rag\": {\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\",\n        \"return_source_documents\": false\n      },\n    \"chat\": {\n        \"conversation_id\": \"222222222222\",\n        \"k_history\": 10,\n        \"return_history\": true\n      }\n    }'\n

And the result will be:

{\n  \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n  \"result\": \" Based on the context provided, there are a few potential reasons a Debezium connector can lose its binlog offset:\\n\\n- The connector was down for an extended period of time, during which the MySQL binlog rotated. When the connector comes back up, it tries to resume from its last recorded offset, but that offset no longer exists in the current binlog.\\n\\n- The connector's configuration does not have heartbeat messages enabled. Heartbeat messages periodically store the connector's latest offset, protecting against offset loss on connector restart.\\n\\n- There is a misconfiguration or bug that causes the connector to not properly store its latest offset in Kafka. This can lead to the offset being lost when the connector is restarted.\\n\\n- If Kafka offsets topic retention is too short, offset messages can be deleted before the connector has a chance to resume from them.\\n\\nSo in summary, the main reasons are binlog rotation during an outage, lack of heartbeat messages, improper offset storage, and short retention of offset messages. Enabling heartbeat messages and configuring appropriate retention is key to avoiding offset loss.\",\n  \"source_documents\": null,\n  \"chat_history\": []\n}\n

Now, let's ask another question using what we learned from the first one:

curl --location --request POST 'https://api.paymentsos.com/hackathon-ai/chat' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n  \"question\": \"How can i solve the first potential reason?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 5000,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\",\n        \"return_source_documents\": false\n    },\n    \"chat\": {\n        \"conversation_id\": \"222222222222\",\n        \"k_history\": 10,\n        \"return_history\": true\n}'\n

And the answer is:

{\n  \"question\": \"How can i solve the first potential reasons?\",\n  \"result\": \" Based on the context provided, here are a few potential ways to solve the issue of a Debezium connector losing its binlog offset due to binlog rotation:\\n\\n1. Increase the binlog retention period in MySQL to be longer than the maximum expected downtime. This will ensure the binlog containing the connector's last offset is still available when the connector restarts.\\n\\n2. Configure the connector to also capture changes from a heartbeat table. This will update the connector's offset regularly even if there are no changes to the main table being captured. \\n\\n3. When restarting the connector after downtime, manually provide the last known good offset the connector had before going down. This can be obtained from the Kafka offsets topic.\\n\\n4. Use MySQL GTIDs to track replication positions instead of binlog offsets. GTIDs are persisted in MySQL and not subject to rotation.\\n\\n5. Set up MySQL replication from the primary to a secondary and have the connector read the binlogs from the secondary instead. This prevents binlog rotation from affecting the connector.\\n\\n6. Increase the frequency of connector offset flushing to Kafka to minimize potential data loss in case of failure.\\n\\nThe key is to ensure the connector's last known offset is still available or can be reconstructed after any downtime event. Let me know if you need any clarification or have additional questions!\",\n  \"source_documents\": null,\n  \"chat_history\": [\n    {\n      \"content\": \"What are the reasons a debezium connector loses its binlog offset?\",\n      \"additional_kwargs\": {},\n      \"example\": false\n    },\n    {\n      \"content\": \" Based on the context provided, there are a few potential reasons a Debezium connector can lose its binlog offset:\\n\\n- The connector was down for an extended period of time, during which the MySQL binlog rotated. When the connector comes back up, it tries to resume from its last recorded offset, but that offset no longer exists in the current binlog.\\n\\n- The connector's configuration does not have heartbeat messages enabled. Heartbeat messages periodically store the connector's latest offset, protecting against offset loss on connector restart.\\n\\n- There is a misconfiguration or bug that causes the connector to not properly store its latest offset in Kafka. This can lead to the offset being lost when the connector is restarted.\\n\\n- If Kafka offsets topic retention is too short, offset messages can be deleted before the connector has a chance to resume from them.\\n\\nSo in summary, the main reasons are binlog rotation during an outage, lack of heartbeat messages, improper offset storage, and short retention of offset messages. Enabling heartbeat messages and configuring appropriate retention is key to avoiding offset loss.\",\n      \"additional_kwargs\": {},\n      \"example\": false\n    }\n  ]\n}\n

We observe that the chat history, which was utilized to feed the LLM, returns in the response under \"chat_history.\"

\"chat_history\": [\n{\n    \"content\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"additional_kwargs\": {},\n    \"example\": false\n},\n{\n    \"content\": \" Based on the context provided, there are a few potential reasons a Debezium connector can lose its binlog offset:\\n\\n- The connector was down for an extended period of time, during which the MySQL binlog rotated. When the connector comes back up, it tries to resume from its last recorded offset, but that offset no longer exists in the current binlog.\\n\\n- The connector's configuration does not have heartbeat messages enabled. Heartbeat messages periodically store the connector's latest offset, protecting against offset loss on connector restart.\\n\\n- There is a misconfiguration or bug that causes the connector to not properly store its latest offset in Kafka. This can lead to the offset being lost when the connector is restarted.\\n\\n- If Kafka offsets topic retention is too short, offset messages can be deleted before the connector has a chance to resume from them.\\n\\nSo in summary, the main reasons are binlog rotation during an outage, lack of heartbeat messages, improper offset storage, and short retention of offset messages. Enabling heartbeat messages and configuring appropriate retention is key to avoiding offset loss.\",\n    \"additional_kwargs\": {},\n    \"example\": false\n}\n]\n
"},{"location":"creds/","title":"Onboarding your team","text":"

Follow the next steps in order to gain access to Jarvis API and the SFTP folder for uploading sensitive or company related data to power up your AI game.

"},{"location":"creds/#prerequisites","title":"Prerequisites","text":""},{"location":"creds/#keybase-username","title":"Keybase username","text":"

Obtain a valid Keybase username. We will share the credentials to Jarvis securely with this user.

"},{"location":"creds/#ssh-keys","title":"SSH keys","text":""},{"location":"creds/#team-name","title":"Team name","text":"

Solve one of programming hardest challenges, and decide on a name to represent your team in the competition! :)

"},{"location":"creds/#signing-up","title":"Signing up","text":"
  1. Go to: Gen AI Hackathon 2023 Teams in OneDrive.
  2. Create a new folder, name it with your team name.
  3. Upload to your new folder the ssh public key.
  4. Create keybase.txt file and add the Keybase username to the file.
  5. You can follow the convention in the example team folder.
"},{"location":"creds/#sharing-credentials","title":"Sharing credentials","text":"

Our magic behind the scenes will setup your access privileges, and the keybase username will be sent the following details: 1. app-id & private-key values used for authentication with Jarvis API. 2. SFTP username - used together with your generated RSA private key to connect to SFTP and upload files.

Good luck :)

"},{"location":"loading/","title":"Loading Documents","text":"

Jarvis supports loading documents from 3 data sources: 1. SFTP 2. Confluence 3. Webpages

"},{"location":"loading/#sftp","title":"SFTP","text":"

Your team will not only receive a private-key and app-id for the API, but it will also receive a set of credentials to use the SFTP server. Private key Username Password Address

Once you connect to the sftp, you will have your own folder where you can upload your files. A recommended SFTP client for mac can be Cyberduck

in this example we uploaded multiple files to the sftp folder

NOTE: At this point in time, those file are not yet known to Jarvis, and not stored in Postgres as vectors. They are only stored in the SFTP.

"},{"location":"loading/#loading-the-files-from-sftp-to-jarvis","title":"Loading the files from SFTP to jarvis","text":"

Jarvis is connected to the sftp and is able via API to get a trigger and upload specific file from the SFTP to Postgres. Prior to this, jarvis will call Amazon titan embedding model and create a vector as we learned in the RAG part. To do that:

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"sftp\",\n        \"files\": [\"mastercard.docx\"]\n    },\n    \"collection\": \"hackathon\"\n}'\n

After a successful response, those file that we loaded will exist in Jarvis, and we will be able to ask question about them. More on that on the next chapter.

supported file types: the supported file types are: pdf, csv, docx, xlsx, pptx, md, tsx, txt

collection: By choosing different collection in the API you can isolate different docs, you will be able to specify the collection on your prompts

"},{"location":"loading/#confluence","title":"Confluence","text":"

Jarvis is also directly integrated to our confluence (https://gpo-engineering.atlassian.net/wiki) You can use Jarvis API to stream confluence pages and space in one API call.

in this example we will publish the whole space but limit to 20 pages

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"confluence\",\n        \"space\": \"HUB\",\n        \"max_pages\": 20\n    },\n    \"collection\": \"confluence\"\n}'\n

and in this example we will choose to load only specific page ids from confluence

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"confluence\",\n        \"space\": \"HUB\",\n        \"page_ids\": [\"114327634\", \"830275595\"]\n    },\n    \"collection\": \"confluence\"\n}'\n

NOTE: Uploading multiple pages from confluence can take some time, and you may get a timeout, we suggest starting with limited number of page_ids. At any case, if there is a timeout from the client, Jarvis will continue uploading the files.

"},{"location":"loading/#webpages","title":"Webpages","text":"

Uploading a webpage is simple as that: 

curl --location 'https://api.paymentsos.com/hackathon-ai/load' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replamce' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"loader\": {\n        \"type\": \"web\",\n        \"urls\": [\"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\", \"https://medium.com/payu-engineering/strengthen-the-platforms-security-with-capture-the-flag-aafc4ee12c65\"]\n    },\n    \"collection\": \"website-hackathon\"\n}'\n
"},{"location":"privacy/","title":"Privacy","text":""},{"location":"privacy/#data-privacy","title":"Data privacy","text":"

if you decide to allow teams to build models based on real personal data stored by PayU you must consider this two points.

"},{"location":"privacy/#preparation-of-training-data","title":"Preparation of training data","text":"

Any data science activities should not be done on raw data but on data which was deidentified (or even anonymized) as far as possible (e.g. by removing names, surnames, addresses, hashing of card data, etc.). If a team intends to use any real user data, the scope of the data used and the manner of deidentification should be agreed between you and the relevant team. As we discussed, there is small number of teams that plans to work on PayU personal data to create their models, so the consultation should not be too engaging.

"},{"location":"privacy/#safe-environment-for-cooperation","title":"Safe environment for cooperation","text":"

Hackathon data science activities should be done in a dedicated safe environment allowing participants to share information and discuss ideas. The tools used by participants should ensure that any information is secure, accessed only by verified users and has appropriate controls limiting export of data. Shortly after the event any personal data should be deleted.

Also, please remember that any AI projects should also be assessed from a fairness and ethics perspective, especially with respect to bias control, transparency, explainability, etc. However, given that these models will not be used in production but removed after the hackathon, in this case we can resign from such additional assessments.

"},{"location":"prompt-context/","title":"Prompts with RAG Pattern","text":""},{"location":"prompt-context/#using-rag","title":"Using RAG","text":"

We discussed how to load Jarvis with documents from different sources in the previous chapter. Let's see how we can use Jarvis API to ask questions about these documents.

Using the website loader, we uploaded two different blog posts. One of the blog posts discussed debeizum, a CDC tool used in GPO engineering.

Let's ask a question about that article that was written after the FM finished its training

The API using RAG is similar to the previous used API with zero shot, the difference is the addition of the rag object.

\n{\n \"return_source_documents\": true, //if true response will contain the docs that the answer is based on\n \"search_type\": \"similarity\", // see API reference for the options.\n \"k\": 5, // how many vectors should be retrieved from the db.\n \"collection\": \"hackathon\" //on which collection of documents look for the vectors - this must be the same collection where doc was loaded.\n}\n
curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: a4b90687-616f-4173-9774-af3cc6bd86bc' \\\n--header 'app-id: com.payu.3ds-latam-brazil' \\\n--header 'x-zooz-app-name: com.payu.3ds-latam-brazil' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 200,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"return_source_documents\": true,\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\"\n    }\n}'\n

This will be our response

{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"result\": \" Here are some common reasons a Debezium connector can lose its binlog offset:\\n\\n- MySQL server restart - This causes the binlog to be rotated, so if the connector misses this event, it will be looking for the old binlog position that no longer exists.\\n\\n- Connector restart - If the connector goes down uncleanly (crash, kill -9, etc), it may not have time to commit the latest offset to Kafka. Upon restart, it will resume from the last committed offset, losing any uncommitted transactions.\\n\\n- Unclean connector shutdown - Similar to above, if the connector is killed or crashes, any uncommitted offsets are lost.\\n\\n- Heartbeats disabled - The heartbeat feature periodically sends dummy events to Kafka to update the offset. If disabled, long periods of inactivity can cause the connector to lose its place.\\n\\n- Kafka retention issues - If Kafka purges messages or offsets stored in __consumer_offsets before the\",\n    \"source_documents\": [\n        {\n            \"page_content\": \"was among the early adopters of Debezium, which led us to face several challenges throughout the process. However, over time, we grew more confident and gained a deeper understanding of the necessary steps we had to make in order to improve our Debezium implementation.We encountered a few minor issues, often originating from misconfigured connectors. However, the most important topic we\u201a\u00c4\u00f4ll discuss is when a connector loses its binlog offset.As mentioned earlier, every connector can be\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"was among the early adopters of Debezium, which led us to face several challenges throughout the process. However, over time, we grew more confident and gained a deeper understanding of the necessary steps we had to make in order to improve our Debezium implementation.We encountered a few minor issues, often originating from misconfigured connectors. However, the most important topic we\u201a\u00c4\u00f4ll discuss is when a connector loses its binlog offset.As mentioned earlier, every connector can be\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"from two tables \u201a\u00c4\u00ee the original table but also changes of the debezium_heartbeat table.With heartbeat set to occur once an hour, we can be certain that the connector\u201a\u00c4\u00f4s offset will remain up-to-date \u201a\u00c4\u00ee This is due to the connector capturing changes from both tables, effectively storing the latest offset out of both within the binlog.Our connector configuration now looks as such in v2.3.0.{   \\\"name\\\":\\\"debezium-packages-connector-sandbox\\\",\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"from two tables \u201a\u00c4\u00ee the original table but also changes of the debezium_heartbeat table.With heartbeat set to occur once an hour, we can be certain that the connector\u201a\u00c4\u00f4s offset will remain up-to-date \u201a\u00c4\u00ee This is due to the connector capturing changes from both tables, effectively storing the latest offset out of both within the binlog.Our connector configuration now looks as such in v2.3.0.{   \\\"name\\\":\\\"debezium-packages-connector-sandbox\\\",\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        },\n        {\n            \"page_content\": \"it debezium_connect_offsets.As seen in the Kafka UI screenshot below, within the specific topic dedicated to Debezium offsets, we can see that the most recent offset message for the packages_sandbox connector is listed as 1435109 (pos).Offset message of \u201a\u00c4\u00fapackages\u201a\u00c4\u00f9 connector, storing its latest recorded offsetIt is crucial to understand the following:A binlog contains changes from all the tables within MySQL, not only from the tables we\u201a\u00c4\u00f4ve configured the connector to capture changes\",\n            \"metadata\": {\n                \"source\": \"https://medium.com/payu-engineering/unlocking-the-power-of-debezium-69ce9170f101\",\n                \"title\": \"Unlocking the Power of Debezium. This blog post aims to share our\u201a\u00c4\u00b6 | by Tomer Guttman | payu-engineering | Aug, 2023 | Medium\",\n                \"description\": \"This blog post aims to share our experience, and insights with Debezium (CDC Tool), and the development phases we went through to improve reliability, and observability while capturing data changes\u201a\u00c4\u00b6\",\n                \"language\": \"en\"\n            }\n        }\n    ]\n}\n

as mentioned above, we will get the source_documeants in the response to help us debug why we got the answer we got and an answer to our question.

"},{"location":"prompt-context/#prompt-template","title":"Prompt Template","text":"

One of the features many FM supports is sending instructions to the model to answer your questions. Most of you saw chatGPT prompts that asks for a brief answer as a pirate. While we can send these instructions to the FM we are working with, this instructions will go to RAG pattern as well as they are part of the questions and will affect the vectors that are found.

To solve this we are happy to introduce the prompt_template:

see this request for example:

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"prompt_template\": \"Answer the following questions as best you can, but speaking as a pirate might speak. Use lots of \\\"Arg\\\"s.\\nBegin\\n\\nHuman: {question}\\n\\nContext: {context}\\n\\nAssistant:\",\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 200,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"return_source_documents\": true,\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\"\n    }\n}'\n

And the response:

{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"result\": \" Arr matey, there be a few reasons a Debezium connector could be losin' its place in the binlog:\\n\\nFirst, if the MySQL server be restarted, that wipes the binlog clean and the connector won't know where it left off. So you gotta be careful not to restart the server if ye can help it, else the connector will start over from the beginning.\\n\\nSecond, network issues can cause the connector to lose its connection. If it can't keep talkin' to the MySQL server, it may miss some binlog events and lose track of the latest position. \\n\\nAnd third, sometimes the connector offset gets out o' sync if Debezium's heartbeat don't match up with the actual binlog position. That's why it be important to keep an eye on both and make sure they line up proper.\\n\\nBut don't be despairin'! There be ways to make sure your connector don't lose\",\n}\n

And if we take it more seriously, lets try this: 

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: a4b90687-616f-4173-9774-af3cc6bd86bc' \\\n--header 'private-key: replaceme' \\\n--header 'app-id: replaceme' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"prompt_template\": \"Answer the  briefly in bullets and as you were a sales person.\\nBegin\\n\\nHuman: {question}\\n\\nContext: {context}\\n\\nAssistant:\",\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 1000,\n        \"temperature\": 0.3,\n        \"top_k\": 250,\n        \"top_p\": 0.9\n    },\n    \"rag\": {\n        \"return_source_documents\": true,\n        \"search_type\": \"similarity\",\n        \"k\": 5,\n        \"collection\": \"hackathon\"\n    }\n}'\n

we will get this

{\n    \"question\": \"What are the reasons a debezium connector loses its binlog offset?\",\n    \"result\": \" Here are some common reasons a Debezium connector can lose its binlog offset:\\n\\n- MySQL server is restarted - This causes the binlog to be rotated, so the connector loses its place\\n\\n- Connector is stopped for too long - If the connector is down for longer than the MySQL binary log expiration period, the logs it needs may be deleted\\n\\n- Heartbeat configuration issue - If heartbeats are not configured properly, the connector may not be updating its offset frequently enough\\n\\n- Kafka retention issues - If Kafka is not retaining the offset topic long enough, offset messages may be deleted \\n\\n- Connector configuration error - An error in the connector configuration, such as the database or table whitelist, can cause offset issues\\n\\n- MySQL GTID mode not enabled - Debezium requires GTID mode to accurately track binlog positions \\n\\n- DB snapshot followed by purge - If a snapshot is followed by a BINLOG PURGE, the connector's recorded offset may be purged\\n\\n- DB admin intervention - An admin may have manually purged logs, inadvertently deleting offsets \\n\\n- Kafka Connect rebalance - Offsets may not be committed properly during a Connect cluster rebalance\\n\\nSo in summary - restarting MySQL, long connector downtime, Kafka retention, Connect errors, and admin actions on the db can all lead to lost offsets. Proper connector setup and config can mitigate many issues.\",}\n
"},{"location":"simple/","title":"Zero shot prompt","text":""},{"location":"simple/#zero-shot-prompt","title":"Zero shot prompt","text":"

Zero-Shot Prompting: If you\u2019ve interacted with an LLM-powered chatbot before, you\u2019ve likely already used zero-shot prompting unwittingly. Zero-shot prompting entails relying solely on an LLM\u2019s pre-trained information to answer a given user prompt.

To create your first request to Jarvis you will need a private-key and an app-id that will be provided by the hackathon team. This are your authentication keys, and they should be used by your team and remain private.

An example of a simple zero shot prompt to jarvis looks like this:

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: replace-me' \\\n--header 'app-id: replace-me' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"question\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"anthropic.claude-v2\",\n        \"max_tokens\": 1000,\n        \"temperature\": 1,\n        \"top_k\": 250,\n        \"top_p\": 0.999\n    }\n}'\n

and this is the answer you will get:

{\n    \"query\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"answer\": \" Python is currently one of the most popular programming languages.\"\n}\n

Now lets go over all the params we had in this http request question - This is the prompt that will be sent to the FM stop_sequences - A set of chars that will stop the response from the FM like \\n for new line * name - This is the model id from the - Bedrock supported models max_tokens - The maximum tokens that will be in the answer - (1 token equal to 3/4 words) temperature - Read here top_k - Read here top_p - Read here

Playing with the request body: different models and different params like temperature, top_k and top_p will give you different answers, play with them and find the right tool for the job.

As an example, lets ask FM from AI21 the same question

curl --location 'https://api.paymentsos.com/hackathon-ai/prompt' \\\n--header 'private-key: replace-me' \\\n--header 'app-id: replace-me' \\\n--header 'Content-Type: application/json' \\\n--data '{\n\"question\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"model\": {\n        \"stop_sequences\": [],\n        \"name\": \"ai21.j2-mid\",\n        \"max_tokens\": 1000,\n        \"temperature\": 1,\n        \"top_k\": 250,\n        \"top_p\": 0.999\n    }\n}'\n

and this is the answer you will get:

{\n    \"query\": \"Can you please tell me what is the most pouplar programing language, please provide short answer\",\n    \"answer\": \" According to recent surveys, JavaScript is currently the most popular programming language among developers.\"\n}\n

As you can see different models trained and tuned on different sets of data can return various responses.

"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 61e9887db8806b13e639e2f2a6b69e28f82a1d93..3acb7429ce9357cee313f13c7b6fb3e8e9170c38 100644 GIT binary patch delta 13 Ucmb=gXP58h;JBD9Gm*Ul036T+AOHXW delta 13 Ucmb=gXP58h;5d{kI+48s033`26aWAK