Search and Scrape Webpages

[toyamarinyon]

Summary

Search web pages using a query derived from the user's request and scrape each page.
Scraped data transform markdown and persistence to Vercel Blob

Private beta users can try out this prototype.
It does not affect other users.

Changes

Add webSearches property into graphv2 column of agents table.

Release flow

1. Merge this pull request
2. Migrate database with following script:

bun drizzle-kit migrate

3. Migrate data with following script:

bun scripts/20241018-db-patch-web-search.ts

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
giselle	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 18, 2024 5:13am

[shige]

@toyamarinyon I got the error in preview https://giselle-1rwcfr8c0-r06-edge.vercel.app/p/agnt_gjujxm4uhoc87hqzumrh2qh0 .

Unhandled Rejection: Error: TAVILY_API_KEY is not defined
    at n (/var/task/.next/server/app/(playground)/p/[agentId]/page.js:31:2671)
    at /var/task/.next/server/app/(playground)/p/[agentId]/page.js:31:1586
    at Array.map (<anonymous>)
    at /var/task/.next/server/app/(playground)/p/[agentId]/page.js:31:1572
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Node.js process exited with exit status: 128. The logs above can help with debugging the issue.

Could you verify the operation in the preview environment?

[shige]

Maybe the environment variables were set after the Vercel deployment? 🤔

[toyamarinyon]

@shige I Yes, a redeployment was necessary. Also, a database migration was needed, so I'll make some adjustments.

[shige]

Thanks! I'll check it after CI passed.

[shige]

LGTM ✅

[shige]

Release flow

1. Merge this pull request
2. Migrate database with following script:

bun drizzle-kit migrate

3. Migrate data with following script:

bun scripts/20241018-db-patch-web-search.ts

All done! 🧑‍💻