[Pinecone] Add rerank task

[ShihChun-H]

Issue Description

Current State*

• Current Pinecone does not support Rerank task

Why We Want to Change?*

• Pinecone introduces the rerank API

Proposed Change*

• Reduce the fields on the node.
• Mark the password as secret

---

Rules for the Component Hackathon

• Each issue will only be assigned to one person/team at a time.
• You can only work on one issue at a time.
• To express interest in an issue, please comment on it and tag @kuroxx, allowing the Instill AI team to assign it to you.
• Ensure you address all feedback and suggestions provided by the Instill AI team.
• If no commits are made within five days, the issue may be reassigned to another contributor.
• Join our Discord to engage in discussions and seek assistance in #hackathon channel. For technical queries, you can tag @chuang8511.

---

Component Contribution Guideline | Documentation | Official Go Tutorial

[linear]

INS-6359 [Pinecone] Add rerank task

[MubeenKodvavi]

@ShihChun-H

Please assign this task to me, I will add this functionility.
We need to leverage this API: https://docs.pinecone.io/reference/api/2024-10/inference/rerank

Some questions:
Pinecone is a data component. Do we need to rerank tasks through a data component?
Should we not introduce a seperate ai component for pinecone, or doing this with data is fine?

[MubeenKodvavi]

This should be integrated once rerank API is in stable release.
Currently, the API is available in version 2024-10 of Pinecone API which is a release-candidate.

[ShihChun-H]

Hi @chuang8511, Could you help clarify the question raised by @MubeenKodvavi? Thanks!

[ShihChun-H]

Hi @MubeenKodvavi, thanks for your offer to take on this task. Currently, our policy allows only one assigned issue per contributor at a time. I’ll assign this to you as soon as your current task is completed. Thanks for understanding! 🙌🏻

[donch1989]

Hi @MubeenKodvavi
Thank you for taking on this task. Although reranking is an advanced AI feature, it is frequently used in the scope of data operations. We believe it is still appropriate to include reranking in this data component as well.

[aazam-gh]

Hi, can I be assigned this issue :)

[ShihChun-H]

Hi @MubeenKodvavi, since you have already completed the issue assigned to you. Would you like to work on this issue next?

[ShihChun-H]

Hi @aazam-gh, there is a queue for this issue, would you mind waiting for a bit? 🙏

[aazam-gh]

Sure I'll look into other ones

[MubeenKodvavi]

@ShihChun-H
As I mentioned before, this issue should be implemented when the Rerank API is available in the stable version of the Pinecone API.
Currently, Pinecone API defaults to the latest release version, which is 2024-07. However, the Rerank API is not available in this version.
We can specify the 2024-10 version in the headers if we need this feature before 2024-10 is stable.

You can assign this task to me, and I can raise a PR. However, I suggest waiting until the 2024-10 version is officially released before merging, as it is expected to be available by the end of this month. Reference: Pinecone API Versioning

[ShihChun-H]

Hi @MubeenKodvavi Totally understand the circumstance.
I will assign this issue to you, and you can submit the PR after the 2024-10 version is officially released. This ensures that the assignment remains with you throughout the process, and won't be taken away from you 🙌🏻

[kuroxx]

Hey @MubeenKodvavi how's it going? Just checking the status of this task - has there been any updates?
In the meantime, feel free to join our Discord server to share & support others with component development: discord.gg/sevxWsqpGh

[MubeenKodvavi]

Hi @kuroxx , I will raise a PR for it this weekend.

[MubeenKodvavi]

@ShihChun-H @kuroxx can you please request a review of linked PR from the team? Thank you :)

[kuroxx]

Hey @MubeenKodvavi yes will pass it on to the team!! Thanks for raising 🤝

[kuroxx]

Hey @MubeenKodvavi could you check the feedback in your PR and update accordingly, thanks! 🙇