Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Gemini Resource #96

Merged
merged 2 commits into from
Jan 15, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/quality-check-dagster-contrib-gemini.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: quality-check-dagster-contrib-gemini
on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- 'libraries/dagster-contrib-gemini/**'

jobs:
check:
uses: ./.github/workflows/template-quality-check.yml
with:
working_directory: ./libraries/dagster-contrib-gemini
14 changes: 14 additions & 0 deletions .github/workflows/release-dagster-contrib-gemini.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: build-and-release-dagster-contrib-gemini

on:
push:
tags:
- 'dagster_contrib_gemini-*.*.*'

jobs:
build-and-release-dagster-contrib-gemini:
uses: ./.github/workflows/template-release.yml
with:
library_name: dagster-contrib-gemini
working_directory: ./libraries/dagster-contrib-gemini
secrets: inherit
9 changes: 9 additions & 0 deletions libraries/dagster-contrib-gemini/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
test:
uv run pytest

ruff:
uv run ruff check --fix .
uv run ruff format .

check:
uv run pyright
58 changes: 58 additions & 0 deletions libraries/dagster-contrib-gemini/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# dagster-contrib-gemini

A dagster module that provides integration with [Gemini](https://deepmind.google/technologies/gemini/).

## Installation

The `dagster_contrib_gemini` module is available as a PyPI package - install with your preferred python
environment manager (We recommend [uv](https://github.com/astral-sh/uv)).

```
source .venv/bin/activate
uv pip install dagster_contrib_gemini
```

## Example Usage

In addition to wrapping the Gemini GenerativeModel class (get_model/get_model_for_asset methods),
this resource logs the usage of the Gemini API to to the asset metadata (both number of calls, and tokens).
This is achieved by wrapping the GenerativeModel.generate_content method.

Note that the usage will only be logged to the asset metadata from an Asset context -
not from an Op context.
Also note that only the synchronous API usage metadata will be automatically logged -
not the streaming or batching API.

```python
from dagster import AssetExecutionContext, Definitions, EnvVar, asset, define_asset_job
from dagster_contrib_gemini import GeminiResource


@asset(compute_kind="gemini")
def gemini_asset(context: AssetExecutionContext, gemini: GeminiResource):
with gemini.get_model(context) as model:
response = model.generate_content(
"Generate a short sentence on tests"
)

defs = Definitions(
assets=[gemini_asset],
resources={
"gemini": GeminiResource(
api_key=EnvVar("GEMINI_API_KEY"),
generative_model_name="gemini-1.5-flash"
),
},
)
```


## Development

The `Makefile` provides the tools required to test and lint your local installation

```sh
make test
make ruff
make check
```
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from dagster_contrib_gemini.resource import GeminiResource as GeminiResource
Loading
Loading