Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(tokenize): add basic tokenizer implementations #109

Merged
merged 7 commits into from
Oct 15, 2024
Merged

feat(tokenize): add basic tokenizer implementations #109

merged 7 commits into from
Oct 15, 2024

Conversation

nbsp
Copy link
Member

@nbsp nbsp commented Oct 14, 2024 •
edited
Loading

  • tokenizer baseclass
  • token stream
  • word
  • sentence
  • paragraph
  • hyphenator
  • full basic

@changeset-bot changeset-bot
Copy link

changeset-bot bot commented Oct 14, 2024 •
edited
Loading

🦋 Changeset detected

Latest commit: 0668b4c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@livekit/agents Minor
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-silero Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@nbsp nbsp requested review from bcherry and lukasIO October 14, 2024 15:43
@nbsp nbsp marked this pull request as ready for review October 14, 2024 19:19
@nbsp nbsp requested a review from theomonnom October 15, 2024 14:35
bcherry
bcherry approved these changes Oct 15, 2024
View reviewed changes
Copy link
Contributor

@bcherry bcherry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good - i assume this is pretty much a straight port from python and all the tokenizer stuff included here is equivalent to what's done there

@nbsp nbsp merged commit 76d09b2 into v0.4.0 Oct 15, 2024
4 checks passed
@nbsp nbsp deleted the nbsp/add/tokenizer branch October 15, 2024 21:34
nbsp added a commit that referenced this pull request Oct 26, 2024
@nbsp
feat(tokenize): add basic tokenizer implementations (#109)
33d66bd
nbsp added a commit that referenced this pull request Nov 4, 2024
@nbsp
feat(tokenize): add basic tokenizer implementations (#109)
d32f247
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bcherry bcherry bcherry approved these changes

@lukasIO lukasIO Awaiting requested review from lukasIO

@theomonnom theomonnom Awaiting requested review from theomonnom

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nbsp @bcherry