Skip to content

Commit

Permalink
Add pluggable functions tutorial and pull from main
Browse files Browse the repository at this point in the history
  • Loading branch information
yuxtang-amazon committed Jul 31, 2023
1 parent 01c1a09 commit 3147a61
Show file tree
Hide file tree
Showing 15 changed files with 333 additions and 408 deletions.
2 changes: 0 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,6 @@ Thank you to all who have contributed!

## [0.12.0] - 2023-06-14

## [0.12.0] - 2023-06-14

### Added

- Adds support for using EXCLUDED within DML ON-CONFLICT-ACTION conditions. Closes #1111.
Expand Down
136 changes: 136 additions & 0 deletions docs/wiki/tutorials/Pluggable Functions Tutorial.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# Pluggable Functions Tutorial

The PartiQL ecosystem has introduced a new method for integrating custom logic - Pluggable Functions. This system allows PartiQL users to implement and introduce their custom functions into the PartiQL Command Line Interface (CLI) without the need for extensive knowledge of the PartiQL codebase or build system.

This document will guide you on creating custom functions and loading them to the PartiQL CLI. The functions created by you will be pluggable, meaning they can be integrated as needed, enhancing the functionality of the CLI without modifying the underlying code.

## Step 1: Add Dependencies and Build Your Module

To build your module, you can use build automation tools like Maven or Gradle. These tools will handle the dependency management and compilation process for you. Here's how you can specify the necessary dependencies:

```Kotlin
dependencies {
implementation("org.partiql:partiql-spi:<latest_version>")
implementation("org.partiql:partiql-types:<latest_version>")
}
```

## Step 2: Create a Custom Function

To create a custom function, you need to implement the `PartiQLFunction` interface. This interface allows you to define the function's behavior and its signature, including its name, return type, parameters, determinism, and optional description.

Here's a basic template to help you start:

```Kotlin
import org.partiql.spi.connector.ConnectorSession
import org.partiql.spi.function.PartiQLFunction
import org.partiql.spi.function.PartiQLFunctionExperimental
import org.partiql.types.PartiQLValueType
import org.partiql.types.function.FunctionParameter
import org.partiql.types.function.FunctionSignature
import org.partiql.value.PartiQLValue
import org.partiql.value.PartiQLValueExperimental
import org.partiql.value.StringValue
import org.partiql.value.stringValue

@OptIn(PartiQLFunctionExperimental::class)
object TrimLead : PartiQLFunction {
override val signature = FunctionSignature(
name = "trim_lead", // Specify your function name
returns = PartiQLValueType.STRING, // Specify the return type
parameters = listOf(
FunctionParameter.ValueParameter(name = "str", type = PartiQLValueType.STRING) // Specify parameters
),
isDeterministic = true, // Specify determinism
description = "Trims leading whitespace of a [str]." // A brief description of your function
)

@OptIn(PartiQLValueExperimental::class)
override operator fun invoke(session: ConnectorSession, arguments: List<PartiQLValue>): PartiQLValue {
// Implement the function logic here
val str = (arguments[0] as? StringValue)?.string ?: ""
val processed = str.trimStart()
return stringValue(processed)
}
}
```

Ensure that you replace the signature and function invoking with your actual implementations.

## Step 3: Implement the Plugin Interface

Next, you need to implement the `Plugin` interface in your code. This allows you to return a list of all the custom `PartiQLFunction` implementations you've created, using the `getFunctions()` method. This step is crucial as it allows the service loader to retrieve all your custom functions.

Here's an example of a `Plugin` implementation:

```Kotlin
package org.partiql.plugins.mockdb

import org.partiql.spi.Plugin
import org.partiql.spi.connector.Connector
import org.partiql.spi.function.PartiQLFunction

public class LocalPlugin implements Plugin {
override fun getConnectorFactories(): List<Connector.Factory> = listOf()

@PartiQLFunctionExperimental
override fun getFunctions(): List<PartiQLFunction> = listOf(
TrimLead // Specify the functions
)
}
```

## Step 4: Create Service Provider Configuration file

In order for the Java's ServiceLoader to recognize your plugin and load your custom functions, you'll need to specify your `Plugin` implementation in a Service Provider Configuration file:

1. In your project's main resources directory (usually `src/main/resources`), create a new directory named `META-INF/services`.

2. Within this directory, create a new file named after the full interface name as `org.partiql.spi.Plugin`.

3. Inside this file, write the fully qualified name of your Plugin implementation class. If you have multiple implementations, each should be listed on a new line.

Here's an example if your `Plugin` implementation class is `org.partiql.plugins.mockdb.LocalPlugin`:
```Kotlin
org.partiql.plugins.mockdb.LocalPlugin
```

## Step 5: Package Your Functions into a .jar File

Compile your function and `Plugin` implementation into bytecode and package it into a .jar file. This .jar file will act as a plugin for the PartiQL CLI.

For example, if you are using Gradle, you can simply run `./gradlew build` to compile your module. And your .jar file can be found under `build/libs`.

## Step 6: Load the Functions into CLI

Each of your .jar files should be stored in its own subdirectory under the `plugins` directory, which itself is inside the .partiql directory in your home directory. Here's what the directory structure should look like:

```
~/.partiql/plugins
├── firstPlugin
│ └── firstPlugin.jar
└── secondPlugin
└── secondPlugin.jar
```

In the example above, `firstPlugin.jar` and `secondPlugin.jar` are the plugins you've created, each in its own directory under `~/.partiql/plugins`.

By default, the PartiQL CLI will search the `~/.partiql/plugins` directory for plugins. However, you can specify a different directory when starting the CLI with the `--plugins` option:

```shell
partiql --plugins /path/to/your/plugin/directory
```

With this command, the CLI will search for .jar files in the specified directory’s subdirectories and load them as plugins. This feature gives you the flexibility to organize your plugins in a manner that best suits your needs.

## Step 7: Invoke Custom Functions in CLI

To use the custom functions, you simply call them as you would any other function in the PartiQL CLI. For example, if you created a function named "trim_lead", you would invoke it like so:

```shell
partiql> trim_lead(string)
```

Please replace "string" with the actual string you want to trim.

That's all there is to it! With this mechanism, you can introduce any number of custom functions to PartiQL, and these functions will be usable just like built-in functions, without the need to modify the PartiQL codebase. It's an excellent way to extend the capabilities of PartiQL and adapt it to the specific needs of your project or organization.
37 changes: 2 additions & 35 deletions partiql-ast/src/main/kotlin/org/partiql/ast/helpers/ToLegacyAst.kt
Original file line number Diff line number Diff line change
Expand Up @@ -831,7 +831,7 @@ private class AstTranslator(val metas: Map<String, MetaContainer>) : AstBaseVisi
translate(node) { metas ->
val prefilter = node.prefilter?.let { visitExpr(it, ctx) }
val variable = node.variable
val label = node.label?.let { visitGraphMatchLabel(it, ctx) }
val label = node.label
node(prefilter, variable, label, metas)
}

Expand All @@ -849,7 +849,7 @@ private class AstTranslator(val metas: Map<String, MetaContainer>) : AstBaseVisi
val quantifier = node.quantifier?.let { visitGraphMatchQuantifier(it, ctx) }
val prefilter = node.prefilter?.let { visitExpr(it, ctx) }
val variable = node.variable
val label = node.label?.let { visitGraphMatchLabel(it, ctx) }
val label = node.label
edge(direction, quantifier, prefilter, variable, label, metas)
}

Expand Down Expand Up @@ -902,39 +902,6 @@ private class AstTranslator(val metas: Map<String, MetaContainer>) : AstBaseVisi
selectorShortestKGroup(k)
}

override fun visitGraphMatchLabel(node: GraphMatch.Label, ctx: Ctx) =
super.visitGraphMatchLabel(node, ctx) as PartiqlAst.GraphLabelSpec

override fun visitGraphMatchLabelName(node: GraphMatch.Label.Name, ctx: Ctx) =
translate(node) { metas ->
graphLabelName(node.name, metas)
}

override fun visitGraphMatchLabelWildcard(node: GraphMatch.Label.Wildcard, ctx: Ctx) =
translate(node) { metas ->
graphLabelWildcard(metas)
}

override fun visitGraphMatchLabelNegation(node: GraphMatch.Label.Negation, ctx: Ctx) =
translate(node) { metas ->
val arg = visitGraphMatchLabel(node.arg, ctx)
graphLabelNegation(arg, metas)
}

override fun visitGraphMatchLabelConj(node: GraphMatch.Label.Conj, ctx: Ctx) =
translate(node) { metas ->
val lhs = visitGraphMatchLabel(node.lhs, ctx)
val rhs = visitGraphMatchLabel(node.rhs, ctx)
graphLabelConj(lhs, rhs, metas)
}

override fun visitGraphMatchLabelDisj(node: GraphMatch.Label.Disj, ctx: Ctx) =
translate(node) { metas ->
val lhs = visitGraphMatchLabel(node.lhs, ctx)
val rhs = visitGraphMatchLabel(node.rhs, ctx)
graphLabelDisj(lhs, rhs, metas)
}

/**
* DML
*/
Expand Down
14 changes: 2 additions & 12 deletions partiql-ast/src/main/pig/partiql.ion
Original file line number Diff line number Diff line change
Expand Up @@ -271,31 +271,21 @@ may then be further optimized by selecting better implementations of each operat
(edge_left_or_right)
(edge_left_or_undirected_or_right))

// A label spec in a node pattern like `MATCH (x : <lab>)` or in an edge pattern like `MATCH −[t : <lab>]−>`
(sum graph_label_spec
(graph_label_name name::symbol) // as in `MATCH (x:Account)` or `MATCH -[x:Transfer]->`
(graph_label_wildcard) // as in `MATCH (x: %)`
(graph_label_negation arg::graph_label_spec) // as in `MATCH (x: !Account)`
(graph_label_conj lhs::graph_label_spec rhs::graph_label_spec) // as in `MATCH (x: City&Country)` - Monaco can do
(graph_label_disj lhs::graph_label_spec rhs::graph_label_spec) // as in `MATCH (x: City|Country)` - either Paris or Germany will do
)

// A part of a graph pattern
(sum graph_match_pattern_part
// A single node in a graph pattern.
(node
prefilter::(? expr) // an optional node pre-filter, e.g.: `WHERE c.name='Alarm'` in `MATCH (c WHERE c.name='Alarm')`
variable::(? symbol) // the optional element variable of the node match, e.g.: `x` in `MATCH (x)`
label::(? graph_label_spec)) // the optional label spec to match for the node, e.g.: `Entity` in `MATCH (x:Entity)`
label::(* symbol 0)) // the optional label(s) to match for the node, e.g.: `Entity` in `MATCH (x:Entity)`

// A single edge in a graph pattern.
(edge
direction::graph_match_direction // edge direction
quantifier::(? graph_match_quantifier) // an optional quantifier for the entire pattern match, e.g. `{2,5}` in `MATCH (a:Account)−[:Transfer]−>{2,5}(b:Account)`
prefilter::(? expr) // an optional edge pre-filter, e.g.: `WHERE t.capacity>100` in `MATCH −[t:hasSupply WHERE t.capacity>100]−>`
variable::(? symbol) // the optional element variable of the edge match, e.g.: `t` in `MATCH −[t]−>`
label::(? graph_label_spec)) // the optional label spec to match for the edge. e.g.: `Target` in `MATCH −[t:Target]−>`

label::(* symbol 0)) // the optional label(s) to match for the edge. e.g.: `Target` in `MATCH −[t:Target]−>`
// A sub-pattern.
(pattern pattern::graph_match_pattern))

Expand Down
13 changes: 2 additions & 11 deletions partiql-ast/src/main/resources/partiql_ast.ion
Original file line number Diff line number Diff line change
Expand Up @@ -651,15 +651,15 @@ graph_match::{
node::{
prefilter: optional::expr, // An optional node pre-filter, e.g.: `WHERE c.name='Alarm'` in `MATCH (c WHERE c.name='Alarm')`
variable: optional::string, // The optional element variable of the node match, e.g.: `x` in `MATCH (x)`
label: optional::label, // The optional label(s) to match for the node, e.g.: `Entity` in `MATCH (x:Entity)`
label: list::[string], // The optional label(s) to match for the node, e.g.: `Entity` in `MATCH (x:Entity)`
},
// A single edge in a graph pattern
edge::{
direction: direction, // Edge Direction
quantifier: optional::quantifier, // An optional quantifier for the entire pattern match, e.g. `{2,5}` in `MATCH (a:Account)−[:Transfer]−>{2,5}(b:Account)`
prefilter: optional::expr, // An optional edge pre-filter, e.g.: `WHERE t.capacity>100` in `MATCH −[t:hasSupply WHERE t.capacity>100]−>`
variable: optional::string, // The optional element variable of the edge match, e.g.: `t` in `MATCH −[t]−>`
label: optional::label, // The optional label spec to match for the edge. e.g.: `Target` in `MATCH −[t:Target]−>`
label: list::[string], // The optional label(s) to match for the edge. e.g.: `Target` in `MATCH −[t:Target]−>`
},
// A sub-pattern
pattern::{
Expand Down Expand Up @@ -702,15 +702,6 @@ graph_match::{
shortest_k::{ k: long }, // SHORTEST k
shortest_k_group::{ k: long }, // SHORTEST k GROUP
], // Fig. 8 — https://arxiv.org/abs/2112.06217

// A label spec in a node pattern like `MATCH (x : <lab>)` or in an edge pattern like `MATCH −[t : <lab>]−>`
label::[
name::{ name: string }, // as in `MATCH (x:Account)` or `MATCH -[x:Transfer]->`
wildcard::{}, // as in `MATCH (x: %)`
negation::{arg: label}, // as in `MATCH (x: !Account)`
conj::{lhs: label, rhs: label}, // as in `MATCH (x: City&Country)` - Monaco can do
disj::{lhs: label, rhs: label}, // as in `MATCH (x: City|Country)` - either Paris or Germany will do
],
],
}

Expand Down
Loading

0 comments on commit 3147a61

Please sign in to comment.