Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(client): v3: ilp over http #26

Merged
merged 114 commits into from
Mar 20, 2024
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
ffeb62a
initial commit
sklarsa Jan 30, 2024
df001f7
move module to v2 dir
sklarsa Feb 1, 2024
d68ef77
starting v3
sklarsa Feb 1, 2024
06852b0
bug fixes
sklarsa Feb 1, 2024
cc840e2
rework config parsing
sklarsa Feb 12, 2024
e1a8a79
more
sklarsa Feb 12, 2024
a195b99
update test path
sklarsa Feb 12, 2024
5796819
v3 tests
sklarsa Feb 12, 2024
5c7553b
testing is hard
sklarsa Feb 12, 2024
0ac49d9
bug fix
sklarsa Feb 12, 2024
9edc928
adding tests
sklarsa Feb 13, 2024
2428fed
better corner case and semicolon escape handling
sklarsa Feb 13, 2024
c23d7fb
more tests
sklarsa Feb 13, 2024
7a7dfac
fix request timeout
sklarsa Feb 13, 2024
9026f74
adds retry
sklarsa Feb 13, 2024
85dc69e
more tests
sklarsa Feb 13, 2024
4683ca0
more
sklarsa Feb 13, 2024
9ba985d
separate v2 and v3 net
sklarsa Feb 20, 2024
21780b1
wip tests
sklarsa Feb 20, 2024
e6087e6
tests pass
sklarsa Feb 20, 2024
90c5c73
add http auth
sklarsa Feb 20, 2024
c56c685
get https auth test working (and fix/add tls)
sklarsa Feb 20, 2024
06330a8
wip
sklarsa Feb 21, 2024
d7ee9fe
bug fix
sklarsa Feb 22, 2024
33b7c75
hopefully fixes tests
sklarsa Feb 22, 2024
5d70c13
fix timeout conversion
sklarsa Feb 22, 2024
0331884
remove v2
sklarsa Feb 22, 2024
7e63a9e
fix readme and examples
sklarsa Feb 22, 2024
6714968
v3->questdb
sklarsa Feb 27, 2024
03e5b19
close resp.Body
sklarsa Feb 27, 2024
696c869
Update sender.go
sklarsa Feb 27, 2024
4785bed
fix
sklarsa Feb 27, 2024
cda2072
Update conf.go
sklarsa Feb 27, 2024
1732892
Update sender.go
sklarsa Feb 27, 2024
5b01e85
Update sender.go
sklarsa Feb 27, 2024
b0f8445
refactoring buffer
sklarsa Feb 27, 2024
a7ca64f
more buffer refactor
sklarsa Feb 27, 2024
cfe94d7
more
sklarsa Feb 27, 2024
b48a427
eod
sklarsa Feb 27, 2024
a98a6e9
split out conf into http/tcp
sklarsa Feb 28, 2024
cdc7c15
revert http/tcp mixed sender tests
sklarsa Feb 28, 2024
21b13ee
test pass... locally...
sklarsa Feb 28, 2024
f1f2c85
splitting everything into separate packages
sklarsa Feb 28, 2024
ce95d9b
stuck trying to alias
sklarsa Feb 28, 2024
66b8b72
fix all
sklarsa Feb 28, 2024
d1b7c2b
staticcheck
sklarsa Feb 28, 2024
cad05a2
remove redundant "http"s
sklarsa Feb 28, 2024
daff80a
move tests around
sklarsa Feb 28, 2024
3acd8f4
adding tests
sklarsa Feb 29, 2024
8b77c2a
some godocs
sklarsa Feb 29, 2024
ccdba3f
rm comment
sklarsa Feb 29, 2024
e9a43b5
move tests to buffer pkg
sklarsa Feb 29, 2024
87d6f5c
move another test
sklarsa Feb 29, 2024
33f325c
cleanup
sklarsa Mar 4, 2024
0ad64c9
upgrade deps and deprecate go 1.17
sklarsa Mar 4, 2024
3678e3a
revert testcontainers version bump because of multiple deprecations i…
sklarsa Mar 4, 2024
45bdd4b
Apply suggestions from code review
sklarsa Mar 5, 2024
db66084
rm const.go
sklarsa Mar 5, 2024
8659b73
remove interop test json (in preparation for re-adding submodule
sklarsa Mar 5, 2024
210de81
use original interop commit
sklarsa Mar 5, 2024
afc66ea
update readme to include http
sklarsa Mar 5, 2024
5409285
adds ilp prefix to examples manifest http names
sklarsa Mar 5, 2024
188316e
rename grace_timeout to request_timeout
sklarsa Mar 6, 2024
4d2d67b
adds better docs for LineSenderFromConf and updates request_timeout d…
sklarsa Mar 6, 2024
7fa27ef
adds note under http in readme with link to docs
sklarsa Mar 6, 2024
05e30b4
remove max buffer size from http client
sklarsa Mar 6, 2024
ea10005
document buffer.At
sklarsa Mar 6, 2024
686c525
remove another bufcap usage in http
sklarsa Mar 6, 2024
9929a2d
fmt
sklarsa Mar 6, 2024
70bcbda
adds defaults for tcp autoflush
sklarsa Mar 6, 2024
8d45908
auto-flush fix
sklarsa Mar 6, 2024
4ce1fa0
remove autoflush from tcp sender
sklarsa Mar 11, 2024
66aacd3
only allocate uri once on http sender init
sklarsa Mar 11, 2024
d2c208f
remove autoFlushRows boolean
sklarsa Mar 11, 2024
e2a0a04
simplify autoflush
sklarsa Mar 11, 2024
a66fca6
fix uri and autoflush test
sklarsa Mar 11, 2024
fb4b331
add flush on close (with close-checks for other funcs) where appropriate
sklarsa Mar 11, 2024
e612d78
improved error handling
sklarsa Mar 12, 2024
dbd82ba
no error on double close
sklarsa Mar 12, 2024
f9de02f
documents Close() auto-flush behavior
sklarsa Mar 12, 2024
66f8c9b
adds test for buffer clear on flush
sklarsa Mar 12, 2024
4752c47
switch internal buffer to private to simplify sender public api
sklarsa Mar 12, 2024
a21a608
adds golden path integration tests for both tcp and http
sklarsa Mar 12, 2024
d02652c
ensure that global transport is cleaned up properly if autoflush fail…
sklarsa Mar 12, 2024
8311a85
add autoflush back to tcp (with tests
sklarsa Mar 12, 2024
3ecef82
fix error msg
sklarsa Mar 13, 2024
7b2263b
adds http transport override option
sklarsa Mar 13, 2024
b22cede
Update pkg/http/errors.go
sklarsa Mar 13, 2024
a47ae6c
Minor clean up
puzpuzpuz Mar 14, 2024
7bbe815
Add auto flush interval
puzpuzpuz Mar 15, 2024
10e55e0
Auto flush support in conf string
puzpuzpuz Mar 15, 2024
00949da
Fix compilation
puzpuzpuz Mar 15, 2024
8cbadb2
Remove auto flush from TCP client
puzpuzpuz Mar 15, 2024
8e4602c
Change package structure to flat one
puzpuzpuz Mar 18, 2024
87c8f4b
Introduce LineSender interface
puzpuzpuz Mar 18, 2024
e7949db
Introduce builder
puzpuzpuz Mar 18, 2024
5396fd9
Hook common interface
puzpuzpuz Mar 18, 2024
7a211ba
Simplify global transport
puzpuzpuz Mar 18, 2024
e55ada8
Remove redundant field
puzpuzpuz Mar 18, 2024
ec81769
fix tests
sklarsa Mar 18, 2024
6b4bef5
remove go 1.22 thing from go.mod
sklarsa Mar 18, 2024
7e12b64
ok lets see if that setting works in an older version of testcontainers
sklarsa Mar 18, 2024
cd6ee5e
group all integration tests into a testify suite
sklarsa Mar 18, 2024
450c977
adds some logical validation for tcp and http senders
sklarsa Mar 18, 2024
2d4f235
Implement max buffer size checks
puzpuzpuz Mar 19, 2024
c75074e
Remove deprecated crypto API usage
puzpuzpuz Mar 19, 2024
18616a8
Add from conf example
puzpuzpuz Mar 19, 2024
fdf4227
Better docs
puzpuzpuz Mar 19, 2024
39cab69
Fix readme
puzpuzpuz Mar 19, 2024
e0094d9
Improve readme
puzpuzpuz Mar 19, 2024
1a5fa73
a few more tests
sklarsa Mar 19, 2024
dc95225
Fix default HTTP address
puzpuzpuz Mar 19, 2024
40f560a
Expect username and password in conf string
puzpuzpuz Mar 19, 2024
d261f2e
Improve docs and add more tests
puzpuzpuz Mar 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[![GoDoc reference](https://img.shields.io/badge/godoc-reference-blue.svg)](https://pkg.go.dev/github.com/questdb/go-questdb-client/v2)
[![GoDoc reference](https://img.shields.io/badge/godoc-reference-blue.svg)](https://pkg.go.dev/github.com/questdb/go-questdb-client/v3)

# go-questdb-client

Expand All @@ -10,7 +10,7 @@ Features:
* Supports TLS encryption and [ILP authentication](https://questdb.io/docs/reference/api/ilp/authenticate).
* Tested against QuestDB 7.3.2 and newer versions.

Documentation is available [here](https://pkg.go.dev/github.com/questdb/go-questdb-client/v2).
Documentation is available [here](https://pkg.go.dev/github.com/questdb/go-questdb-client/v3).

## Usage

sklarsa marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -23,7 +23,7 @@ import (
"log"
"time"

qdb "github.com/questdb/go-questdb-client/v2"
qdb "github.com/questdb/go-questdb-client/v3"
)

func main() {
Expand Down
216 changes: 216 additions & 0 deletions conf.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
/*******************************************************************************
* ___ _ ____ ____
* / _ \ _ _ ___ ___| |_| _ \| __ )
* | | | | | | |/ _ \/ __| __| | | | _ \
* | |_| | |_| | __/\__ \ |_| |_| | |_) |
* \__\_\\__,_|\___||___/\__|____/|____/
*
* Copyright (c) 2014-2019 Appsicle
* Copyright (c) 2019-2022 QuestDB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
******************************************************************************/

package v3
sklarsa marked this conversation as resolved.
Show resolved Hide resolved

import (
"strconv"
"strings"
"time"
)

type schemaType string

const (
schemaHttp schemaType = "http"
schemaHttps schemaType = "https"
schemaTcp schemaType = "tcp"
schemaTcps schemaType = "tcps"
)

func parseConfigString(conf string) ([]LineSenderOption, error) {
var (
key = &strings.Builder{}
value = &strings.Builder{}
isKey = true

nextRune rune
isEscaping bool
hasTrailingSemicolon bool
opts []LineSenderOption
user, pass, token string
)

schemaStr, conf, found := strings.Cut(string(conf), "::")
sklarsa marked this conversation as resolved.
Show resolved Hide resolved
if !found {
return opts, NewConfigStrParseError("no schema separator found '::'")
}

schema := schemaType(schemaStr)
switch schema {
case schemaHttp:
opts = append(opts, WithHttp())
case schemaHttps:
opts = append(opts, WithHttp(), WithTls())
case schemaTcp:
opts = append(opts, WithTcp())
case schemaTcps:
opts = append(opts, WithTcp(), WithTls())
default:
return opts, NewConfigStrParseError("invalid schema %q", schema)
}

if len(conf) == 0 {
return opts, NewConfigStrParseError("'addr' key not found")
}

if strings.HasSuffix(conf, ";") {
hasTrailingSemicolon = true
} else {
conf = conf + ";" // add trailing semicolon
}

keyValueStr := []rune(conf)
for idx, rune := range keyValueStr {
if idx < len(conf)-1 {
nextRune = keyValueStr[idx+1]
} else {
nextRune = 0
}
switch rune {
case ';':
if isKey {
if nextRune == 0 && !hasTrailingSemicolon {
return opts, NewConfigStrParseError("unexpected end of string")
}
return opts, NewConfigStrParseError("invalid key character ';'")
}

if !isEscaping && nextRune == ';' {
isEscaping = true
continue
}

if isEscaping {
value.WriteRune(rune)
isEscaping = false
continue
}

switch strings.ToLower(key.String()) {
case "addr":
opts = append(opts, WithAddress(value.String()))
case "user":
user = value.String()
if user != "" && pass != "" {
opts = append(opts, WithBasicAuth(user, pass))
}
if token != "" && user != "" {
opts = append(opts, WithAuth(user, token))
}
case "pass":
pass = value.String()
if user != "" && pass != "" {
opts = append(opts, WithBasicAuth(user, pass))
}
case "token":
token = value.String()
switch schema {
case schemaHttp, schemaHttps:
opts = append(opts, WithBearerToken(token))
case schemaTcp, schemaTcps:
if token != "" && user != "" {
opts = append(opts, WithAuth(user, token))
}
}
case "auto_flush":
if value.String() == "on" {
return opts, NewConfigStrParseError("auto_flush option is not supported")
sklarsa marked this conversation as resolved.
Show resolved Hide resolved
}
case "auto_flush_rows", "auto_flush_bytes":
return opts, NewConfigStrParseError("auto_flush option is not supported")
case "min_throughput", "init_buf_size", "max_buf_size":
parsedVal, err := strconv.Atoi(value.String())
if err != nil {
return opts, NewConfigStrParseError("invalid %s value, %q is not a valid int", key, value.String())

}
switch key.String() {
case "min_throughput":
opts = append(opts, WithMinThroughput(parsedVal))
case "init_buf_size":
opts = append(opts, WithInitBufferSize(parsedVal))
case "max_buf_size":
opts = append(opts, WithBufferCapacity(parsedVal))
default:
panic("add a case for " + key.String())
}

case "grace_timeout", "retry_timeout":
timeout, err := strconv.Atoi(value.String())
if err != nil {
return opts, NewConfigStrParseError("invalid %s value, %q is not a valid int", key, value)
}

timeoutDur := time.Duration(timeout * int(time.Millisecond))

switch key.String() {
case "grace_timeout":
opts = append(opts, WithGraceTimeout(timeoutDur))
case "retry_timeout":
opts = append(opts, WithRetryTimeout(timeoutDur))
default:
panic("add a case for " + key.String())
}
case "tls_verify":
switch value.String() {
case "on":
opts = append(opts, WithTls())
case "unsafe_off":
opts = append(opts, WithTlsInsecureSkipVerify())
default:
return opts, NewConfigStrParseError("invalid tls_verify value, %q is not 'on' or 'unsafe_off", value)
}
case "tls_roots":
return opts, NewConfigStrParseError("tls_roots is not available in the go client")
case "tls_roots_password":
return opts, NewConfigStrParseError("tls_roots_password is not available in the go client")
default:
return opts, NewConfigStrParseError("unsupported option %q", key)
}

key.Reset()
value.Reset()
isKey = true
case '=':
if isKey {
isKey = false
} else {
value.WriteRune(rune)
}
default:
if isKey {
key.WriteRune(rune)
} else {
value.WriteRune(rune)
}
}
}

if isEscaping {
return opts, NewConfigStrParseError("unescaped ';'")
}

return opts, nil
}
Loading
Loading