Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V2 #164

Merged
merged 164 commits into from
Mar 11, 2024
Merged

V2 #164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
164 commits
Select commit Hold shift + click to select a range
c177d2d
V2 refactoring
kke Jan 23, 2024
caf8196
Eliminate exec/errors
kke Jan 30, 2024
98adbfd
Ditch pkg/
kke Jan 30, 2024
b4bceeb
Reduce configurer functions
kke Jan 30, 2024
7eba19d
Deterministic initsystem resolving
kke Jan 30, 2024
eca6d2c
Ditch configurer.InstallPackage in favor of pluggable package managers
kke Jan 30, 2024
6d7289b
Get rid of "configurers", implement all in fsys
kke Jan 31, 2024
f7a2014
A round of linting
kke Jan 31, 2024
b088719
Universal package manager
kke Jan 31, 2024
851abc2
A round of linting
kke Jan 31, 2024
909ed7e
Something removed stripansi dependency(?)
kke Jan 31, 2024
acfad65
Fix tests, ditch atomic.Value
kke Feb 1, 2024
4361a59
Add missing files
kke Feb 1, 2024
dd1c13a
Fix nsec touch fallback
kke Feb 1, 2024
61f40b9
A round of linting
kke Feb 1, 2024
5ff64cc
Fix stream passing
kke Feb 1, 2024
9776141
Use Get-Content to read content
kke Feb 1, 2024
1d2e7d3
Silence rigrcp binary logs
kke Feb 2, 2024
d41836f
Implement service log reading
kke Feb 2, 2024
09297de
Different readfile
kke Feb 2, 2024
15667a3
Use regular read/write for windows writefile/readfile
kke Feb 2, 2024
d6be84d
Improve chtimes
kke Feb 2, 2024
cc709d7
Correct precision for windows
kke Feb 2, 2024
fd23717
Add TruncateTime and improve equality comparison
kke Feb 2, 2024
3b27715
Add a warning about softtime false positives
kke Feb 5, 2024
6e7d983
Refactor checkpoint, go vet passes
kke Feb 6, 2024
7a06f88
Improve connection creation
kke Feb 6, 2024
4bd3560
Improve init and injection
kke Feb 7, 2024
c292986
Split workflows
kke Feb 7, 2024
59930fc
Why is the host key mismatching now, i dont know
kke Feb 7, 2024
bc52bb4
No strict
kke Feb 7, 2024
d7f5b43
Add openssh control master logging
kke Feb 7, 2024
dd63d52
A round of linting
kke Feb 7, 2024
255ff13
Move functions to connection
kke Feb 8, 2024
10018b0
Rename softtime to stattime
kke Feb 9, 2024
a51902d
More repos
kke Feb 9, 2024
4336e9e
Renew logging
kke Feb 9, 2024
def684a
Init runner
kke Feb 9, 2024
5ad6bdd
Nilchecks
kke Feb 9, 2024
da50c73
Tweak injectables initialization
kke Feb 10, 2024
c163ee9
Another round of linting
kke Feb 10, 2024
9f02bb7
Rename repo => provider, injectables => dependencies
kke Feb 11, 2024
f5a4e34
Rename rigfs => remotefs and all Fsys to FS
kke Feb 11, 2024
7aed357
A round of linting
kke Feb 11, 2024
1364f44
Rename Repository => Provider
kke Feb 11, 2024
7c43bf7
Add top package getters for subsystems
kke Feb 11, 2024
5d58317
Return slash-separated paths from windows too
kke Feb 12, 2024
d75e9aa
Improve the client interface
kke Feb 12, 2024
223ecef
Add a testutil package rigtest
kke Feb 12, 2024
3e2a98b
"Providerize" OSRelease getter
kke Feb 12, 2024
c41b7e7
Add a toplevel GetService
kke Feb 12, 2024
ad8af06
Rename OSRelease => os.Release
kke Feb 12, 2024
60bd16e
Add protocol constructors
kke Feb 12, 2024
084c5fc
Rename OSRelease => os.Release (fixup)
kke Feb 12, 2024
d0fb703
Add top level rig.GetFS
kke Feb 12, 2024
ae609a7
Exec writer doc comments
kke Feb 12, 2024
45d9b67
Runner tests
kke Feb 12, 2024
d7472eb
Another round of linting fixes
kke Feb 12, 2024
8f4808b
Small visibity fixes and function renames
kke Feb 13, 2024
a4a3eff
Move protocol implementations into their own dirs
kke Feb 13, 2024
f5ed5cb
Dont try to unmarshal Connection automatically, user should DIY
kke Feb 13, 2024
c5ce13b
Provide v0 like serializable DefaultConnection that uses defaults
kke Feb 13, 2024
bda8ac5
Another round of linting fixes
kke Feb 14, 2024
5799359
Rename Client to Protocol
kke Feb 14, 2024
7e66f71
Rename Connection to Client
kke Feb 14, 2024
e5471ce
Rename ClientConfigurer to ProtocolConfigurer
kke Feb 14, 2024
c95a531
Rename ClientConfig to CompositeConfig
kke Feb 14, 2024
7afc1d0
Rename Protocol to Connection and adapt implementations to match
kke Feb 14, 2024
c93666c
Split protocols into connection.go and config.go
kke Feb 14, 2024
520fae7
Rename WithClient to WithConnection
kke Feb 14, 2024
137f568
Remove exec.Confirm
kke Feb 14, 2024
db55dab
Eliminate localhost.Config, just use bool
kke Feb 14, 2024
8018040
Fix ssh homedir.Expand usage
kke Feb 14, 2024
dd0dbf5
Rename rig.NewConnection to rig.NewClient
kke Feb 14, 2024
e48775c
Reorder fields
kke Feb 14, 2024
ab568b7
Dont memoize String
kke Feb 14, 2024
b2219de
A round of linting
kke Feb 14, 2024
9570341
Adjust interface returning
kke Feb 14, 2024
209a1db
Doccomment updates
kke Feb 14, 2024
c746e1d
Fix bastion
kke Feb 14, 2024
d4c6f21
Rename MockClient to MockConnection
kke Feb 14, 2024
e098947
Eliminate errors.go
kke Feb 15, 2024
4471355
Doc comment updates
kke Feb 15, 2024
83200a6
Finetune homedir.Expand
kke Feb 15, 2024
a3a2645
Implement a generic Service type and adapt subsystems
kke Feb 18, 2024
cb0a915
Generic provider
kke Feb 19, 2024
ae7f125
A round of linting
kke Feb 19, 2024
9419300
Move abort.ErrAbort to protocol.ErrAbort
kke Feb 19, 2024
24a0c10
Fix data race in DefaultClient
kke Feb 19, 2024
83c37eb
Minor renames and doc comment updates
kke Feb 19, 2024
bc306ae
Doc comment updates
kke Feb 19, 2024
53eff67
Options validation
kke Feb 19, 2024
5931d0b
Rename Options to ClientOptions
kke Feb 20, 2024
559aa8a
Rename DefaulOpenSSHOptions to DefaultOptionArguments
kke Feb 20, 2024
390b7c6
SSH signals todo notes
kke Feb 20, 2024
af8f8ac
Rename services.go to aliases.go
kke Feb 20, 2024
1355a5d
Expose less errors
kke Feb 20, 2024
c321b22
Add tests for plumbing
kke Feb 20, 2024
0093e83
Implement shellescape locally
kke Feb 20, 2024
8516b8c
Drop creasty/defaults
kke Feb 20, 2024
fb77c34
Remove sprintf-formatting from execs and ensure unprefixed windows co…
kke Feb 20, 2024
1c4e350
Add exec.Command for formatting commands
kke Feb 20, 2024
cbe7c08
Check for printf-errors in commands, refuse to exec.
kke Feb 20, 2024
1a61e7e
Accept anything.exe as cmd prefix in windows
kke Feb 21, 2024
798b026
Move shellescape to sh/shellescape
kke Feb 21, 2024
4170a97
Move exec.Command to sh.Command and implement CommandBuilder
kke Feb 21, 2024
1cbb2f5
Use new sh.Command for command generation
kke Feb 21, 2024
49de9f3
Prematurely optimize ReadDir functions
kke Feb 22, 2024
272fac3
Minor optimizations and beautifications
kke Feb 22, 2024
3a0bffc
Fix os-release extra fields parsing test
kke Feb 22, 2024
1d033ef
Remove the osversion.go after rebase
kke Feb 22, 2024
88afb9b
Copy & paste failures in package managers
kke Feb 22, 2024
1fcc9d2
Make mock starter matcher order deterministic
kke Feb 22, 2024
470303e
Files sudo.go and doas.go had opposite contents
kke Feb 22, 2024
61c1a6a
Add tests for top level getters
kke Feb 22, 2024
5f56cc3
Implement ScannerRunner for bufio.Scanning command output
kke Feb 22, 2024
9a657b0
Finetune string builders
kke Feb 22, 2024
b0f1b88
A round of linting
kke Feb 22, 2024
2403981
Improve the rigtest mocking tools
kke Feb 23, 2024
ae7e5c9
Implement rigtest assertions
kke Feb 23, 2024
a0902ba
Use the new rigtest features in tests
kke Feb 23, 2024
f4a550f
Make homedir_test windows compatible
kke Feb 23, 2024
ec7da7b
Doc comment typos
kke Feb 23, 2024
61076fd
Move exec.HostRunner to rig.Executor
kke Feb 26, 2024
673fab0
Move exec.Options to rig.ExecOptions
kke Feb 26, 2024
3f9265a
Use pool for shellescape builders
kke Feb 26, 2024
29441c6
Implement posix-like shellescape.Unquote
kke Feb 26, 2024
3e98a67
Implement kv.Split for splitting key-value pairs
kke Feb 26, 2024
1fec221
Implement kv.Decoder, an encoding/json like KEY=VALUE decoder
kke Feb 26, 2024
61e2183
Relax linting rules
kke Feb 26, 2024
ef17899
Byteslice and redaction tools
kke Mar 1, 2024
742fe8c
Rename exec to cmd, redo logging
kke Mar 5, 2024
d35ff56
Log attrs
kke Mar 5, 2024
49a5afa
Better logger initialization to avoid duplicate keys
kke Mar 5, 2024
ca92744
More logger initialization
kke Mar 5, 2024
9e70062
Unknown why logger is quiet in tests
kke Mar 5, 2024
a7c1d24
Various fixes to make unit tests pass again
kke Mar 6, 2024
e283385
Reduce excessive trace logging
kke Mar 6, 2024
f1350a7
A round of linting
kke Mar 7, 2024
b890c42
Add an example for ScanWriter
kke Mar 7, 2024
acd9148
Remove test for a removed feature
kke Mar 7, 2024
8687c79
Fix kv-decoder error logic
kke Mar 7, 2024
e4d061f
OSRelease IDLike now a slice
kke Mar 7, 2024
43c479a
Fixes for failing tests
kke Mar 7, 2024
b746968
Fix logger injection
kke Mar 7, 2024
d3b57db
Double HostAttr from executor logging
kke Mar 7, 2024
6c358d4
Use bufio.Scanner internally in ScanWriter
kke Mar 8, 2024
f766aff
Use the iostream types in exec logging
kke Mar 8, 2024
d8b6c7a
Remove the [SSH] type protocol prefixes from host strings
kke Mar 8, 2024
19a3caa
Move bytecounter to iostream
kke Mar 8, 2024
61c89dc
Switch all to k0sproject/rig/v2
kke Mar 8, 2024
ac06ac3
Update readme with v2 wip note
kke Mar 8, 2024
a4b4df0
Executor test was not in cmd/
kke Mar 8, 2024
e6ab22a
Include stderr snippets in exec failure errors
kke Mar 8, 2024
af0a62a
Typo in filename
kke Mar 8, 2024
1a8c643
Small doc tweaks
kke Mar 8, 2024
1fe62da
Implement retry locally
kke Mar 10, 2024
4008afb
Connect() is now Connect(ctx) with auto retry
kke Mar 10, 2024
12457b3
Allow disabling Connect auto-retry
kke Mar 10, 2024
7f1c008
Move protocol implementations under protocol/
kke Mar 10, 2024
c33959c
Fix go.sum after rebase
kke Mar 10, 2024
1c7c60f
Doc touch up and a couple of examples
kke Mar 10, 2024
87043bb
Additional README for shellescape
kke Mar 10, 2024
f1aa362
A round of linting
kke Mar 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .github/logo.webp
Binary file not shown.
105 changes: 7 additions & 98 deletions .github/workflows/go.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,113 +4,22 @@ on: [pull_request]

jobs:

unit-linux:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod

- name: Test
run: |
go mod download
go test -v ./...

integration-linux:
unit-test:
strategy:
fail-fast: false
matrix:
image:
- quay.io/k0sproject/bootloose-ubuntu20.04
- quay.io/k0sproject/bootloose-debian12
- quay.io/k0sproject/bootloose-alpine3.18
needs: unit-linux
runs-on: ubuntu-20.04
runs-on:
- ubuntu-latest
- windows-latest
runs-on: ${{ matrix.runs-on }}
steps:
- uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod

- name: install test dependencies
run: |
sudo apt-get update
sudo apt-get install expect

- name: Run integration tests
env:
LINUX_IMAGE: ${{ matrix.image }}
run: |
cd test
go mod download
make test

windows:
runs-on: windows-2022
steps:
- name: Set up WinRM
shell: pwsh
run: |
Set-Item WSMan:\localhost\Service\AllowUnencrypted -Value $True
Get-ChildItem WSMan:\Localhost\listener | Remove-Item -Recurse

Set-Item -Path WSMan:\LocalHost\MaxTimeoutms -Value '1800000'
Set-Item -Path WSMan:\LocalHost\Shell\MaxMemoryPerShellMB -Value '1024'
Set-Item -Path WSMan:\LocalHost\Service\AllowUnencrypted -Value 'false'
Set-Item -Path WSMan:\LocalHost\Service\Auth\Basic -Value 'true'
Set-Item -Path WSMan:\LocalHost\Service\Auth\CredSSP -Value 'true'

New-NetFirewallRule -Name "WINRM-HTTPS-In-TCP" `
-DisplayName "Windows Remote Management (HTTPS-In)" `
-Description "Inbound rule for Windows Remote Management via WS-Management. [TCP 5986]" `
-Group "Windows Remote Management" `
-Program "System" `
-Protocol TCP `
-LocalPort "5986" `
-Action Allow `
-Profile Domain,Private

$Hostname = [System.Net.Dns]::GetHostByName((hostname)).HostName.ToUpper()
$pfx = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName $Hostname
$certThumbprint = $pfx.Thumbprint
$certSubjectName = $pfx.SubjectName.Name.TrimStart("CN = ").Trim()

New-Item -Path WSMan:\LocalHost\Listener -Address * -Transport HTTPS -Hostname $certSubjectName -CertificateThumbPrint $certThumbprint -Port "5986" -force

Restart-Service WinRM

$Username = "winrmuser"
$Password = ConvertTo-SecureString "Password123" -AsPlainText -Force
New-LocalUser -Name $Username -Password $Password -FullName "WinRM User" -Description "Local user for WinRM testing"
Add-LocalGroupMember -Group "Administrators" -Member $Username

$Credentials = New-Object System.Management.Automation.PSCredential ($Username, $Password)

# try it out:
$SessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$Session = New-PSSession -ComputerName "localhost" -Credential $Credentials -Authentication Basic -UseSSL:$true -SessionOption $SessionOptions

Invoke-Command -Session $Session -ScriptBlock { cmd.exe /c ver.exe }

- uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Unit test

- name: Test
run: |
go mod download
go test -v ./...

- name: Integration test
run: |
cd test
go mod download
go test -v ./ -args -protocol winrm -host 127.0.0.1 -port 5986 -user winrmuser -winrm-password Password123 -winrm-https
12 changes: 0 additions & 12 deletions .github/workflows/golangci-lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,18 +32,6 @@ jobs:
go-version-file: go.mod
check-latest: true

- name: Go modules cache
uses: actions/cache@v4
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/Library/Caches/go-build
%LocalAppData%\go-build
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-

- name: Check go.mod/go.sum to be consistent
run: go mod tidy -v && git diff --exit-code

Expand Down
99 changes: 99 additions & 0 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: Integration

on: [pull_request]

jobs:
integration-test-linux:
strategy:
fail-fast: false
matrix:
image:
- quay.io/k0sproject/bootloose-ubuntu20.04
- quay.io/k0sproject/bootloose-debian12
- quay.io/k0sproject/bootloose-alpine3.18
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod

- name: install test dependencies
run: |
sudo apt-get update
sudo apt-get install expect

- name: Run integration tests
env:
LINUX_IMAGE: ${{ matrix.image }}
run: |
cd test
go mod download
make test

windows:
runs-on: windows-2022
steps:
- name: Set up WinRM
shell: pwsh
run: |
Set-Item WSMan:\localhost\Service\AllowUnencrypted -Value $True
Get-ChildItem WSMan:\Localhost\listener | Remove-Item -Recurse

Set-Item -Path WSMan:\LocalHost\MaxTimeoutms -Value '1800000'
Set-Item -Path WSMan:\LocalHost\Shell\MaxMemoryPerShellMB -Value '1024'
Set-Item -Path WSMan:\LocalHost\Service\AllowUnencrypted -Value 'false'
Set-Item -Path WSMan:\LocalHost\Service\Auth\Basic -Value 'true'
Set-Item -Path WSMan:\LocalHost\Service\Auth\CredSSP -Value 'true'

New-NetFirewallRule -Name "WINRM-HTTPS-In-TCP" `
-DisplayName "Windows Remote Management (HTTPS-In)" `
-Description "Inbound rule for Windows Remote Management via WS-Management. [TCP 5986]" `
-Group "Windows Remote Management" `
-Program "System" `
-Protocol TCP `
-LocalPort "5986" `
-Action Allow `
-Profile Domain,Private

$Hostname = [System.Net.Dns]::GetHostByName((hostname)).HostName.ToUpper()
$pfx = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName $Hostname
$certThumbprint = $pfx.Thumbprint
$certSubjectName = $pfx.SubjectName.Name.TrimStart("CN = ").Trim()

New-Item -Path WSMan:\LocalHost\Listener -Address * -Transport HTTPS -Hostname $certSubjectName -CertificateThumbPrint $certThumbprint -Port "5986" -force

Restart-Service WinRM

$Username = "winrmuser"
$Password = ConvertTo-SecureString "Password123" -AsPlainText -Force
New-LocalUser -Name $Username -Password $Password -FullName "WinRM User" -Description "Local user for WinRM testing"
Add-LocalGroupMember -Group "Administrators" -Member $Username

$Credentials = New-Object System.Management.Automation.PSCredential ($Username, $Password)

# try it out:
$SessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$Session = New-PSSession -ComputerName "localhost" -Credential $Credentials -Authentication Basic -UseSSL:$true -SessionOption $SessionOptions

Invoke-Command -Session $Session -ScriptBlock { cmd.exe /c ver.exe }

- uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod

- name: Unit test
run: |
go test -v ./...

- name: Integration test
run: |
cd test
go mod download
go test -v ./ -args -protocol winrm -host 127.0.0.1 -port 5986 -user winrmuser -winrm-password Password123 -winrm-https

9 changes: 7 additions & 2 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ run:
linters:
enable-all: true
disable:
- nonamedreturns
- exhaustivestruct
- exhaustruct
- forbidigo
- gochecknoglobals
- gochecknoinits
- godot
- gochecknoglobals
- godox
- golint
- interfacer
Expand Down Expand Up @@ -56,6 +56,11 @@ linters-settings:
- mu sync.Mutex
- wg sync.WaitGroup
- h Host
- fs FS
- fs remotefs.FS
- sb strings.Builder
- sb *strings.Builder
- s string
cyclop:
max-complexity: 12

Expand Down
75 changes: 15 additions & 60 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,79 +1,34 @@
**Note:** The main branch contains a work in progress for rig v2. If you are looking for the stable version, please browse the tags of v0.x releases.

### Rig

[![GoDoc](https://godoc.org/github.com/k0sproject/rig?status.svg)](https://godoc.org/github.com/k0sproject/rig)
[![Go Report Card](https://goreportcard.com/badge/github.com/k0sproject/rig)](https://goreportcard.com/report/github.com/k0sproject/rig)
[![Build Status](https://travis-ci.com/k0sproject/rig.svg?branch=main)](https://travis-ci.com/k0sproject/rig)
[![codecov](https://codecov.io/gh/k0sproject/rig/branch/main/graph/badge.svg)](https://codecov.io/gh/k0sproject/rig)

<img src=".github/logo.webp" alt="Rig" width="200" align="left"/>

A golang package for adding multi-protocol connectivity and multi-os operation functionality to your application's Host objects.

#### Design goals

Rig's intention is to be easy to use and extend.

It should be easy to add support for new operating systems and to add new commands to the multi-os support mechanism without breaking go's type checking.

All of the relevant structs have YAML tags and default values to make unmarshaling from YAML configurations as easy as possible.
It should be easy to add support for new operating systems and to add new components to the multi-os support mechanism without breaking type safety and testability.

#### Protocols

Currently rig comes with the most common ways to connect to hosts:

- SSH for connecting to hosts that accept SSH connections. **Pageant**
or [**openssh agent**](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse)
- SSH for connecting to hosts that accept SSH connections. With ssh agent and config support and sane familiar defaults. Pageant
or [openssh agent](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse)
can be used on Windows.
- OpenSSH for connecting to hosts using the system's own openssh "ssh" executable and utilizing session multiplexing for performance.
- WinRM as an alternative to SSH for windows hosts (SSH works too)
- Local for treating the localhost as it was one of the remote hosts
- Localhost for treating the local host as it was a remote host using go's os/exec.

#### Usage

The intended way to use rig is to embed the `rig.Connection` struct into your own.

Example:

```go
package main

import "github.com/k0sproject/rig"

type host struct {
rig.Connection
}

func main() {
h := host{
connection: rig.Connection{
SSH: &rig.SSH{
Address: 10.0.0.1
}
}
}

if err := h.Connect(); err != nil {
panic(err)
}

output, err := h.ExecOutput("ls -al")
if err != nil {
panic(err)
}
println(output)
}
```

But of course you can use it directly on its own too:

```go
package main

import "github.com/k0sproject/rig"

func main() {
h := rig.Connection{
SSH: &rig.SSH{
Address: 10.0.0.1
}
}

if err := h.Connect(); err != nil {
panic(err)
}
}
```
TBD - for now see godoc, tests and sources.

See more usage examples in the [examples/](examples/) directory.
Loading