Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: evaluator v2 #909

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

feat: evaluator v2 #909

wants to merge 5 commits into from

Conversation

i4ki
Copy link
Contributor

@i4ki i4ki commented Apr 22, 2023

What this PR does / why we need it:

This PR introduces lazy evaluation to the globals and lets blocks, then for each stack, not used variables are not evaluated.

We need this because for the case of importing lots of files, with thousand stacks, Terramate wastes a lot of time evaluating not used variables.
This implementation gives 60-90% performance gain but it heavily depends on the use case. For the case where all stacks requires all variables, then it's slightly slower (around 5% in simple tests).

Which issue(s) this PR fixes:

Special notes for your reviewer:

TODO

Does this PR introduce a user-facing change?

yes, but details are yet to be documented.

Checklist

  • Implement nested object key evaluation.
  • Documentation.
  • Decide about the non-breaking changes (some tests are disabled waiting for the decision)
  • Decide about the breaking changes.
  • In-depth analysis and testing on real-world projects are required.

@terramate-io terramate-io deleted a comment from github-actions bot Apr 24, 2023
@netlify
Copy link

netlify bot commented Apr 26, 2023

Deploy Preview for terramate-io-docs canceled.

Name Link
🔨 Latest commit f144098
🔍 Latest deploy log https://app.netlify.com/sites/terramate-io-docs/deploys/644dcccd6d955800078bfff4

@codecov
Copy link

codecov bot commented Apr 26, 2023

Codecov Report

Patch coverage: 80.62% and project coverage change: +0.28 🎉

Comparison is base (a4fcf6e) 71.28% compared to head (f144098) 71.56%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #909      +/-   ##
==========================================
+ Coverage   71.28%   71.56%   +0.28%     
==========================================
  Files          84       84              
  Lines       13333    13443     +110     
==========================================
+ Hits         9504     9621     +117     
+ Misses       3491     3469      -22     
- Partials      338      353      +15     
Flag Coverage Δ
tests 71.56% <80.62%> (+0.28%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
cmd/terramate/cli/cli.go 0.00% <0.00%> (ø)
cmd/terramate/cli/project.go 0.00% <ø> (ø)
project/project.go 50.66% <ø> (ø)
stdlib/funcs.go 54.13% <9.09%> (-2.79%) ⬇️
hcl/ast/expr.go 94.94% <15.38%> (-1.92%) ⬇️
stack/manager.go 56.88% <50.00%> (ø)
test/hclwrite/hclwrite.go 90.82% <66.66%> (-1.77%) ⬇️
hcl/eval/ref.go 74.50% <74.50%> (ø)
test/hcl.go 78.88% <75.00%> (+0.23%) ⬆️
lets/lets.go 74.13% <75.51%> (-16.09%) ⬇️
... and 15 more

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@github-actions
Copy link

github-actions bot commented Apr 26, 2023

metric: time/op
CloudReadLines-4: old 1.02ms ± 5%: new 1.01ms ± 4%: delta: 0.00%
CloudReadLine-4: old 7.41ms ± 1%: new 6.99ms ± 1%: delta: -5.70%
ListFiles-4: old 52.2µs ± 1%: new 52.2µs ± 1%: delta: 0.00%
Generate-4: old 2.53s ± 2%: new 0.03s ± 1%: delta: -98.95%
GenerateRegex-4: old 1.72s ± 1%: new 0.05s ± 1%: delta: -97.06%
TokensForExpressionComplex-4: old 1.26ms ± 0%: new 1.27ms ± 0%: delta: 0.39%
TokensForExpressionPlainStringNoNewline-4: old 893ns ± 1%: new 924ns ± 1%: delta: 3.55%
TokensForExpressionStringWith100Newlines-4: old 21.8µs ± 1%: new 22.5µs ± 1%: delta: 3.35%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 1.45ms ± 0%: new 1.45ms ± 0%: delta: 0.00%
TokensForExpression-4: old 1.27ms ± 0%: new 1.27ms ± 0%: delta: 0.30%
PartialEvalComplex-4: old 434µs ± 1%: new 937µs ± 1%: delta: 116.00%
PartialEvalSmallString-4: old 3.62µs ± 0%: new 3.94µs ± 0%: delta: 8.81%
PartialEvalHugeString-4: old 1.87ms ± 0%: new 2.02ms ± 1%: delta: 8.04%
PartialEvalHugeInterpolatedString-4: old 4.90ms ± 1%: new 8.94ms ± 1%: delta: 82.38%
PartialEvalObject-4: old 21.9µs ± 1%: new 37.6µs ± 1%: delta: 72.05%
TmAllTrueLiteralList-4: old 6.27ms ± 0%: new 6.41ms ± 0%: delta: 2.15%
TmAllTrueFuncall-4: old 161µs ± 0%: new 166µs ± 0%: delta: 3.60%
TmAnyTrueLiteralList-4: old 147ms ± 0%: new 150ms ± 0%: delta: 1.86%
TmAnyTrueFuncall-4: old 161µs ± 0%: new 167µs ± 0%: delta: 3.39%
TmTernary-4: old 2.80µs ± 1%: new 3.67µs ± 0%: delta: 31.07%
TmTry-4: old 52.0µs ± 0%: new 48.3µs ± 0%: delta: -7.09%
check failed: time/op=+20%
metric: alloc/op
CloudReadLines-4: old 3.12MB ± 0%: new 3.12MB ± 0%: delta: 0.00%
CloudReadLine-4: old 3.37MB ± 0%: new 3.37MB ± 0%: delta: 0.00%
ListFiles-4: old 22.0kB ± 0%: new 22.0kB ± 0%: delta: 0.00%
Generate-4: old 2.32GB ± 0%: new 0.02GB ± 0%: delta: -99.13%
GenerateRegex-4: old 955MB ± 0%: new 33MB ± 0%: delta: -96.53%
TokensForExpressionComplex-4: old 412kB ± 0%: new 412kB ± 0%: delta: 0.00%
TokensForExpressionPlainStringNoNewline-4: old 592B ± 0%: new 592B ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-4: old 12.4kB ± 0%: new 12.4kB ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 402kB ± 0%: new 402kB ± 0%: delta: 0.00%
TokensForExpression-4: old 412kB ± 0%: new 412kB ± 0%: delta: 0.00%
PartialEvalComplex-4: old 356kB ± 0%: new 676kB ± 0%: delta: 90.06%
PartialEvalSmallString-4: old 1.74kB ± 0%: new 1.74kB ± 0%: delta: 0.00%
PartialEvalHugeString-4: old 166kB ± 0%: new 166kB ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-4: old 4.38MB ± 0%: new 8.06MB ± 0%: delta: 84.10%
PartialEvalObject-4: old 20.4kB ± 0%: new 31.6kB ± 0%: delta: 54.88%
TmAllTrueLiteralList-4: old 1.74MB ± 0%: new 1.74MB ± 0%: delta: 0.09%
TmAllTrueFuncall-4: old 45.5kB ± 0%: new 46.6kB ± 0%: delta: 2.35%
TmAnyTrueLiteralList-4: old 37.9MB ± 0%: new 37.9MB ± 0%: delta: 0.00%
TmAnyTrueFuncall-4: old 45.6kB ± 0%: new 46.7kB ± 0%: delta: 2.35%
TmTernary-4: old 1.20kB ± 0%: new 1.60kB ± 0%: delta: 33.33%
TmTry-4: old 11.2kB ± 0%: new 10.2kB ± 0%: delta: -9.05%
metric: allocs/op
CloudReadLines-4: old 5.54k ± 0%: new 5.54k ± 0%: delta: 0.00%
CloudReadLine-4: old 60.0k ± 0%: new 60.0k ± 0%: delta: 0.00%
ListFiles-4: old 321 ± 0%: new 321 ± 0%: delta: 0.00%
Generate-4: old 25.9M ± 0%: new 0.1M ± 0%: delta: -99.56%
GenerateRegex-4: old 18.6M ± 0%: new 0.2M ± 0%: delta: -98.81%
TokensForExpressionComplex-4: old 4.93k ± 0%: new 4.93k ± 0%: delta: 0.00%
TokensForExpressionPlainStringNoNewline-4: old 21.0 ± 0%: new 21.0 ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-4: old 228 ± 0%: new 228 ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 3.29k ± 0%: new 3.29k ± 0%: delta: 0.00%
TokensForExpression-4: old 4.93k ± 0%: new 4.93k ± 0%: delta: 0.00%
PartialEvalComplex-4: old 2.86k ± 0%: new 6.39k ± 0%: delta: 123.47%
PartialEvalSmallString-4: old 23.0 ± 0%: new 23.0 ± 0%: delta: 0.00%
PartialEvalHugeString-4: old 35.0 ± 0%: new 35.0 ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-4: old 23.1k ± 0%: new 61.1k ± 0%: delta: 164.80%
PartialEvalObject-4: old 125 ± 0%: new 245 ± 0%: delta: 96.00%
TmAllTrueLiteralList-4: old 13.6k ± 0%: new 13.7k ± 0%: delta: 0.38%
TmAllTrueFuncall-4: old 460 ± 0%: new 494 ± 0%: delta: 7.39%
TmAnyTrueLiteralList-4: old 252k ± 0%: new 252k ± 0%: delta: 0.02%
TmAnyTrueFuncall-4: old 462 ± 0%: new 496 ± 0%: delta: 7.36%
TmTernary-4: old 28.0 ± 0%: new 42.0 ± 0%: delta: 50.00%
TmTry-4: old 147 ± 0%: new 147 ± 0%: delta: 0.00%
check failed: allocs/op=+20%

@netlify
Copy link

netlify bot commented Jun 11, 2023

Deploy Preview for terramate-io-docs canceled.

Name Link
🔨 Latest commit 06c6a55
🔍 Latest deploy log https://app.netlify.com/sites/terramate-io-docs/deploys/64ff08232ab8cf0007bea71a

@codecov-commenter
Copy link

codecov-commenter commented Jun 11, 2023

Codecov Report

Attention: Patch coverage is 79.07180% with 239 lines in your changes are missing coverage. Please review.

Project coverage is 64.19%. Comparing base (5a4d973) to head (06c6a55).
Report is 1059 commits behind head on main.

❗ Current head 06c6a55 differs from pull request most recent head c15bcfa. Consider uploading reports for the commit c15bcfa to get more accurate results

Files Patch % Lines
cmd/terramate/cli/cli.go 0.00% 71 Missing ⚠️
stdlib/funcs.go 8.57% 32 Missing ⚠️
hcl/eval/eval.go 90.00% 20 Missing and 7 partials ⚠️
hcl/eval/ref.go 75.00% 21 Missing and 5 partials ⚠️
hcl/eval/stmt.go 83.33% 22 Missing and 4 partials ⚠️
globals/globals_resolver.go 87.39% 9 Missing and 6 partials ⚠️
lets/lets.go 75.51% 7 Missing and 5 partials ⚠️
hcl/ast/expr.go 15.38% 10 Missing and 1 partial ⚠️
generate/genhcl/genhcl.go 94.49% 5 Missing and 1 partial ⚠️
run/env.go 93.02% 2 Missing and 1 partial ⚠️
... and 6 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #909      +/-   ##
==========================================
+ Coverage   62.75%   64.19%   +1.44%     
==========================================
  Files         102      100       -2     
  Lines       16395    16058     -337     
==========================================
+ Hits        10288    10309      +21     
+ Misses       5683     5326     -357     
+ Partials      424      423       -1     
Flag Coverage Δ
tests 64.19% <79.07%> (+1.44%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

netlify bot commented Dec 5, 2023

Deploy Preview for docs-terramate-io canceled.

Name Link
🔨 Latest commit 6ff219d
🔍 Latest deploy log https://app.netlify.com/sites/docs-terramate-io/deploys/657070ffdeb4390008ab749d

@i4ki i4ki changed the title feat: globals v2 feat: evaluator v2 Dec 5, 2023
Signed-off-by: Tiago Natel <[email protected]>
i4ki and others added 4 commits December 5, 2023 19:37
Some variables are scoped by their expression declaration, that's the
case for `[for ...]`, `{for ...}` and `tm_dynamic`.
In this case, the `tm_ternary()` implementation must use the provided
closure evaluator variables instead of the outer Terramate evaluator.

Signed-off-by: Tiago Natel <[email protected]>
Co-authored-by: Sebastian <[email protected]>
## What this PR does / why we need it:

Some variables are scoped by their expression declaration, that's the
case for `[for ...]`, `{for ...}` and `tm_dynamic`. In this case, the
`tm_ternary()` implementation must use the provided closure evaluator
variables instead of the outer Terramate evaluator.

This fixes an issue where `tm_ternary()` reported the error below:
```
eval expression: evaluating global["val"] from /stack scope: Call to function "tm_ternary" failed: /sandbox/stack/terramate.tm.hcl:5,14-15: partial evaluation failed: evaluating tm_ternary branch: eval expression: There is no variable named "a".
```

## Which issue(s) this PR fixes:

## Special notes for your reviewer:

- This is a fix for PR #909 
- This issue was reported by @zied-elouaer and reproduced in a customer
repository.
- It was fixed in the Hackathon but just now isolated into a separate
PR.

## Does this PR introduce a user-facing change?
```
no
```
evalctx.SetEnv(os.Environ())
evalctx := eval.New(st.Dir, globalsResolver, runtime.NewResolver(root, st))
evalctx.SetFunctions(stdlib.Functions(evalctx, st.HostDir(root)))
//evalctx.SetEnv(os.Environ())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be fixed later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants