Remove graph dependency from src/constraints/consumer.jl

[abelsiqueira]

Part of #942
Closes #943

[github-actions]

Benchmark Results

	b34be57...	26a9cd9...	b34be57.../26a9cd91e9004a...
energy_problem/create_model	41.3 ± 1 s	43.9 ± 1.6 s	0.941
energy_problem/input_and_constructor	15.3 ± 0.24 s	18 ± 0.19 s	0.851
time_to_load	4.2 ± 0.015 s	4.27 ± 0.03 s	0.983

	b34be57...	26a9cd9...	b34be57.../26a9cd91e9004a...
energy_problem/create_model	0.614 G allocs: 21.3 GB	0.61 G allocs: 21.3 GB	1
energy_problem/input_and_constructor	0.0436 G allocs: 1.71 GB	0.0532 G allocs: 1.98 GB	0.866
time_to_load	0.159 k allocs: 11.2 kB	0.159 k allocs: 11.2 kB	1

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.88%. Comparing base (b34be57) to head (26a9cd9).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #944      +/-   ##
==========================================
- Coverage   94.95%   94.88%   -0.07%     
==========================================
  Files          29       29              
  Lines        1070     1075       +5     
==========================================
+ Hits         1016     1020       +4     
- Misses         54       55       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[abelsiqueira]

@datejada, I've updated this to follow the new strategy. The code is still slow, though, so we might want to wait until all rep_periods_profiles are removed to see if it's possible to make it better.

[abelsiqueira]

@datejada and I evaluated the 3 options with DuckDB 1.1.2 locally (1-Using graph, 2-Using full DuckDB, which is the version from 2 weeks ago, and 3-Using the new profiles object). Here are the results:

Option 1 - Using graph[a].profiles_rep_periods

(1/1) benchmarking "energy_problem"...
  (1/2) benchmarking "input_and_constructor"...
  done (took 29.587801709 seconds)
  (2/2) benchmarking "create_model"...
  done (took 139.582342084 seconds)
done (took 172.174956 seconds)
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "energy_problem" => 2-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "input_and_constructor" => Trial(6.525 s)
          "create_model" => Trial(33.750 s)


Option 2 - Remove graph[a].profiles_rep_periods and compute aggregation with DuckDB

(1/1) benchmarking "energy_problem"...
  (1/2) benchmarking "input_and_constructor"...
  done (took 28.673210792 seconds)
  (2/2) benchmarking "create_model"...
  done (took 144.815846917 seconds)
done (took 175.86473875 seconds)
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "energy_problem" => 2-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "input_and_constructor" => Trial(6.508 s)
          "create_model" => Trial(34.858 s)


Option 3 - Change graph[a].profiles_rep_periods[...] to get profile_name with DuckDB and create profiles structure

(1/1) benchmarking "energy_problem"...
  (1/2) benchmarking "input_and_constructor"...
  done (took 34.309520792 seconds)
  (2/2) benchmarking "create_model"...
  done (took 127.688325292 seconds)
done (took 164.306446625 seconds)
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "energy_problem" => 2-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "input_and_constructor" => Trial(7.747 s)
          "create_model" => Trial(30.999 s)

Option 3 was a bit faster, and should be reasonable to maintain, so we're using it for now

[abelsiqueira]

For future reference, this is the DuckDB query to join and aggregate the profiles (option 2):

         SELECT
             cons.*,
             ANY_VALUE(asset.type) AS type,
             ANY_VALUE(asset.consumer_balance_sense) AS consumer_balance_sense,
             ANY_VALUE(asset_milestone.peak_demand) AS peak_demand,
             COALESCE(MEAN(profile.value), 1) AS demand_agg
         FROM $cons AS cons
         LEFT JOIN asset
             ON cons.asset = asset.asset
         LEFT JOIN asset_milestone
             ON cons.asset = asset_milestone.asset
             AND cons.year = asset_milestone.milestone_year
         LEFT JOIN assets_profiles
             ON cons.asset = assets_profiles.asset
             AND cons.year = assets_profiles.commission_year
             AND assets_profiles.profile_type = 'demand'
         LEFT JOIN profiles_rep_periods AS profile
             ON profile.profile_name = assets_profiles.profile_name
             AND cons.year = profile.year
             AND cons.rep_period = profile.rep_period
             AND cons.time_block_start <= profile.timestep
             AND profile.timestep <= cons.time_block_end
         GROUP BY cons.*
         ORDER BY cons.index -- order is important
         

[datejada]

Reviewed during an online meeting