Inconsistent gix status performance with clean trees

[eatradish]

Hi all, I have written a git status plugin for Bash called bash-git-status, which previously leveraged git-status to detect and prompt users about a Git repository's status:

• Red if the repository contains uncommited changes.
• Purpose if the repository contains untracked files.
• Green if the repository is clean.

Now that gix status is ready, I followed starship#6476 to re-implement this function in hope of better performance. You may find the current implementation here:

AOSC-Dev/bash-git-status#1

However, I found that the performance improvements, while significant, is not consistently found in all cases. For instance, in a "clean" (unchanged) aosc-os-abbs copy, I observed an 1.3x improvement in response time; however, in a "clean" linux copy, the reponse time worsened by about 5x.

The benchmark results are as follows, using

hyperfine -i --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' '/home/saki/bash-git-status/target/release/bash-git-status' 'bash-git-status'

linux get 5x slower with unchanged or untracked:

Benchmark 1: /home/saki/bash-git-status/target/release/bash-git-status
  Time (mean ± σ):      1.567 s ±  0.009 s    [User: 0.185 s, System: 1.061 s]
  Range (min … max):    1.547 s …  1.580 s    10 runs

  Warning: Ignoring non-zero exit code.

Benchmark 2: bash-git-status
  Time (mean ± σ):     316.9 ms ±   8.2 ms    [User: 108.7 ms, System: 699.8 ms]
  Range (min … max):   307.1 ms … 332.7 ms    10 runs

  Warning: Ignoring non-zero exit code.

Summary
  bash-git-status ran
    4.95 ± 0.13 times faster than /home/saki/bash-git-status/target/release/bash-git-status

changed:

Benchmark 1: /home/saki/bash-git-status/target/release/bash-git-status
  Time (mean ± σ):      87.1 ms ±   2.1 ms    [User: 24.4 ms, System: 153.6 ms]
  Range (min … max):    84.1 ms …  91.1 ms    10 runs

  Warning: Ignoring non-zero exit code.

Benchmark 2: bash-git-status
  Time (mean ± σ):     314.9 ms ±  10.1 ms    [User: 110.4 ms, System: 683.4 ms]
  Range (min … max):   299.4 ms … 327.3 ms    10 runs

  Warning: Ignoring non-zero exit code.

Summary
  /home/saki/bash-git-status/target/release/bash-git-status ran
    3.62 ± 0.15 times faster than bash-git-status

---

Test Enviromment:

• OS: AOSC OS (Linux Kernel 6.12.9)
• CPU: AMD Ryzen 8845H @ 5.14Ghz (8c16t)
• RAM: 32GiB
• Storage: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less)

[Byron]

Thanks a lot for reporting, these are interesting findings!

Admittedly, I never dropped caches before when testing, and maybe that makes all the difference. I did, however, notice occasional very large slowdowns, maybe in the same range even.

Ideally there was a way to reproduce this on MacOS as well so it can be profiled more easily.

Something I'd find interesting is if you'd get similar results without dropping caches.

[Byron]

It looks like sometimes the cache isn't active which is when the value goes up to 1.5s like before. But even on the lowest end the performance isn't any better than Git, but a little slower.

There is one more thing that could be tried.

cargo build --release --bin gix --no-default-features --features small

This binary should have gix status as well, but it runs a less parallel version, maybe even without using any threads, I don't know. Ideally that is more consistent, but is also expected to be much slower than Git, particularly if it doesn't use threads to do the index-worktree comparison.

Let me turn this into an issue in any case.

[eatradish]

@Byron Hi, I tried comparing gix status with git status --porcelain with the above command (I haven't looked closely at the code, it should be equivalent?). The result is as follows: (test https://github.com/torvalds/linux)

Benchmark 1: ../gitoxide/target/release/gix status
  Time (mean ± σ):     566.1 ms ± 429.7 ms    [User: 147.4 ms, System: 312.9 ms]
  Range (min … max):   405.7 ms … 1788.3 ms    10 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (1.788 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Benchmark 2: git status --porcelain
  Time (mean ± σ):     130.7 ms ±   5.6 ms    [User: 65.9 ms, System: 194.5 ms]
  Range (min … max):   123.3 ms … 142.0 ms    21 runs

Summary
  git status --porcelain ran
    4.33 ± 3.29 times faster than ../gitoxide/target/release/gix status

with --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches':

Benchmark 1: ../gitoxide/target/release/gix status
  Time (mean ± σ):      3.064 s ±  0.023 s    [User: 0.238 s, System: 1.168 s]
  Range (min … max):    3.033 s …  3.104 s    10 runs

Benchmark 2: git status --porcelain
  Time (mean ± σ):     299.3 ms ±  14.8 ms    [User: 100.6 ms, System: 686.8 ms]
  Range (min … max):   272.9 ms … 319.3 ms    10 runs

Summary
  git status --porcelain ran
   10.23 ± 0.51 times faster than ../gitoxide/target/release/gix status

[Byron]

Thank you.

If this is the single-threaded version of gix status, then the result is also not at all what I thought it would be.
Also, the system times of gix status are 50% higher or nearly twice as high when dropping caches. User time is also way higher compared to Git.

When running this in MacOS on Linux, I get quite different results.

❯ hyperfine -N -w1 'gix status' 'git status'
Benchmark 1: gix status
  Time (mean ± σ):     133.3 ms ±   9.5 ms    [User: 111.4 ms, System: 226.1 ms]
  Range (min … max):   128.4 ms … 171.2 ms    22 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: git status
  Time (mean ± σ):     187.2 ms ±   4.5 ms    [User: 60.6 ms, System: 1160.5 ms]
  Range (min … max):   183.6 ms … 201.4 ms    16 runs

Summary
  gix status ran
    1.40 ± 0.11 times faster than git status

Also here the 2x user time in gix is quite notable and I should definitely look into that.

When looking at the system time in the Linux run it also feels like gix does too much IO somehow.

[NobodyXu]

If it's an I/O issue, then adapting io-uring can help improving the performance on Linux

[Byron]

Git is using standard sync APIs, just like gitoxide, and I think it should be possible to make gitoxide match Git in performance, and that this should be done first. If there is still something left on the table and is feasible, then adding an io-uring special case might be feasible.