NDK v26 compiling C++ MUCH slower than v25

[AndrewAtAvenza]

I just upgraded to v26 and I'm noticing that it's compiling C++ enormously slower. I'm guessing its because I need to change some flag or option somewhere, but I'm at a loss about where to start.

When I build in AS, I can see that it seems to be building about ~4 files in parallel, but it's only tapping 30-40% of my CPU. v25 basically redlined my CPU with a dozen jobs. So that might be the whole thing right there. When I build dependencies on the command line, it looks like it's only using a single file at a time, which is insanely slow.

Any idea what changed and what I can do to fix it? I can't merge this stuff into my main branch when it's literally compiling our app orders of magnitude slower!

Any help appreciated!

[AndrewAtAvenza]

I added timing to my build logs and then compiled a table comparing builds with NDK 25.0.8775105 vs NDK 26.1.10909125:

That's a 10x slower build throughput 😢

As a test, I swapped the LLVM folders from the NDKs and when I do that, v26 builds (with Clang 14.0.6) just as fast as v25. So it seems to be a clang issue.

[AndrewAtAvenza]

I appreciate the reply!

I can whip up a ticket, but my investigation currently seems to point the finger at clang more than the NDK itself. Is that still something you'd look at? If so, I'll do a bit more verification (i.e. get some colleagues to try this and see if they see the same behaviour as me) and then I can file something. I imagine building expat would be a good repro case, assuming others on my team see the same issue; it has no dependencies so I can just post the build commands easily.

I originally posted here on the off chance hoping someone would just poke their head in and say "Oh, silly, you need to update your CMake/ninja" or "You need to add flag X".

[pirama-arumuga-nainar]

What platform are you building on (Mac/Windows/Linux, and OS Version)?

[AndrewAtAvenza]

I'm building on Windows thru Git Bash. That's probably odd, but it's because our team works variously on Linux, Windows & OSX. I haven't had a chance to see if this problem is replicated on OSX or Linux; that was on my agenda for tomorrow, as soon as I can find a guinea pig to agree to try out my branch. Sadly, I'm basically the only C++ guy in a sea of Kotlin/Java developers so I wanted to determine what I could before I started trying to grab one of them to walkthrough some diagnostics.

[DanAlbert]

Even if it is a Clang problem, a regression that severe is something we'd want to cherry-pick a fix for. You can file it upstream with LLVM if you want to do that and we can mirror it on our end, or you can file it here and we'll do the mirroring (it's obviously less work for us if you file it with LLVM, so I won't talk you out of that if that's what you want to do 😉).

There certainly are no expected slowdowns of that magnitud. Our test build times on Windows are actually twice as fast in r26 as they were in r25, but that's not useful data since the libc++ tests were a large part of the test build time and they've moved to a different build in r26.

[AndrewAtAvenza]

Just a note for anyone who comes across this: this has been filed as #1971