Handle more cases in is_normalizable

[samueltardieu]

By assuming that a recursive type is normalizable within the deeper calls to is_normalizable_helper(), more cases can be handled by this function.

In order to fix stack overflows, a recursion limit has also been added for recursive generic type instantiations.

Fix #9798
Fix #10508
Fix #11915

changelog: [large_enum_variant]: more precise detection of variants with large size differences

[rustbot]

r? @llogiq

rustbot has assigned @llogiq.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[llogiq]

Thank you!

[Jarcho]

This is only used to determine the layout, and the layout of a raw pointer is known, whatever the designated type is.

This is wrong. The size of both references and raw pointers depend on whether the pointee is Sized.

[samueltardieu]

This is only used to determine the layout, and the layout of a raw pointer is known, whatever the designated type is.
This is wrong. The size of both references and raw pointers depend on whether the pointee is Sized.

This is what I meant by "known" instead of "constant", because I thought that at this stage the sizedness of the pointee type would be known. I will try to double-check this.

[Jarcho]

The sizedness of pointee types is the reason layout_of caused ICEs in the first place. For most cases it returns an error, but for pointers and references it sometimes caused a delayed bug to be issued.

[samueltardieu]

@llogiq I haven't checked, but it probably fails because of the missing changelog entry. Anyhow, I don't know if we should merge it right now given @Jarcho's comments. I'll set the PR to draft, even if I couldn't notice crashes in lintchecks for example, I'll have to double-check that this cannot happen.

[y21]

For reference (maybe this helps), a concrete repro that starts ICE-ing with this change (taken from #12550 which also intends to fix the same issue)

#![warn(clippy::zero_sized_map_values)]

trait Foo { type Foo; }

struct Bar<T: Foo>(T::Foo);

type Bad<T> = std::collections::HashMap<u32, *const Bar<T>>;

fn main() {}

[samueltardieu]

Thanks @y21 that will be useful

[llogiq]

@rustbot author

Please ping me again when there's something to review.

[samueltardieu]

By limiting the recursion and assuming that recursive types are normalizable by default when recursing (which will be fixed when going up if needed), more cases are detected and some ICEs are fixed.

@rustbot review

Ping @llogiq and also ping @Jarcho as this collides with #12550 (but can serve as a stopgap until is_normalizable() is no longer needed).

[samueltardieu]

@llogiq Do you see anything to be changed here? This has fixed all ICE I know of. Also, looking at the two commits separately, you can see that some diagnostics have been improved (which was the initial goal of this PR before I was made aware of the ICE):

LL | |     Recursive(Box<WithRecursion>),
   | |     ----------------------------- the second-largest variant contains at least 0 bytes
   | |     ----------------------------- the second-largest variant contains at least 8 bytes
LL | | }
   | |_^ the entire enum is at least 0 bytes
   | |_^ the entire enum is at least 520 bytes