collect arrays of objects prior to filling tuple in fixed-size conversions

[davidhewitt]

Replacement to #3296.

As discussed there, we believe it's safe to use PyTuple_New when there is confidence of no Python code running while filling the uninitialized tuple.

Therefore in this PR I adjust the fixed-size conversions to collect into an array and then fill the tuple. I also happened to manage to optimize this by use of a const-generic helper which uses PyTuple_SET_ITEM (on the stable abi).

Benchmark shows this is ~20% faster than main, and should be safer too:

main
tuple_into_py           time:   [59.337 ns 59.947 ns 60.588 ns]

safer-tuple-into-py
tuple_into_py           time:   [46.143 ns 46.567 ns 47.005 ns]

I tried the same thing for PyTuple::new (the variable sized constructor) with a Vec, but that worked out around 20% slower, so I haven't included that here for now.

[adamreichold]

I tried the same thing for PyTuple::new (the variable sized constructor) with a Vec, but that worked out around 20% slower, so I haven't included that here for now.

While it is nice that this is an improvement, I think we should compare the cost of the above to the safe version using PyTuple_Pack. We can either use the Vec and PyTuple_New or PyTuple_Pack for the dynamically sized case.

[davidhewitt]

I tried the same thing for PyTuple::new (the variable sized constructor) with a Vec, but that worked out around 20% slower, so I haven't included that here for now.

While it is nice that this is an improvement, I think we should compare the cost of the above to the safe version using PyTuple_Pack. We can either use the Vec and PyTuple_New or PyTuple_Pack for the dynamically sized case.

I haven't worked out how to use PyTuple_Pack for the dynamically sized case; if there's something I'm missing, would be happy for someone to show me the trick :)

[adamreichold]

I haven't worked out how to use PyTuple_Pack for the dynamically sized case; if there's something I'm missing, would be happy for someone to show me the trick :)

Ah, they chose a variadic interface which is not nice for composeability, c.f. https://c-faq.com/varargs/invvarargs.html

I guess libffi could do it, e.g. https://docs.rs/libffi/latest/libffi/low/fn.prep_cif_var.html, but I am pretty sure we do not actually want that.

So I guess optimizing the common case using something like SmallVec<[PyObject; 12]> might be helpful? Otherwise, we just leave that one open until a better replace for PyTuple_New comes along?

[davidhewitt]

So I guess optimizing the common case using something like SmallVec<[PyObject; 12]> might be helpful? Otherwise, we just leave that one open until a better replace for PyTuple_New comes along?

Yes, if someone is interested in trying this out it might be interesting. It sounds like there's some upstream debate on what the best API is so I'm inclined to leave as-is for now and let the dust settle.