cases_generator inserts code to save the stack pointer in the incorrect location when statements containing escaping calls are next to preprocessor directives

[mpage]

Bug report

Bug description:

The cases generator will insert code to save the stack pointer at the incorrect location if the statement containing an escaping call is next to a preprocessor directive. For example, running the cases generator on the following code:

#ifdef Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompare(&entries[index].me_value, res_o);
DEOPT_IF(!increfed);
#else
Py_INCREF(res_o);
#endif

will produce

_PyFrame_SetStackPointer(frame, stack_pointer);
#ifdef Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompare(&entries[index].me_value, res_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
DEOPT_IF(!increfed, LOAD_GLOBAL);
#else
Py_INCREF(res_o);
#endif

Notice that _PyFrame_SetStackPointer(frame, stack_pointer); is placed before the #ifdef Py_GIL_DISABLED, rather than after it.

I think this is caused by the way that the we determine the beginning of the statement that includes the escaping call. We search backwards from an escaping call to find the beginning of statement, stopping when we encounter one of a handful of tokens:

cpython/Tools/cases_generator/analyzer.py

Lines 638 to 643 in 951cb2c

	while True:
	tkn = node.block.tokens[idx-1]
	if tkn.kind in {"SEMI", "LBRACE", "RBRACE"}:
	break
	idx -= 1
	assert idx > 0

The set of terminal tokens does not include CMACRO. When we encounter such a token we'll treat it as part of the statement containing the escaping call.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-126211: Exclude preprocessor directives from statements containing escaping calls #126213