Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C API] Private functions used by mypyc #121489

Closed
cdce8p opened this issue Jul 8, 2024 · 12 comments
Closed

[C API] Private functions used by mypyc #121489

cdce8p opened this issue Jul 8, 2024 · 12 comments

Comments

@cdce8p
Copy link
Contributor

cdce8p commented Jul 8, 2024

I've been working on adding support for 3.13 to mypyc. After updating the codebase, there are still some functions left we depend on which are now private after #106320. Would it make sense to export these again? That would make adopting 3.13 a bit easier. /CC @vstinner

  • _PyBytes_Join defined in pycore_bytesobject.h
  • _PyObject_CallMethodIdObjArgs defined in pycore_call.h
  • _PyType_CalculateMetaclass defined in pycore_object.h
  • _PyUnicode_EQ defined in pycore_unicodeobject.h
  • _PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).

Other functions include these two. However, as they are inlined, it isn't a problem to include them.

  • _PyObject_CallMethodIdNoArgs and _PyObject_CallMethodIdOneArg defined in pycore_call.h

It would be fine to keep all of them in internal/pycore_* and just add PyAPI_FUNC.

/CC @hauntsaninja

Linked PRs

@vstinner
Copy link
Member

vstinner commented Jul 8, 2024

_PyBytes_Join defined in pycore_bytesobject.h

You can call something like:

PyObject *empty = PyBytes_FromString("");
PyObject *res = PyObject_CallMethod(empty, "join", ...);
Py_DECREF(empty);

_PyObject_CallMethodIdObjArgs defined in pycore_call.h

We are trying to get rid of _Py_IDENTIFIER() API. You should use your own mecanism for constant strings.

You can call PyObject_CallMethodObjArgs().

_PyType_CalculateMetaclass defined in pycore_object.h

For this one, I don't know. You might reimplement it, use the internal C API, or we should expose it as a public C API.

_PyUnicode_EQ defined in pycore_unicodeobject.h

You can use PyUnicode_Compare(). Maybe we should expose it.

_PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).

Would you mind to elaborate your use case? Can you use the new https://docs.python.org/dev/c-api/unicode.html#pyunicodewriter API instead?

@cdce8p
Copy link
Contributor Author

cdce8p commented Jul 8, 2024

_PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).

Would you mind to elaborate your use case? Can you use the new https://docs.python.org/dev/c-api/unicode.html#pyunicodewriter API instead?

It's used here https://github.com/python/mypy/blob/4e3346ee1dee83868adc2411c4a0f4050cf2f95a/mypyc/lib-rt/str_ops.c#L120 in a function which basically reimplements _PyUnicode_JoinArray(). Guess it would work if we replaced it with PyUnicode_CopyCharacters.

@vstinner
Copy link
Member

vstinner commented Jul 8, 2024

It's used here https://github.com/python/mypy/blob/4e3346ee1dee83868adc2411c4a0f4050cf2f95a/mypyc/lib-rt/str_ops.c#L120 in a function which basically reimplements _PyUnicode_JoinArray().

IMO PyUnicodeWriter would be a good fit for this function, especially if you can compute the exact output string length in advance.

@vstinner
Copy link
Member

cc @erlend-aasland @serhiy-storchaka

@vstinner
Copy link
Member

_PyBytes_Join defined in pycore_bytesobject.h

I created issue gh-121645: [C API] Add PyBytes_Join() function.

cdce8p added a commit to cdce8p/cpython that referenced this issue Jul 21, 2024
cdce8p added a commit to cdce8p/cpython that referenced this issue Jul 25, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 25, 2024
vstinner pushed a commit that referenced this issue Jul 25, 2024
…122287)

gh-121489: Export private _PyBytes_Join() again (GH-122267)
(cherry picked from commit aef95eb)

Co-authored-by: Marc Mueller <[email protected]>
@vstinner
Copy link
Member

vstinner commented Aug 2, 2024

@cdce8p: Python 3.13rc1 has been released. What's the mypyc status? Which functions do you still need?

@cdce8p
Copy link
Contributor Author

cdce8p commented Aug 2, 2024

@cdce8p: Python 3.13rc1 has been released. Which functions do you still need?

I believe we've now resolved all issues here. There is one open PR on the mypy side which needs to get merged but that should be it python/mypy#17526.

What's the mypyc status?

Not strictly related to this issue. There are still two test issues with mypyc before we can support 3.13. Maybe you've an idea.

  1. mypyc still uses some functions from internal/pycore_*.h. I've added #define Py_CORE_BUILD which works. However, there seems to be an issue with linking mimalloc. This is one of the error messages. pythonsupport.h is part of mypyc.
/usr/include/python3.13/internal/mimalloc/mimalloc.h:433:1: error: template with C linkage
  433 | template<class T> struct _mi_stl_allocator_common {
      | ^~~~~~~~
In file included from /usr/include/python3.13/internal/pycore_runtime.h:17,
                 from /usr/include/python3.13/internal/pycore_pystate.h:12,
                 from /usr/include/python3.13/internal/pycore_call.h:12,
                 from pythonsupport.h:21,
                 from CPy.h:12,
                 from test_capi.cc:5:
/usr/include/python3.13/internal/pycore_interp.h:4:1: note: ‘extern "C"’ linkage started here
    4 | extern "C" {
      | ^~~~~~~~~~
In file included from /usr/include/python3.13/internal/pycore_mimalloc.h:39,
                 from /usr/include/python3.13/internal/pycore_interp.h:31,
                 from /usr/include/python3.13/internal/pycore_runtime.h:17,
                 from /usr/include/python3.13/internal/pycore_pystate.h:12,
                 from /usr/include/python3.13/internal/pycore_call.h:12,
                 from pythonsupport.h:21,
                 from CPy.h:12,
                 from test_capi.cc:5:
  1. The other category of errors is extension module '...' is already cached. I've bisected that to gh-117953: Track Extra Details in Global Extensions Cache #118532. It might be something which can be fixed in the mypyc test runner.
  Traceback (most recent call last): (diff)
    File "driver.py", line 1, in <module> (diff)
      from native import f1 (diff)
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load (diff)
    File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked (diff)
    File "<frozen importlib._bootstrap>", line 921, in _load_unlocked (diff)
    File "<frozen importlib._bootstrap>", line 819, in module_from_spec (diff)
    File "<frozen importlib._bootstrap>", line 782, in _init_module_attrs (diff)
  SystemError: extension module 'native' is already cached (diff)

@vstinner
Copy link
Member

vstinner commented Aug 5, 2024

error: template with C linkage

The problem is that mimalloc is included inside extern "C" { ... }.

Ah, another issue was already created: issue gh-122584.

@vstinner
Copy link
Member

vstinner commented Aug 5, 2024

SystemError: extension module 'native' is already cached (diff)

Please open a separated issue for that.

@vstinner
Copy link
Member

@cdce8p: Can you close the issue? Or are there still issues not tracked by other open issues?

@cdce8p
Copy link
Contributor Author

cdce8p commented Aug 27, 2024

@cdce8p: Can you close the issue? Or are there still issues not tracked by other open issues?

Yes. Thanks again for your help!

@vstinner
Copy link
Member

_PyUnicode_EQ defined in pycore_unicodeobject.h

I created #124502 to add PyUnicode_Equal() function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants