Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-102471, PEP 757: Add PyLong import and export API #121339

Draft
wants to merge 33 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
f4fdbf2
gh-102471: Add PyLong import and export API
vstinner Jul 2, 2024
c2e568e
Add layout
vstinner Jul 23, 2024
f0d9525
Merge branch 'main' into long_export
vstinner Sep 3, 2024
b19764f
Rename word_endian to digits_order
vstinner Sep 3, 2024
6f7fd11
Replace Py_digit* type with void*
vstinner Sep 3, 2024
080e079
Add PyLongWriter_Discard() function
vstinner Sep 4, 2024
1a7902f
Fixes
vstinner Sep 5, 2024
b70a6dd
Use unsigned type for ndigits
vstinner Sep 5, 2024
07552a7
Remove again layout
vstinner Sep 5, 2024
0d0f942
Revert "Use unsigned type for ndigits"
vstinner Sep 6, 2024
762c33a
doc: adjust ndigits documentation
vstinner Sep 6, 2024
20be7a3
Update doc
vstinner Sep 13, 2024
d92bf1e
Make PyLong_DigitArray.obj private
vstinner Sep 16, 2024
b3b02a2
Remove reserved documentation
vstinner Sep 16, 2024
caca2d7
PyLong_FreeDigitArray() only clears _reserved
vstinner Sep 16, 2024
4221a49
Make PyLong_LAYOUT static
vstinner Sep 16, 2024
37b1d49
Add PyLong_AsDigitArray.value
vstinner Sep 16, 2024
d70a121
Inline PyLong_AsInt64() to avoid the exception
vstinner Sep 17, 2024
4aa25f6
Remove Py_digit type; update the doc
vstinner Sep 17, 2024
90973d4
Merge branch 'main' into long_export
vstinner Sep 17, 2024
5d3e224
Add long_asnativebytes() function
vstinner Sep 17, 2024
c7d7cb2
Remove reference to removed Py_digit type
vstinner Sep 17, 2024
a3d601a
Address Antoine's review
vstinner Sep 17, 2024
c049268
Merge branch 'main' into long_export
vstinner Sep 17, 2024
06b196b
Merge branch 'main' into long_export
skirpichev Sep 18, 2024
3e8d296
Apply suggestions from code review
skirpichev Sep 18, 2024
86c68c2
Merge branch 'main' into long_export
skirpichev Sep 18, 2024
a8fd669
Revert "Add long_asnativebytes() function"
vstinner Sep 18, 2024
a04f9d0
Use PyLong_AsLongAndOverflow()
vstinner Sep 18, 2024
b2be94a
Try PyLong_AsLongLongAndOverflow() first
vstinner Sep 18, 2024
ca98ad1
Merge branch 'main' into long_export
vstinner Sep 18, 2024
167d75e
Update Doc/c-api/long.rst
vstinner Sep 19, 2024
5e53a5b
Sync implementation with PEP (#8)
skirpichev Oct 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 144 additions & 0 deletions Doc/c-api/long.rst
Original file line number Diff line number Diff line change
Expand Up @@ -597,10 +597,154 @@ distinguished from a number. Use :c:func:`PyErr_Occurred` to disambiguate.
Exactly what values are considered compact is an implementation detail
and is subject to change.

.. versionadded:: 3.12


.. c:function:: Py_ssize_t PyUnstable_Long_CompactValue(const PyLongObject* op)

If *op* is compact, as determined by :c:func:`PyUnstable_Long_IsCompact`,
return its value.

Otherwise, the return value is undefined.

.. versionadded:: 3.12


Export API
^^^^^^^^^^

.. versionadded:: 3.14

.. c:struct:: PyLongLayout

Layout of an array of digits, used by Python :class:`int` object.

Use :c:func:`PyLong_GetNativeLayout` to get the native layout of Python
:class:`int` objects.

See also :attr:`sys.int_info` which exposes similar information to Python.
vstinner marked this conversation as resolved.
Show resolved Hide resolved

.. c:member:: uint8_t bits_per_digit

Bits per digit.

.. c:member:: uint8_t digit_size

Digit size in bytes.

.. c:member:: int8_t digits_order

Digits order:

- ``1`` for most significant digit first
- ``-1`` for least significant digit first

.. c:member:: int8_t endian
vstinner marked this conversation as resolved.
Show resolved Hide resolved

Digit endianness:

- ``1`` for most significant byte first (big endian)
- ``-1`` for least significant first (little endian)


.. c:function:: const PyLongLayout* PyLong_GetNativeLayout(void)

Get the native layout of Python :class:`int` objects.

See the :c:struct:`PyLongLayout` structure.


.. c:struct:: PyLongExport

Export of a Python :class:`int` object.

There are two cases:

* If :c:member:`digits` is ``NULL``, only use the :c:member:`value` member.
Calling :c:func:`PyLong_FreeExport` is optional in this case.
* If :c:member:`digits` is not ``NULL``, use :c:member:`negative`,
:c:member:`ndigits` and :c:member:`digits` members.
Calling :c:func:`PyLong_FreeExport` is mandatory in this case.

.. c:member:: int64_t value

The native integer value of the exported :class:`int` object.
Only valid if :c:member:`digits` is ``NULL``.

.. c:member:: uint8_t negative

1 if the number is negative, 0 otherwise.
vstinner marked this conversation as resolved.
Show resolved Hide resolved
Only valid if :c:member:`digits` is not ``NULL``.

.. c:member:: Py_ssize_t ndigits

Number of digits in :c:member:`digits` array.
vstinner marked this conversation as resolved.
Show resolved Hide resolved
Only valid if :c:member:`digits` is not ``NULL``.

.. c:member:: const void *digits

Read-only array of unsigned digits. Can be ``NULL``.


.. c:function:: int PyLong_Export(PyObject *obj, PyLongExport *export_long)

Export a Python :class:`int` object.

On success, set *\*export_long* and return 0.
On error, set an exception and return -1.

vstinner marked this conversation as resolved.
Show resolved Hide resolved
This function always succeeds if *obj* is a Python :class:`int` object or a
subclass.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vstinner, after some thinking I believe we should drop this contract, if we take into account future changes in internals of CPython's integers. If single layout view will be invalid - this function, probably, will allocate temporary buffers. That might fail.

Can we offer a different contract instead, something like this: "This function always succeeds if obj is a Python int object or a subclass and it's value can't be converted to C long."? In this case users have a clear hint: "try something like PyLong_AsLongAndOverflow and if it fails - fallback to this function". If not, I think this is a severe issue with that part of API.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No fallback to PyLong_AsLongAndOverflow should be needed with this API, thanks to the embedded int64_t value.


If *export_long.digits* is not ``NULL``, :c:func:`PyLong_FreeExport` must be
called when the export is no longer needed.


.. c:function:: void PyLong_FreeExport(PyLongExport *export_long)

Release the export *export_long* created by :c:func:`PyLong_Export`.


PyLongWriter API
^^^^^^^^^^^^^^^^

The :c:type:`PyLongWriter` API can be used to import an integer.

.. versionadded:: 3.14

.. c:struct:: PyLongWriter

A Python :class:`int` writer instance.

The instance must be destroyed by :c:func:`PyLongWriter_Finish`.


.. c:function:: PyLongWriter* PyLongWriter_Create(int negative, Py_ssize_t ndigits, void **digits)

Create a :c:type:`PyLongWriter`.

On success, set *\*digits* and return a writer.
On error, set an exception and return ``NULL``.

*negative* is ``1`` if the number is negative, or ``0`` otherwise.

*ndigits* is the number of digits in the *digits* array. It must be
greater than or equal to 0.

vstinner marked this conversation as resolved.
Show resolved Hide resolved
The caller must initialize the array of digits *digits* and then call
:c:func:`PyLongWriter_Finish` to get a Python :class:`int`. Digits must be
in the range [``0``; ``PyLong_BASE - 1``]. Unused digits must be set to
``0``.


.. c:function:: PyObject* PyLongWriter_Finish(PyLongWriter *writer)

Finish a :c:type:`PyLongWriter` created by :c:func:`PyLongWriter_Create`.

On success, return a Python :class:`int` object.
On error, set an exception and return ``NULL``.


.. c:function:: void PyLongWriter_Discard(PyLongWriter *writer)

Discard the internal object and destroy the writer instance.
2 changes: 2 additions & 0 deletions Doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,8 @@
('c:type', 'size_t'),
('c:type', 'ssize_t'),
('c:type', 'time_t'),
('c:type', 'int8_t'),
('c:type', 'uint8_t'),
('c:type', 'uint32_t'),
('c:type', 'uint64_t'),
('c:type', 'uintmax_t'),
Expand Down
11 changes: 11 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -554,6 +554,17 @@ New Features

(Contributed by Victor Stinner in :gh:`107954`.)

* Add a new import and export API for Python :class:`int` objects:

* :c:func:`PyLong_GetNativeLayout`;
* :c:func:`PyLong_Export`;
* :c:func:`PyLong_FreeExport`;
* :c:func:`PyLongWriter_Create`;
* :c:func:`PyLongWriter_Finish`;
* :c:func:`PyLongWriter_Discard`.

(Contributed by Victor Stinner in :gh:`102471`.)

* Add :c:func:`PyType_GetBaseByToken` and :c:data:`Py_tp_token` slot for easier
superclass identification, which attempts to resolve the `type checking issue
<https://peps.python.org/pep-0630/#type-checking>`__ mentioned in :pep:`630`
Expand Down
49 changes: 49 additions & 0 deletions Include/cpython/longintrepr.h
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,55 @@ _PyLong_CompactValue(const PyLongObject *op)
#define PyUnstable_Long_CompactValue _PyLong_CompactValue


/* --- Import/Export API -------------------------------------------------- */

typedef struct PyLongLayout {
// Bits per digit
uint8_t bits_per_digit;

// Digit size in bytes
uint8_t digit_size;

// Word endian:
// * 1 for most significant word first (big endian)
// * -1 for least significant first (little endian)
int8_t digits_order;

// Array endian:
// * 1 for most significant byte first (big endian)
// * -1 for least significant first (little endian)
int8_t endian;
} PyLongLayout;

PyAPI_FUNC(const PyLongLayout*) PyLong_GetNativeLayout(void);

typedef struct PyLongExport {
int64_t value;
uint8_t negative;
Py_ssize_t ndigits;
const void *digits;
// Member used internally, must not be used for other purpose.
Py_uintptr_t _reserved;
} PyLongExport;

PyAPI_FUNC(int) PyLong_Export(
PyObject *obj,
PyLongExport *export_long);
PyAPI_FUNC(void) PyLong_FreeExport(
PyLongExport *export_long);


/* --- PyLongWriter API --------------------------------------------------- */

typedef struct PyLongWriter PyLongWriter;

PyAPI_FUNC(PyLongWriter*) PyLongWriter_Create(
int negative,
Py_ssize_t ndigits,
void **digits);
PyAPI_FUNC(PyObject*) PyLongWriter_Finish(PyLongWriter *writer);
PyAPI_FUNC(void) PyLongWriter_Discard(PyLongWriter *writer);

#ifdef __cplusplus
}
#endif
Expand Down
88 changes: 88 additions & 0 deletions Lib/test/test_capi/test_long.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

NULL = None


class IntSubclass(int):
pass

Expand Down Expand Up @@ -669,5 +670,92 @@ def test_long_asuint64(self):
self.check_long_asint(as_uint64, 0, UINT64_MAX,
negative_value_error=ValueError)

def test_long_layout(self):
# Test PyLong_GetNativeLayout()
int_info = sys.int_info
layout = _testcapi.get_pylong_layout()
expected = {
'bits_per_digit': int_info.bits_per_digit,
'digit_size': int_info.sizeof_digit,
'digits_order': -1,
'endian': -1 if sys.byteorder == 'little' else 1,
}
self.assertEqual(layout, expected)

def test_long_export(self):
# Test PyLong_Export()
vstinner marked this conversation as resolved.
Show resolved Hide resolved
layout = _testcapi.get_pylong_layout()
base = 2 ** layout['bits_per_digit']

pylong_export = _testcapi.pylong_export

# value fits into int64_t
self.assertEqual(pylong_export(0), 0)
self.assertEqual(pylong_export(123), 123)
self.assertEqual(pylong_export(-123), -123)

# use an array, doesn't fit into int64_t
self.assertEqual(pylong_export(base**10 * 2 + 1),
(0, [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2]))
self.assertEqual(pylong_export(-(base**10 * 2 + 1)),
(1, [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2]))

for value in (1.0, 0+1j, "abc"):
with self.subTest(value=value):
with self.assertRaises(TypeError):
pylong_export(value)

def test_longwriter_create(self):
# Test PyLong_Import()
vstinner marked this conversation as resolved.
Show resolved Hide resolved
layout = _testcapi.get_pylong_layout()
base = 2 ** layout['bits_per_digit']

pylongwriter_create = _testcapi.pylongwriter_create
self.assertEqual(pylongwriter_create(0, []), 0)
self.assertEqual(pylongwriter_create(0, [0]), 0)
self.assertEqual(pylongwriter_create(0, [123]), 123)
self.assertEqual(pylongwriter_create(1, [123]), -123)
self.assertEqual(pylongwriter_create(1, [1, 2]),
-(base * 2 + 1))
self.assertEqual(pylongwriter_create(0, [1, 2, 3]),
base**2 * 3 + base * 2 + 1)
max_digit = base - 1
self.assertEqual(pylongwriter_create(0, [max_digit, max_digit, max_digit]),
base**2 * max_digit + base * max_digit + max_digit)

# normalize
self.assertEqual(pylongwriter_create(0, [123, 0, 0]), 123)

# test singletons + normalize
for num in (-2, 0, 1, 5, 42, 100):
self.assertIs(pylongwriter_create(bool(num < 0), [abs(num), 0]),
num)

def to_digits(num):
digits = []
while True:
num, digit = divmod(num, base)
digits.append(digit)
if not num:
break
return digits

# round trip: Python int -> export -> Python int
pylong_export = _testcapi.pylong_export
numbers = [*range(0, 10), 12345, 0xdeadbeef, 2**100, 2**100-1]
numbers.extend(-num for num in list(numbers))
for num in numbers:
with self.subTest(num=num):
data = pylong_export(num)
if isinstance(data, tuple):
negative, digits = data
else:
value = data
negative = int(value < 0)
digits = to_digits(abs(value))
self.assertEqual(pylongwriter_create(negative, digits), num,
(negative, digits))


if __name__ == "__main__":
unittest.main()
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Add a new import and export API for Python :class:`int` objects:

* :c:func:`PyLong_GetNativeLayout`;
* :c:func:`PyLong_Export`;
* :c:func:`PyLong_FreeExport`;
* :c:func:`PyLongWriter_Create`;
* :c:func:`PyLongWriter_Finish`;
* :c:func:`PyLongWriter_Discard`.

Patch by Victor Stinner.
Loading
Loading