gh-75459: Doc: C API: Improve object life cycle documentation

  * Add "cyclic isolate" to the glossary.
  * Add a new "Object Life Cycle" page.
    * Illustrate the order of life cycle functions.
    * Document `PyObject_CallFinalizer` and
      `PyObject_CallFinalizerFromDealloc`.
  * `PyObject_Init` does not call `tp_init`.
  * `PyObject_New`:
    * also initializes the memory
    * does not call `tp_alloc`, `tp_new`, or `tp_init`
    * should not be used for GC-enabled objects
    * memory must be freed by `PyObject_Free`
  * `PyObject_GC_New` memory must be freed by `PyObject_GC_Del`.
  * Warn that garbage collector functions can be called from any
    thread.
  * `tp_finalize` and `tp_clear`:
    * Only called when there's a cyclic isolate.
    * Only one object in the cyclic isolate is finalized/cleared at a
      time.
    * Clearly warn that they might not be called.
    * They can optionally be manually called from `tp_dealloc` (via
      `PyObject_CallFinalizerFromDealloc` in the case of
      `tp_finalize`).
  * `tp_finalize`:
    * Reference `object.__del__`.
    * The finalizer can resurrect the object.
    * Suggest `PyErr_GetRaisedException` and
      `PyErr_SetRaisedException` instead of the deprecated
      `PyErr_Fetch` and `PyErr_Restore` functions.
    * Add links to `PyErr_GetRaisedException` and
      `PyErr_SetRaisedException`.
    * Suggest using `PyErr_WriteUnraisable` if an exception is raised
      during finalization.
    * Rename the example function from `local_finalize` to
      `foo_finalize` for consistency with the `tp_dealloc`
      documentation and as a hint that the name isn't special.
    * Minor wording and sylistic tweaks.
    * Warn that `tp_finalize` can be called during shutdown.

.github/workflows/reusable-docs.yml

@@ -80,6 +80,12 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 60
     steps:
+    - name: 'Install Dependencies'
+      run: |
+        sudo apt-get update &&
+        sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+            graphviz \
+            ;
     - uses: actions/checkout@v4
     - name: 'Set up Python'
       uses: actions/setup-python@v5

.readthedocs.yml

@@ -11,7 +11,8 @@ build:
   os: ubuntu-24.04
   tools:
     python: "3"
-
+  apt_packages:
+    - graphviz
   commands:
     # https://docs.readthedocs.io/en/stable/build-customization.html#cancel-build-based-on-a-condition
     #

Doc/c-api/allocation.rst

@@ -18,7 +18,8 @@ Allocating Objects on the Heap
    reference.  Returns the initialized object.  If *type* indicates that the
    object participates in the cyclic garbage detector, it is added to the
    detector's set of observed objects. Other fields of the object are not
-   affected.
+   initialized.  Specifically, this function does **not** call the object's
+   :meth:`~object.__init__` method (:c:member:`~PyTypeObject.tp_init` slot).
 
 
 .. c:function:: PyVarObject* PyObject_InitVar(PyVarObject *op, PyTypeObject *type, Py_ssize_t size)
@@ -29,27 +30,44 @@ Allocating Objects on the Heap
 
 .. c:macro:: PyObject_New(TYPE, typeobj)
 
-   Allocate a new Python object using the C structure type *TYPE*
-   and the Python type object *typeobj* (``PyTypeObject*``).
-   Fields not defined by the Python object header are not initialized.
-   The caller will own the only reference to the object
-   (i.e. its reference count will be one).
-   The size of the memory allocation is determined from the
-   :c:member:`~PyTypeObject.tp_basicsize` field of the type object.
+   Calls :c:func:`PyObject_Malloc` to allocate memory for a new Python object
+   using the C structure type *TYPE* and the Python type object *typeobj*
+   (``PyTypeObject*``), then initializes the memory like
+   :c:func:`PyObject_Init`.  The caller will own the only reference to the
+   object (i.e. its reference count will be one).  The size of the memory
+   allocation is determined from the :c:member:`~PyTypeObject.tp_basicsize`
+   field of the type object.
+
+   This does not call :c:member:`~PyTypeObject.tp_alloc`,
+   :c:member:`~PyTypeObject.tp_new` (:meth:`~object.__new__`), or
+   :c:member:`~PyTypeObject.tp_init` (:meth:`~object.__init__`).
+
+   This should not be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set
+   in :c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_New`
+   instead.
+
+   Memory allocated by this function must be freed with :c:func:`PyObject_Free`.
 
 
 .. c:macro:: PyObject_NewVar(TYPE, typeobj, size)
 
-   Allocate a new Python object using the C structure type *TYPE* and the
-   Python type object *typeobj* (``PyTypeObject*``).
-   Fields not defined by the Python object header
-   are not initialized.  The allocated memory allows for the *TYPE* structure
-   plus *size* (``Py_ssize_t``) fields of the size
-   given by the :c:member:`~PyTypeObject.tp_itemsize` field of
-   *typeobj*.  This is useful for implementing objects like tuples, which are
-   able to determine their size at construction time.  Embedding the array of
-   fields into the same allocation decreases the number of allocations,
-   improving the memory management efficiency.
+   Like :c:macro:`PyObject_New` except:
+
+   * It allocates enough memory for the *TYPE* structure plus *size*
+     (``Py_ssize_t``) fields of the size given by the
+     :c:member:`~PyTypeObject.tp_itemsize` field of *typeobj*.
+   * The memory is initialized like :c:func:`PyObject_InitVar`.
+
+   This is useful for implementing objects like tuples, which are able to
+   determine their size at construction time.  Embedding the array of fields
+   into the same allocation decreases the number of allocations, improving the
+   memory management efficiency.
+
+   This should not be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set
+   in :c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_NewVar`
+   instead.
+
+   Memory allocated by this function must be freed with :c:func:`PyObject_Free`.
 
 
 .. c:function:: void PyObject_Del(void *op)

Doc/c-api/gcsupport.rst

@@ -57,11 +57,17 @@ rules:
    Analogous to :c:macro:`PyObject_New` but for container objects with the
    :c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
 
+   Memory allocated by this function must be freed with
+   :c:func:`PyObject_GC_Del`.
+
 .. c:macro:: PyObject_GC_NewVar(TYPE, typeobj, size)
 
    Analogous to :c:macro:`PyObject_NewVar` but for container objects with the
    :c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
 
+   Memory allocated by this function must be freed with
+   :c:func:`PyObject_GC_Del`.
+
 .. c:function:: PyObject* PyUnstable_Object_GC_NewWithExtraData(PyTypeObject *type, size_t extra_size)
 
    Analogous to :c:macro:`PyObject_GC_New` but allocates *extra_size*
@@ -73,6 +79,9 @@ rules:
    The extra data will be deallocated with the object, but otherwise it is
    not managed by Python.
 
+   Memory allocated by this function must be freed with
+   :c:func:`PyObject_GC_Del`.
+
    .. warning::
       The function is marked as unstable because the final mechanism
       for reserving extra data after an instance is not yet decided.

Doc/c-api/lifecycle.rst

@@ -0,0 +1,173 @@
+.. highlight:: c
+
+.. _life-cycle:
+
+Object Life Cycle
+=================
+
+Stages
+------
+
+The following is an illustration of the stages of life of an object.  Arrows
+indicate a "happens before" relationship.  Octagons indicate functions specific
+to :ref:`garbage collection support <supporting-cycle-detection>`.
+
+.. digraph:: callorder
+
+   graph [
+     fontname="svg"
+     fontsize=10.0
+     layout="dot"
+     ranksep=0.25
+   ]
+   node [
+     fontname="Courier"
+     fontsize=10.0
+   ]
+   edge [
+     fontname="Times-Italic"
+     fontsize=10.0
+   ]
+
+   "start" [fontname="Times-Italic" shape=plain label=<  start  > style=invis]
+   "tp_alloc" [href="typeobj.html#c.PyTypeObject.tp_alloc" target="_top"]
+   "tp_new" [href="typeobj.html#c.PyTypeObject.tp_new" target="_top"]
+   "tp_init" [href="typeobj.html#c.PyTypeObject.tp_init" target="_top"]
+   {
+     rank="same"
+     "alive" [
+       fontname="Times-Italic"
+       label=<alive, ref count &gt; 0>
+       shape=box
+     ]
+     "tp_traverse" [
+       href="typeobj.html#c.PyTypeObject.tp_traverse"
+       shape=octagon
+       target="_top"
+     ]
+   }
+   "tp_finalize" [
+     href="typeobj.html#c.PyTypeObject.tp_finalize"
+     shape=octagon
+     target="_top"
+   ]
+   "tp_clear" [
+     href="typeobj.html#c.PyTypeObject.tp_clear"
+     shape=octagon
+     target="_top"
+   ]
+   "ref0" [
+     fontname="Times-Italic"
+     label=<ref count == 0>
+     ordering="in"
+     shape=box
+   ]
+   "tp_dealloc" [href="typeobj.html#c.PyTypeObject.tp_dealloc" target="_top"]
+   "tp_free" [href="typeobj.html#c.PyTypeObject.tp_free" target="_top"]
+
+   "start" -> "tp_alloc"
+   "tp_alloc" -> "tp_new"
+   "tp_new" -> "tp_init"
+   "tp_init" -> "alive"
+   "tp_traverse" -> "alive"
+   "alive" -> "tp_traverse"
+   "alive" -> "tp_clear" [label=<  cyclic    <br/>isolate    >]
+   "alive" -> "tp_finalize" [
+     dir="back"
+     label=<  resurrected  >
+   ]
+   "alive" -> "tp_finalize" [label=<  cyclic    <br/>isolate    >]
+   "tp_finalize" -> "tp_clear"
+   "tp_finalize" -> "ref0"
+   "tp_clear" -> "ref0"
+   "tp_clear" -> "tp_dealloc" [
+     dir="back"
+     label=<  optional<br/>direct call  >
+   ]
+   "alive" -> "ref0"
+   "ref0" -> "tp_dealloc"
+   "tp_finalize" -> "tp_dealloc" [
+     dir="back"
+     href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc"
+     label=<
+       <table border="0" cellborder="0" cellpadding="0" cellspacing="0">
+         <tr>
+           <td rowspan="4"> </td>
+           <td align="left">optional call to</td>
+           <td rowspan="4">      </td>
+         </tr>
+         <tr>
+           <td align="left"><font face="Courier">PyObject_Call</font></td>
+         </tr>
+         <tr>
+           <td align="left"><font face="Courier">FinalizerFrom</font></td>
+         </tr>
+         <tr><td align="left"><font face="Courier">Dealloc</font></td></tr>
+       </table>
+     >
+     target="_top"
+   ]
+   "tp_dealloc" -> "tp_free" [label=<  directly calls  >]
+
+Explanation:
+
+* :c:member:`~PyTypeObject.tp_alloc`, :c:member:`~PyTypeObject.tp_new`, and
+  :c:member:`~PyTypeObject.tp_init` are called to allocate memory for a new
+  object and initialize the object.
+* If the reference count for an object drops to 0,
+  :c:member:`~PyTypeObject.tp_dealloc` is called to destroy the object.
+* :c:member:`~PyTypeObject.tp_dealloc` can optionally call
+  :c:member:`~PyTypeObject.tp_finalize` (if non-``NULL``) via
+  :c:func:`PyObject_CallFinalizerFromDealloc` if it wishes to reuse that code
+  to help with object destruction.
+* :c:member:`~PyTypeObject.tp_finalize` may increase the object's reference
+  count, halting the destruction.  The object is said to be resurrected.
+* :c:member:`~PyTypeObject.tp_dealloc` can optionally call
+  :c:member:`~PyTypeObject.tp_clear` (if non-``NULL``) if it wishes to reuse
+  that code to help with object destruction.
+* When :c:member:`~PyTypeObject.tp_dealloc` finishes object destruction, it
+  directly calls :c:member:`~PyTypeObject.tp_free` to deallocate the memory.
+
+If the object is marked as supporting garbage collection (the
+:c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in
+:c:member:`~PyTypeObject.tp_flags`), the following stages are also possible:
+
+* The garbage collector occasionally calls
+  :c:member:`~PyTypeObject.tp_traverse` to identify :term:`cyclic isolates
+  <cyclic isolate>`.
+* When the garbage collector discovers a cyclic isolate, it finalizes one of
+  the objects in the group by calling its :c:member:`~PyTypeObject.tp_finalize`
+  function.  This repeats until the cyclic isolate doesn't exist or all of the
+  objects have been finalized.
+* The :c:member:`~PyTypeObject.tp_finalize` function can optionally increase
+  the object's reference count, causing it (and other objects it references) to
+  become resurrected and no longer a member of a cyclic isolate.
+* When the garbage collector discovers a cyclic isolate and all of the objects
+  in the group have already been finalized, the garbage collector clears one of
+  the objects in the group by calling its :c:member:`~PyTypeObject.tp_clear`
+  function.  This repeats until the cyclic isolate doesn't exist or all of the
+  objects have been cleared.
+
+
+Functions
+---------
+
+To allocate and free memory, see :ref:`Allocating Objects on the Heap
+<allocating-objects>`.
+
+
+.. c:function:: void PyObject_CallFinalizer(PyObject *op)
+
+   Calls the object's finalizer (:c:member:`~PyTypeObject.tp_finalize`) if it
+   has not already been called.
+
+
+.. c:function:: int PyObject_CallFinalizerFromDealloc(PyObject *op)
+
+   Calls the object's finalizer (:c:member:`~PyTypeObject.tp_finalize`) if it
+   has not already been called.  This function is intended to be called at the
+   beginning of the object's destructor (:c:member:`~PyTypeObject.tp_dealloc`).
+   The object's reference count must already be 0.  If the object's finalizer
+   increases the object's reference count, the object is resurrected and this
+   function returns -1; no further destruction should happen.  Otherwise, this
+   function returns 0 and destruction can continue normally.

Doc/c-api/objimpl.rst

@@ -12,6 +12,7 @@ object types.
 .. toctree::
 
    allocation.rst
+   lifecycle.rst
    structures.rst
    typeobj.rst
    gcsupport.rst