The new repl sometimes does not show the error location when `SyntaxError` is raised #121804

tomasr8 · 2024-07-15T18:17:16Z

Bug report

Bug description:

When I run def f[Foo, Foo](): ... from a file, I get a nice error message including the location of the error:

  File ".../test.py", line 1
    def f[Foo, Foo](): ...
               ^^^
SyntaxError: duplicate type parameter 'Foo'

However, when I run this in the new repl, the error location is not shown:

>>> def f[Foo, Foo](): ...
  File "<python-input-0>", line 1
SyntaxError: duplicate type parameter 'Foo'

The same thing happens with this error as well:

>>> def f(x, x): ...
  File "<python-input-0>", line 1
SyntaxError: duplicate argument 'x' in function definition

The missing location seems to be the case for these two errors only as other errors do show it:

>>> def f(42): ...
  File "<unknown>", line 1
    def f(42): ...
          ^^
SyntaxError: invalid syntax

What's interesting is that for the first two errors the file shows up as <python-input-0> while for the last one it's <unknown>. This could be related to the last error coming from the parser while the previous errors come from symtable:

cpython/Python/symtable.c

Lines 1421 to 1435 in 6522f0e

    
           if ((o = PyDict_GetItemWithError(dict, mangled))) { 
        
               val = PyLong_AS_LONG(o); 
        
               if ((flag & DEF_PARAM) && (val & DEF_PARAM)) { 
        
                   /* Is it better to use 'mangled' or 'name' here? */ 
        
                   PyErr_Format(PyExc_SyntaxError, DUPLICATE_ARGUMENT, name); 
        
                   SET_ERROR_LOCATION(st->st_filename, loc); 
        
                   goto error; 
        
               } 
        
               if ((flag & DEF_TYPE_PARAM) && (val & DEF_TYPE_PARAM)) { 
        
                   PyErr_Format(PyExc_SyntaxError, DUPLICATE_TYPE_PARAM, name); 
        
                   SET_ERROR_LOCATION(st->st_filename, loc); 
        
                   goto error; 
        
               } 
        
               val |= flag; 
        
           }

Though it looks like that the error location is being set with SET_ERROR_LOCATION..

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

The text was updated successfully, but these errors were encountered:

skirpichev · 2024-07-16T07:28:28Z

This issue is valid for basic repl as well:

$ PYTHON_BASIC_REPL=1 ./python
Python 3.14.0a0 (heads/main:bf74db731b, Jul 15 2024, 11:06:00) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def f[Foo, Foo](): ...
...
  File "<stdin>-0", line 1
SyntaxError: duplicate type parameter 'Foo'
>>> def f(x, x): ...
... 
  File "<stdin>-1", line 1
SyntaxError: duplicate argument 'x' in function definition

sobolevn · 2024-07-16T08:26:23Z

These errors happen in different places (parser vs symtable):

// symtable.c:1425
PyErr_Format(PyExc_SyntaxError, DUPLICATE_ARGUMENT, name);
SET_ERROR_LOCATION(st->st_filename, loc);

So, I guess it is not related to the new repl.

skirpichev · 2024-07-16T11:05:07Z

Exceptions metadata differ when compile() has valid filename argument wrt fake filename like <input>. It seems, the only way to workaround this is using a temporary file to dump code:

>>> def f(x, x): ...
  File "/tmp/tmpuygzaco0", line 1
    def f(x, x): ...
             ^
SyntaxError: duplicate argument 'x' in function definition

POC diff

diff --git a/Lib/_pyrepl/console.py b/Lib/_pyrepl/console.py
index a8d3f52034..0df16c2b3d 100644
--- a/Lib/_pyrepl/console.py
+++ b/Lib/_pyrepl/console.py
@@ -168,6 +168,12 @@ def showtraceback(self):
         super().showtraceback(colorize=self.can_colorize)
 
     def runsource(self, source, filename="<input>", symbol="single"):
+        import tempfile
+        import os
+        file = tempfile.NamedTemporaryFile('w+', delete=False)
+        file.write(source)
+        file.close()
+        filename = file.name
         try:
             tree = ast.parse(source)
         except (SyntaxError, OverflowError, ValueError):
@@ -189,11 +195,15 @@ def runsource(self, source, filename="<input>", symbol="single"):
                         f" top-level 'await' and run background asyncio tasks."
                     )
                 self.showsyntaxerror(filename)
+                os.unlink(filename)
                 return False
             except (OverflowError, ValueError):
                 self.showsyntaxerror(filename)
+                os.unlink(filename)
                 return False
 
+            os.unlink(filename)
+
             if code is None:
                 return True

sobolevn · 2024-07-16T11:21:06Z

There are multiple systems / setups where temporary files could not be created:

readonly mounts
permissions
etc

Plus, I am not sure that users expect to see File "/tmp/tmpuygzaco0", line 1, I think that it should be a bit more complicated:

We can try to create and open this temp file, if not - fallback to default behaviour
If it is opened, then still show an old <input> tag as a visible filename

There's also a question about how to append / rewrite / recreate this file on new statements.

Storing state is hard :(

tomasr8 · 2024-07-16T11:32:37Z

imo fixing the core issue with the metadata is probably better than using a temp file for the reasons @sobolevn mentioned, but I don't know enough about the code to tell how hard that would be..

skirpichev · 2024-07-16T14:12:47Z

There are multiple systems / setups where temporary files could not be created

Yes, this is not easy:)

Plus, I am not sure that users expect to see File "/tmp/tmpuygzaco0", line 1

I think this can be fixed.

On another hand, we have almost all required metadata for exception in it's attributes:

>>> try:
...     compile("def f(x,x): ...\n", "spam.py", "single")
... except Exception as e:
...     print(e.lineno, e.offset)
...     
1 9

The text attribute is empty.

So, we can provide source as an optional kwarg for showsyntaxerror() method. Given this, we can "fix" empty text attribute of the exception. With this patch (see details below):

>>> def f(x, x): ...
  File "<python-input-0>", line 1
    def f(x, x): ...
             ^
SyntaxError: duplicate argument 'x' in function definition
>>> def f[Foo, Foo](): ...
  File "<python-input-1>", line 1
    def f[Foo, Foo](): ...
               ^^^
SyntaxError: duplicate type parameter 'Foo'

POC patch

diff --git a/Lib/_pyrepl/console.py b/Lib/_pyrepl/console.py
index a8d3f52034..1eda92d49c 100644
--- a/Lib/_pyrepl/console.py
+++ b/Lib/_pyrepl/console.py
@@ -161,8 +161,8 @@ def __init__(
         super().__init__(locals=locals, filename=filename, local_exit=local_exit)  # type: ignore[call-arg]
         self.can_colorize = _colorize.can_colorize()
 
-    def showsyntaxerror(self, filename=None):
-        super().showsyntaxerror(colorize=self.can_colorize)
+    def showsyntaxerror(self, filename=None, **kwargs):
+        super().showsyntaxerror(colorize=self.can_colorize, **kwargs)
 
     def showtraceback(self):
         super().showtraceback(colorize=self.can_colorize)
@@ -188,7 +188,7 @@ def runsource(self, source, filename="<input>", symbol="single"):
                         f"Try the asyncio REPL ({python} -m asyncio) to use"
                         f" top-level 'await' and run background asyncio tasks."
                     )
-                self.showsyntaxerror(filename)
+                self.showsyntaxerror(filename, source=source)
                 return False
             except (OverflowError, ValueError):
                 self.showsyntaxerror(filename)
diff --git a/Lib/code.py b/Lib/code.py
index a55fced070..78bcdcdca5 100644
--- a/Lib/code.py
+++ b/Lib/code.py
@@ -64,7 +64,7 @@ def runsource(self, source, filename="<input>", symbol="single"):
             code = self.compile(source, filename, symbol)
         except (OverflowError, SyntaxError, ValueError):
             # Case 1
-            self.showsyntaxerror(filename)
+            self.showsyntaxerror(filename, source=source)
             return False
 
         if code is None:
@@ -107,6 +107,7 @@ def showsyntaxerror(self, filename=None, **kwargs):
 
         """
         colorize = kwargs.pop('colorize', False)
+        source = kwargs.pop('source', "")
         type, value, tb = sys.exc_info()
         sys.last_exc = value
         sys.last_type = type
@@ -124,6 +125,8 @@ def showsyntaxerror(self, filename=None, **kwargs):
                 value = SyntaxError(msg, (filename, lineno, offset, line))
                 sys.last_exc = sys.last_value = value
         if sys.excepthook is sys.__excepthook__:
+            if source:
+                value.text = source.splitlines()[value.lineno-1]
             lines = traceback.format_exception_only(type, value, colorize=colorize)
             self.write(''.join(lines))
         else:

If this is not too hackish way to deal with the issue - I'll make a pr.

Edit:

Similar fix is possible for basic repl.

POC patch for basic repl

diff --git a/Python/pythonrun.c b/Python/pythonrun.c
index ce7f194e92..43e12acb0e 100644
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -283,6 +283,19 @@ PyRun_InteractiveOneObjectEx(FILE *fp, PyObject *filename,
     _PyArena_Free(arena);
     Py_DECREF(main_module);
     if (res == NULL) {
+        PyThreadState *tstate = _PyThreadState_GET();
+        PyObject *exc = tstate->current_exception;
+        if ((PyObject *)Py_TYPE(exc) == PyExc_SyntaxError) {
+            /* fix "text" attribute */
+            PyObject *xs = PyUnicode_Splitlines(interactive_src, 1);
+            PyObject *ln = PyObject_GetAttr(exc, &_Py_ID(lineno));
+            PyObject *line = PyList_GetItem(xs, PyLong_AsLong(ln) - 1);
+            Py_DECREF(ln);
+            Py_INCREF(line);
+            Py_DECREF(xs);
+            PyObject_SetAttr(exc, &_Py_ID(text), line);
+            Py_DECREF(line);
+        }
         return -1;
     }
     Py_DECREF(res);

…repl >>> def good(x, y): ... ... def bad(x, x): ... File "<python-input-13>", line 2 def bad(x, x): ... ^ SyntaxError: duplicate argument 'x' in function definition

skirpichev · 2024-07-17T03:46:46Z

pr is ready for review: #121886

skirpichev · 2024-08-18T14:40:21Z

cc @pablogsal, maybe you could take look on pr?

pablogsal · 2024-08-18T19:35:45Z

Will do today

…121886)

…in new repl (pythonGH-121886) (cherry picked from commit 354d55e) Co-authored-by: Sergey B Kirpichev <[email protected]>

… repl (GH-121886) (#123148) (cherry picked from commit 354d55e)

…repl (python#121886)

skirpichev · 2024-08-20T02:13:54Z

@pablogsal, this is still valid, in the basic repl. Does it not worth fixing?

pablogsal · 2024-08-20T10:47:44Z

@pablogsal, this is still valid, in the basic repl. Does it not worth fixing?

Unless the fix is trivial I would prefer not to add extra complexity for just the fallback.

skirpichev · 2024-08-20T10:56:02Z

You can see fixes (drafts) few commits above (for basic repl too): #121804 (comment)

pablogsal · 2024-08-20T11:33:10Z

You can see fixes (drafts) few commits above (for basic repl too): #121804 (comment)

Ok, go for it, but that patch needs a bunch of handling for errors in these calls. If you decide to create a PR, please be aware that if it's a lot of code we may still decide to not go with that (although I personally I am happy to go forward).

…c repl

skirpichev · 2024-08-21T14:24:37Z

@pablogsal, here is a new pr: #123202

Still marked as a draft, because it crashes on debug builds. gdb shows it happens in PyUnicode_Check(interactive_src).

Python 3.14.0a0 (heads/syntaxerr-location-basicrepl-121804-dirty:99f7d742a9, Aug 21 2024, 14:26:0) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def f(x, x): ...
...

Program received signal SIGSEGV, Segmentation fault.
ensure_unicode (obj=obj@entry=<unknown at remote 0x7ffff713f700>) at ./Include/object.h:763
763         return ((flags & feature) != 0);
(gdb) bt
#0  ensure_unicode (obj=obj@entry=<unknown at remote 0x7ffff713f700>) at ./Include/object.h:763
#1  0x0000555555726951 in PyUnicode_Splitlines (string=<unknown at remote 0x7ffff713f700>, keepends=keepends@entry=1)
    at Objects/unicodeobject.c:10193
#2  0x000055555581e4fd in PyRun_InteractiveOneObjectEx (fp=fp@entry=0x7ffff7eb4a80 <_IO_2_1_stdin_>,
    filename=filename@entry='<stdin>', flags=flags@entry=0x7fffffffd580) at Python/pythonrun.c:290
#3  0x000055555581ff97 in _PyRun_InteractiveLoopObject (fp=fp@entry=0x7ffff7eb4a80 <_IO_2_1_stdin_>,
    filename=filename@entry='<stdin>', flags=flags@entry=0x7fffffffd580) at Python/pythonrun.c:131
#4  0x0000555555820430 in _PyRun_AnyFileObject (fp=fp@entry=0x7ffff7eb4a80 <_IO_2_1_stdin_>, filename=filename@entry='<stdin>',
    closeit=closeit@entry=0, flags=flags@entry=0x7fffffffd580) at Python/pythonrun.c:71
#5  0x00005555558204e9 in PyRun_AnyFileExFlags (fp=0x7ffff7eb4a80 <_IO_2_1_stdin_>, filename=filename@entry=0x5555558b3886 "<stdin>",
    closeit=closeit@entry=0, flags=flags@entry=0x7fffffffd580) at Python/pythonrun.c:98
#6  0x0000555555845b36 in pymain_run_stdin (config=config@entry=0x555555b5f358 <_PyRuntime+199800>) at Modules/main.c:571
#7  0x000055555584615e in pymain_run_python (exitcode=exitcode@entry=0x7fffffffd624) at Modules/main.c:699
#8  0x00005555558463f4 in Py_RunMain () at Modules/main.c:775
#9  0x000055555584646b in pymain_main (args=args@entry=0x7fffffffd680) at Modules/main.c:805
#10 0x0000555555846531 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:829
#11 0x00005555555d5842 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

It could be reproduced on the current master with:

diff --git a/Python/pythonrun.c b/Python/pythonrun.c
index ce7f194e92..2601c9268c 100644
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -282,6 +282,10 @@ PyRun_InteractiveOneObjectEx(FILE *fp, PyObject *filename,
     PyObject *res = run_mod(mod, filename, main_dict, main_dict, flags, arena, interactive_src, 1);
     _PyArena_Free(arena);
     Py_DECREF(main_module);
+    if (interactive_src) {
+        printf("float? - %d\n", PyFloat_Check(interactive_src));
+        printf("unicode? - %d\n", PyUnicode_Check(interactive_src));
+    }
     if (res == NULL) {
         return -1;
     }

Works on non-debug builds (prints 0 and then 1) and crashes on the first check for debug builds. I think it's a bug, but not with my code: the interactive_src is a PyObject, type checks on it shouldn't crash interpreter.

Should I report new issue?

pablogsal · 2024-08-21T17:47:31Z

Should I report new issue?

It's not a bug, the interactive_src object it's own by the arena and you are freeing the arena two lines before that, which in turn frees the object because it's only own by thee arena. If you incref the interactive_src object before freeing the arena it won't crash.

In debug mode crashes because the allocator has a special code that writes 0xfdddddddddd to the object so it forces crashes, but in release mode the memory is marked as freed but still has not being used so you don't crash immediately.

skirpichev · 2024-08-22T07:47:00Z

Thanks for help! Now pr is ready for review.

…repl (python#121886)

To aid future backports here.

picnixz · 2024-08-28T15:08:47Z

I'm reopening this issue (and add the labels) until we've merged the second PR (just for tracking purposes).

…#123202)

…c repl (pythonGH-123202) (cherry picked from commit 6822cb2) Co-authored-by: Sergey B Kirpichev <[email protected]>

…ic repl (GH-123202) (#123631) gh-121804: always show error location for SyntaxError's in basic repl (GH-123202) (cherry picked from commit 6822cb2) Co-authored-by: Sergey B Kirpichev <[email protected]>

tomasr8 added the type-bug An unexpected behavior, bug, or error label Jul 15, 2024

ambv added the topic-repl Related to the interactive shell label Jul 15, 2024

sobolevn removed the topic-repl Related to the interactive shell label Jul 16, 2024

skirpichev mentioned this issue Jul 17, 2024

gh-121804: always show error location for SyntaxError's in new repl #121886

Merged

pablogsal pushed a commit that referenced this issue Aug 19, 2024

gh-121804: Always show error location for SyntaxError's in new repl (#…

354d55e

…121886)

bedevere-app bot mentioned this issue Aug 19, 2024

[3.13] gh-121804: Always show error location for SyntaxError's in new repl (GH-121886) #123148

Merged

pablogsal pushed a commit that referenced this issue Aug 19, 2024

[3.13] gh-121804: Always show error location for SyntaxError's in new…

c8f4069

… repl (GH-121886) (#123148) (cherry picked from commit 354d55e)

pablogsal closed this as completed Aug 19, 2024

jeremyhylton pushed a commit to jeremyhylton/cpython that referenced this issue Aug 19, 2024

pythongh-121804: Always show error location for SyntaxError's in new …

cc9893b

…repl (python#121886)

skirpichev added a commit to skirpichev/cpython that referenced this issue Aug 21, 2024

pythongh-121804: always show error location for SyntaxError's in basi…

7dd8f83

…c repl

bedevere-app bot mentioned this issue Aug 21, 2024

gh-121804: always show error location for SyntaxError's in basic repl #123202

Merged

blhsing pushed a commit to blhsing/cpython that referenced this issue Aug 22, 2024

pythongh-121804: Always show error location for SyntaxError's in new …

737a58a

…repl (python#121886)

terryjreedy added a commit to terryjreedy/cpython that referenced this issue Aug 26, 2024

[3.12] pythongh-121804: Backport idlelib.pyshell change

3a17748

To aid future backports here.

bedevere-app bot mentioned this issue Aug 26, 2024

[3.12] gh-121804: Backport idlelib.pyshell change (GH-121886) #123366

Merged

terryjreedy added a commit that referenced this issue Aug 26, 2024

[3.12] gh-121804: Backport idlelib.pyshell change (GH-121886) (#123366)

4deb32a

To aid future backports here.

picnixz reopened this Aug 28, 2024

picnixz added topic-repl Related to the interactive shell interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Aug 28, 2024

pablogsal pushed a commit that referenced this issue Sep 3, 2024

gh-121804: always show error location for SyntaxError's in basic repl (…

6822cb2

…#123202)

bedevere-app bot mentioned this issue Sep 3, 2024

[3.13] gh-121804: always show error location for SyntaxError's in basic repl (GH-123202) #123631

Merged

pablogsal closed this as completed Sep 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The new repl sometimes does not show the error location when `SyntaxError` is raised #121804

The new repl sometimes does not show the error location when `SyntaxError` is raised #121804

tomasr8 commented Jul 15, 2024 •

edited by bedevere-app bot

Loading

skirpichev commented Jul 16, 2024

sobolevn commented Jul 16, 2024

skirpichev commented Jul 16, 2024

sobolevn commented Jul 16, 2024 •

edited

Loading

tomasr8 commented Jul 16, 2024

skirpichev commented Jul 16, 2024 •

edited

Loading

skirpichev commented Jul 17, 2024

skirpichev commented Aug 18, 2024

pablogsal commented Aug 18, 2024

skirpichev commented Aug 20, 2024

pablogsal commented Aug 20, 2024

skirpichev commented Aug 20, 2024

pablogsal commented Aug 20, 2024 •

edited

Loading

skirpichev commented Aug 21, 2024

pablogsal commented Aug 21, 2024 •

edited

Loading

skirpichev commented Aug 22, 2024

picnixz commented Aug 28, 2024 •

edited

Loading

The new repl sometimes does not show the error location when SyntaxError is raised #121804

The new repl sometimes does not show the error location when SyntaxError is raised #121804

Comments

tomasr8 commented Jul 15, 2024 • edited by bedevere-app bot Loading

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

skirpichev commented Jul 16, 2024

sobolevn commented Jul 16, 2024

skirpichev commented Jul 16, 2024

sobolevn commented Jul 16, 2024 • edited Loading

tomasr8 commented Jul 16, 2024

skirpichev commented Jul 16, 2024 • edited Loading

skirpichev commented Jul 17, 2024

skirpichev commented Aug 18, 2024

pablogsal commented Aug 18, 2024

skirpichev commented Aug 20, 2024

pablogsal commented Aug 20, 2024

skirpichev commented Aug 20, 2024

pablogsal commented Aug 20, 2024 • edited Loading

skirpichev commented Aug 21, 2024

pablogsal commented Aug 21, 2024 • edited Loading

skirpichev commented Aug 22, 2024

picnixz commented Aug 28, 2024 • edited Loading

The new repl sometimes does not show the error location when `SyntaxError` is raised #121804

The new repl sometimes does not show the error location when `SyntaxError` is raised #121804

tomasr8 commented Jul 15, 2024 •

edited by bedevere-app bot

Loading

sobolevn commented Jul 16, 2024 •

edited

Loading

skirpichev commented Jul 16, 2024 •

edited

Loading

pablogsal commented Aug 20, 2024 •

edited

Loading

pablogsal commented Aug 21, 2024 •

edited

Loading

picnixz commented Aug 28, 2024 •

edited

Loading