python -c: Line causing exception not shown for exceptions other than SyntaxErrors

@terryjreedy

BPO	23035
Nosy	@terryjreedy, @pitrou, @vstinner, @benjaminp, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-12-12.01:44:07.380>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'python -c: Line causing exception not shown for exceptions other than SyntaxErrors'
updated_at = <Date 2021-06-15.16:58:00.697>
user = 'https://bugs.python.org/Arfrever'

bugs.python.org fields:

activity = <Date 2021-06-15.16:58:00.697>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2014-12-12.01:44:07.380>
creator = 'Arfrever'
dependencies = []
files = []
hgrepos = []
issue_num = 23035
keywords = []
message_count = 7.0
messages = ['232506', '232537', '232578', '232582', '232583', '232584', '232585']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'pitrou', 'vstinner', 'benjamin.peterson', 'Arfrever', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23035'
versions = ['Python 3.11']

Linked PRs

When 'python -c ${command}' is used and exception other than SyntaxError occurs, then line causing exception is not shown.

Problem seen in output of last 2 commands below:

$ cat /tmp/test1
1 /
$ cat /tmp/test2
1 / 0
$ cat /tmp/test3
a
$ python3.5 /tmp/test1
  File "/tmp/test1", line 1
    1 /
      ^
SyntaxError: invalid syntax
$ python3.5 /tmp/test2
Traceback (most recent call last):
  File "/tmp/test2", line 1, in <module>
    1 / 0
ZeroDivisionError: division by zero
$ python3.5 /tmp/test3
Traceback (most recent call last):
  File "/tmp/test3", line 1, in <module>
    a
NameError: name 'a' is not defined
$ python3.5 -c '1 /'
  File "<string>", line 1
    1 /
      ^
SyntaxError: invalid syntax
$ python3.5 -c '1 / 0'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ZeroDivisionError: division by zero
$ python3.5 -c 'a'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'a' is not defined

SyntaxError exceptions have a text attribute which contains the line where the error occurred. It's really a special case. For other exceptions, Python only knows that the error occurred in the file called "<string>".

Being able to display the line for any exception requires a complex development. I'm not interested to implement it, I don't think that it's very useful (compared to the time needed to develop it).

It should not be more complex to read a line from a command line argument than to read a line from a regular file.

Code entered with -c seems to be treated the same as code entered at the >>> prompt of the interactive interpreter.

>>> 1/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

In both cases, the offending code is right there to be seen, so I can understand reluctance to echo it. For SyntaxErrors (and only them) echoing the code is needed to have something to point to.

Idle's Shell does what you want.
>>> 1/0
Traceback (most recent call last):
  File "<pyshell#12>", line 1, in <module>
    1/0
ZeroDivisionError: division by zero

Shell can do this because it has easy, platform-independent access to the tkinter Text widget storing and displaying previously entered code. I presume accessing a system-dependent console history buffer is much harder.

Where the difference really matters is when the error is in previously defined objects.

>>> def f():
...   return a

>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
NameError: name 'a' is not defined

versus (Shell)

>>> def f():
	return a

>>> f()
Traceback (most recent call last):
  File "<pyshell#16>", line 1, in <module>
    f()
  File "<pyshell#15>", line 2, in f
    return a
NameError: name 'a' is not defined

Argument of -c option can have multiple lines, while only 1 line can be directly entered in interactive interpreter.

python -c $'line1\nline2\nline3\nline4\n...'

One can paste multiple lines, comprising multiple statements, into the console interprer. (Shell only recognizes a single pasted statement.)
I agree, however, that it seems that Python could keep the split version of the input line for the purpose of tracebacks. I just tried

C:\Users\Terry>python -c "import sys; print(sys.argv)"
['-c']

I expected to see to see a list of 3 strings, not 1.

Arguments after argument of -c option are included in sys.argv:

$ python -c "import sys; print(sys.argv)" a b
['-c', 'a', 'b']

@pablogsal

CC @pablogsal

pythongh-67224: Show source lines in tracebacks when using the -c opt…

After freezing linecache I get:

import time: self [us] | cumulative | imported package
import time:       454 |        454 |   _io
import time:        82 |         82 |   marshal
import time:       736 |        736 |   posix
import time:      1455 |       2726 | _frozen_importlib_external
import time:       211 |        211 |   time
import time:       528 |        738 | zipimport
import time:       130 |        130 |     _codecs
import time:      1317 |       1446 |   codecs
import time:       952 |        952 |   encodings.aliases
import time:      1076 |       3474 | encodings
import time:       336 |        336 | encodings.utf_8
import time:       150 |        150 | _signal
import time:        51 |         51 |     _abc
import time:       266 |        317 |   abc
import time:       284 |        600 | io
import time:        84 |         84 |       _stat
import time:       269 |        353 |     stat
import time:      1519 |       1519 |     _collections_abc
import time:       125 |        125 |       genericpath
import time:       243 |        368 |     posixpath
import time:       885 |       3124 |   os
import time:       175 |        175 |   _sitebuiltins
import time:       243 |        243 |   sitecustomize
import time:        65 |         65 |   usercustomize
import time:       535 |       4139 | site
import time:       118 |        118 | linecache

@brandtbucher

That reduces the import time by 3 times, which hopefully it's enough. What do you think @brandtbucher ?

Flamegraph of startup time (download the svg and play locally):

If you explore the flamegraph seems that the higher cost is getting the os module:

After making moving import sys, import os to the functions that use it I can get a big chunk out of the time:

## Main

Benchmark 1: ./python -S -c pass
  Time (mean ± σ):      22.5 ms ±   5.5 ms    [User: 17.6 ms, System: 4.0 ms]
  Range (min … max):     7.6 ms …  30.1 ms    102 runs

## After
hyperfine "./python -S -c pass"
Benchmark 1: ./python -S -c pass
  Time (mean ± σ):      18.7 ms ±   4.5 ms    [User: 15.2 ms, System: 3.6 ms]
  Range (min … max):     4.9 ms …  27.3 ms    120 runs

With both effects, I get:

pythongh-67224: Make linecache imports relative to improve startup speed

This analysis has being delivered to you thanks to the Perf integration 😉

363 µs (not ms!) looks small in any case.

pythongh-67224: Make linecache imports relative to improve startup speed

gh-67224: Make linecache imports relative to improve startup speed (#…

@pablogsal

Thanks for tackling this, @pablogsal. Your fix made the benchmark 27% faster, more than recovering the lost performance (and keeping the feature)! :)

pythongh-67224: Make linecache imports relative to improve startup sp…

pythongh-67224: Show source lines in tracebacks when using the -c opt…

Navigation Menu

Repository navigation

python -c: Line causing exception not shown for exceptions other than SyntaxErrors #67224

Linked PRs

15 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

python -c: Line causing exception not shown for exceptions other than SyntaxErrors #67224

Description

Linked PRs

Activity

Arfrever commented on Dec 12, 2014

vstinner commented on Dec 12, 2014

Arfrever commented on Dec 12, 2014

terryjreedy commented on Dec 13, 2014

Arfrever commented on Dec 13, 2014

terryjreedy commented on Dec 13, 2014

Arfrever commented on Dec 13, 2014

iritkatriel commented on Oct 22, 2023

15 remaining items

pablogsal commented on Apr 3, 2024

pablogsal commented on Apr 3, 2024

pablogsal commented on Apr 3, 2024

pablogsal commented on Apr 3, 2024

pablogsal commented on Apr 3, 2024

pablogsal commented on Apr 3, 2024

pitrou commented on Apr 3, 2024

brandtbucher commented on Apr 16, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions