All Questions
Tagged with python-internals cpython
72
questions
0
votes
2answers
46 views
Possible to get “value of address” in python?
In the following, I can see that when adding an integer in python, it adds the integer, assigns that resultant value to a new memory address and then sets the variable to point to that memory address:
...
3
votes
1answer
121 views
Why is @property slower that an attribute while bytecode is the same
Consider this chunk of code:
import timeit
import dis
class Bob(object):
__slots__ = "_a",
def __init__(self):
self._a = "a"
@property
def a_prop(self):
return self....
2
votes
1answer
135 views
Why doesn't the Python interpreter return the explicit SyntaxError message?
When looking at CPython's tokenizer.c, the tokenizer returns specific error messages.
As an example, you can take a look at the part where the tokenizer tries to parse a decimal number. When trying ...
4
votes
1answer
96 views
Definition of math_sin function in the CPython source code?
I am studing the codebase of CPython.
I would like to know where can I find the definition of the math_sin function that appears in the mathmethods table in mathmodule.c:
{"sin", ...
4
votes
2answers
76 views
Large memory footprint of integers compared with result of sys.getsizeof()
Python-Integer-objects in the range [1,2^30) need 28 byte, as provided by sys.getsizeof() and explained for example in this SO-post.
However, when I measure the memory footprint with the following ...
6
votes
2answers
171 views
Python string concatenation internal details
Assume that we have a list of strings and we want to create a string by concatenating all element in this list. Something like this:
def foo(str_lst):
result = ''
for element in str_lst:
...
0
votes
2answers
52 views
Unable to assign result of map() to 2-D list
The below code is for taking a set of rows of students' marks in array, and need find the row with maximum marks.
Below is incomplete code, as need to search for the maximum sum row still; but stuck ...
4
votes
2answers
97 views
How python resolves private(double underscore) methods inside a class?
Consider the following class
class Foo(object):
@staticmethod
def __is():
print('__is')
def m(self):
Foo.__is() # executes
Foo.__is() # fails because of mangling
...
2
votes
1answer
78 views
Run cython code when extension module gets garbage collected
Is there a way to register a function within an Cython extenstion module such that this function is called when the Python garbage collector destroys the module object ?
Something like
def ...
1
vote
1answer
44 views
Where can I view the code for the built-in Python method id() (Python 3.x)? [duplicate]
Where can I view the code for the built-in Python method id() (Python 3.x)?
I've been searching for it on Python's GitHub page, but am not having any luck. I've looked at other questions related to ...
3
votes
2answers
162 views
CPython: Why does += for strings change the id of string variable
Cpython optimizes string increment operations,When initializing memory for a string, the program leaves extra expansion space for it,so, when incrementing, the original string is not copied to the new ...
-7
votes
1answer
229 views
explanation of C implementation python's len function [closed]
I was reading about implementation of builtin functions of python when I came across this C implementation of len function
static PyObject *
builtin_len(PyObject *module, PyObject *obj)
/*[clinic end ...
68
votes
4answers
3k views
Different object size of True and False in Python 3
Experimenting with magic methods (__sizeof__ in particular) on different Python objects I stumbled over the following behaviour:
Python 2.7
>>> False.__sizeof__()
24
>>> True....
2
votes
2answers
234 views
Finding the source code of methods implemented in C?
Please note I am asking this question for informational purposes only
I know the title sound like a duplicate of Finding the source code for built-in Python functions?. But let me explain.
Say for ...
4
votes
1answer
241 views
Who should call PyErr_Fetch?
Many functions in the C API for Python are not safe to use if the error indicator might be set. In particular, PyFloat_AsDouble and similar functions are ambiguous in that they have no return value ...
2
votes
1answer
198 views
how to find the implementation of [::-1] ( reversing list in python ) in CPython source code
I was trying to reverse a list in python.
There are many methods out there and [::-1] seems to be a great one!
But I'm curious how [::-1] is done? What's the time complexity of it?
I searched ...
13
votes
4answers
1k views
Returning int from a __str__ method in Python
I know, that the purpose of str() method is to return the string representation of an object, so I wanted to test what happens if I force it to make something else.
I've created a class and an object:...
7
votes
3answers
468 views
How is ternary operator implemented in Python
I understand that conditional expressions (or ternary operators) are lazy in Python. They represent conditional execution rather than conditional selection. In other words, only one of a or b is ...
3
votes
1answer
78 views
Causes for inconsistent behavior when adding NaNs to a set
There is puzzling (at least for me) behavior of Python's set in combination with NaNs (here live):
>>> float('nan') in {float('nan')} # example 1
False
>>> nan = float('nan') ...
1
vote
2answers
196 views
Initialize PyObject without calling __init__
I have a wrapper class as in the example below, written in Python.
class Foo:
def __init__(self, bar=None):
if bar:
self._bar = bar
else:
self._bar = ...
3
votes
1answer
221 views
How does Python's reversed() function work?
According to Python's docs, reversed() uses __getitem__ and __len__ if __reversed__ is not implemented.
I've encountered a weird behavior and failed to explain it:
>>> class A(dict):
... ...
6
votes
2answers
124 views
Is it possible to restore corrupted “interned” bytes-objects
It is well known, that small bytes-objects are automatically "interned" by CPython (similar to the intern-function for strings). Correction: As explained by @abarnert it is more like the integer-pool ...
1
vote
1answer
118 views
Is an explicit NUL-byte necessary at the end of a bytearray for cython to be able to convert it to a null-terminated C-string
When converting a bytearray-object (or a bytes-object for that matter) to a C-string, the cython-documentation recommends to use the following:
cdef char * cstr = py_bytearray
there is no overhead, ...
2
votes
1answer
81 views
CPython: Why does a 3-line script require far more than 3 cycles in the interpreter to execute?
I just watched this Youtube lecture about CPython Internals by Philip Guo, I am puzzled at one thing.
At 25:55, he modifies the C source of CPython by inserting printf(“hello\n”) at the start of the ...
4
votes
2answers
88 views
How does CPython's interpreter know to print the result of the last expression?
I've been digging around the source code to figure-out at which point the result is printed. For example:
>>> x = 1
>>> x + 2
3
The above two statements are compiled to:
1 ...
13
votes
1answer
744 views
What exactly is the optimization `functools.partial` is making?
CPython 3.6.4:
from functools import partial
def add(x, y, z, a):
return x + y + z + a
list_of_as = list(range(10000))
def max1():
return max(list_of_as , key=lambda a: add(10, 20, 30, a))
...
2
votes
2answers
138 views
Documentation for Python Internals?
Perl provides internal documentation in perlguts. This is a great introduction to the perl source code, and helps you navigate and make sense of it. Does Python have an alternative that supplements ...
51
votes
2answers
5k views
Why is the size of 2⁶³ 36 bytes, but 2⁶³-1 is only 24 bytes?
Everything in Python is an object. So the size of an int in Python will be larger than usual.
>>> sys.getsizeof(int())
24
OK, but why does it take 12 more bytes for 2⁶³ compared too 2⁶³ - 1 ...
4
votes
1answer
424 views
Should importlib.reload restore a deleted attribute in Python 3.6?
I'm looking into these two related questions: here and here.
I am seeing a behavior I do not expect in Python 3.6, which differs from behavior using plain reload in Python 2.7 (and 3.4). Namely, it ...
1
vote
1answer
118 views
Do Python Implementation use Cache Oblivious Data Structures?
I was reading an article on the benefits of cache oblivious data structures and found myself wondering if the Python implementations (CPython) use this approach? If not, is there a technical ...
1
vote
1answer
187 views
Should a C extension fail in module init if a PyModule_Add* function fails?
I was just reviewing some code that created a C extension module for Python which didn't contain enough error-checking. It was easy enough in most cases but when it comes to the module-init function I ...
3
votes
2answers
50 views
What was/is that “void* <unused>” argument for, that some __sizeof__ methods have in CPython?
Several C types in the CPython source have a __sizeof__ method so they can present approximately accurate sizes (in bytes) for instances with sys.getsizeof.
These methods are declared METH_NOARG but ...
30
votes
2answers
7k views
Are sets ordered like dicts in python3.6
Due to changes in dict implementation in Python 3.6 it is now ordered by default. Do sets preserve order as well now?
I could not find any information about it but as both of those data structures ...
23
votes
2answers
1k views
Why is deque implemented as a linked list instead of a circular array?
CPython deque is implemented as a doubly-linked list of 64-item sized "blocks" (arrays). The blocks are all full, except for the ones at either end of the linked list. IIUC, the blocks are freed when ...
2
votes
1answer
65 views
When are python objects created?
Python's id() function returns the unique identifier for an object. So when in my terminal I do something like:
>> a = 23
>> id(a)
28487496
Now, I know that python keeps track of all the ...
13
votes
3answers
1k views
Why does naive string concatenation become quadratic above a certain length?
Building a string through repeated string concatenation is an anti-pattern, but I'm still curious why its performance switches from linear to quadratic after string length exceeds approximately 10 ** ...
4
votes
1answer
142 views
Why does Python's CONTINUE_LOOP allow an outer loop, when BREAK_LOOP doesn't?
I've noticed an intriguing peculiarity in Python's bytecode here.
The CONTINUE_LOOP opcode takes in a target argument that denotes the instruction of the loop to continue executing, which is a ...
2
votes
1answer
2k views
RuntimeError: lost sys.stdout
I was trying to debug an issue with abc.ABCMeta - in particular a subclass check that didn't work as expected and I wanted to start by simply adding a print to the __subclasscheck__ method (I know ...
3
votes
1answer
53 views
Does CPython implement the mentioned optimizations from PEP 380?
PEP 380 mentions that the yield from expr syntax can be optimized in Python.
PEP 380 - Optimizations
Using a specialised syntax opens up possibilities for optimisation when there is a long ...
16
votes
2answers
2k views
What is Python's sequence protocol?
Python does a lot with magic methods and most of these are part of some protocol. I am familiar with the "iterator protocol" and the "number protocol" but recently stumbled over the term "sequence ...
3
votes
1answer
1k views
why python dict update insanely slow?
I had a python program which reads lines from files and puts them into dict, for simple, it looks like:
data = {'file_name':''}
with open('file_name') as in_fd:
for line in in_fd:
data['...
8
votes
2answers
1k views
Why is 'new_file += line + string' so much faster than 'new_file = new_file + line + string'? [duplicate]
Our code takes 10 minutes to siphon thru 68,000 records when we use:
new_file = new_file + line + string
However when we do the following it takes just 1 second:
new_file += line + string
Here is ...
78
votes
2answers
3k views
list() uses slightly more memory than list comprehension
So i was playing with list objects and found little strange thing that if list is created with list() it uses more memory, than list comprehension? I'm using Python 3.5.2
In [1]: import sys
In [2]: a ...
2
votes
1answer
230 views
How to raise an exception in the init method for a CPython module
I am writing a CPython extension on top of a library that was written in C and I couldn't find a solution on how to raise an exception in the init method. So, I split it up and, basically, the ...
7
votes
2answers
113 views
Non-monotonic memory consumption in Python2 dictionaries
Can somebody explain this non-monotonic memory usage of a dictionary in CPython 2.7?
>>> import sys
>>> sys.getsizeof({})
280
>>> sys.getsizeof({'one': 1, 'two': 2, 'three'...
76
votes
2answers
5k views
Why is code using intermediate variables faster than code without?
I have encountered this weird behavior and failed to explain it. These are the benchmarks:
py -3 -m timeit "tuple(range(2000)) == tuple(range(2000))"
10000 loops, best of 3: 97.7 usec per loop
py -3 -...
4
votes
1answer
89 views
Python: infos about the implementation of a Python function
I'm discovering the CPython implementation, the structure of Python objects and the Python bytecodes.
Playing with functions, I've found out that empty functions have a stack size of 1.
Why? What ...
0
votes
0answers
53 views
Python: why are there no stack frames for `warnings.warn`?
Consider this code:
#!python3
import warnings
import sys
def my_showwarning(*args, **kwargs):
frame = sys._getframe()
while frame is not None:
print('name:', frame.f_code.co_name, ...
11
votes
1answer
2k views
How are variables names stored and mapped internally?
I read https://stackoverflow.com/a/19721096/1661745 and it seems that in CPython, variables are simply names that are associated with references.
There are several things going on with the ...
8
votes
2answers
137 views
Does cPython use multiple cores for built-in functions such as sort, any, all?
I understand that cPython has a GIL so that your script can't run on multiple cores without using the multiprocessing module. But is there anything to stop the built in functions such as sorting using ...
