gh-124285: document assumptions on __bool__/__len__ behaviour by skirpichev · Pull Request #124723 · python/cpython

skirpichev · 2024-09-28T05:59:46Z

Issue: Behavior change for foo and 1 or 2: 3.12 newly converts foo to bool twice #124285

📚 Documentation preview 📚: https://cpython-previews--124723.org.readthedocs.build/

iritkatriel · 2024-09-28T13:23:59Z

Doc/reference/datamodel.rst

   true.

+   Two successive calls to :meth:`!__bool__` on the same object must
+   return same value.


Not if the object is mutated.

But that means - some other method was called on the object in between. Thus, calls to __bool__ weren't actually successive.

The object could have been mutated by another thread between the two calls...

by another thread

... that calls some other object's method

between the two calls...

:)

Hmm, how about this: "The __bool__ method can't mutate any objects."? (Ditto for __len__.)

This probably is a more strong requirement than actually need in current optimizations, but I doubt it blocks something useful.

JelleZijlstra · 2024-09-28T22:36:25Z

As @AlexWaygood implied above, even builtin types can't provide the guarantee your current text for: if you call bool() or len() twice on a list, another thread might have run in between and mutated the list.

I think instead, we should say that in cases where the interpreter implicitly calls bool(), it is unspecified how often bool() is called, and it is unspecified what happens if multiple calls happen and they return different results. Not sure where to put that, though.

AlexWaygood · 2024-09-28T23:01:17Z

The language spec already says with regard to hashability:

The __hash__() method should return an integer. The only required property is that objects which compare equal have the same hash value

So some interesting questions that come to mind for me here are:

Does it ever make sense for objects that compare equal to have different boolean values?
Does it ever make sense for objects that have different boolean values to compare equal?
Does it ever make sense for x == y but len(x) != len(y)?

If we think that well-behaved code should never do those things, and that doing those things could violate some assumptions made by Python in some places, then we could consider adding a similar note to the language spec for __bool__, i.e.

A well-defined __bool__() method should never report different results for two objects that compare equal

But I'm not sure if it's worth doing so for __len__ as well/I'm not sure what a good phrasing might be there.

Additionally, this sort-of feels like an implicit requirement for an awful lot of dunders, really. I would find it pretty surprising if bytes(x) and bytes(y) produced different results for two objects x and y where x == y. Similarly for __int__, __index__, __neg__, etc... So that's an argument for not putting a special note in the entry for either __bool__ or __len__.

skirpichev · 2024-09-29T06:11:23Z

another thread might have run in between and mutated the list.

Then this example will not satisfy to added remark: calls to __bool__ weren't actually successive.

doing those things could violate some assumptions made by Python in some places

Do you have examples? While your points look reasonable, I'm not sure that such assumptions are actually used somewhere.

But I'm not sure if it's worth doing so for __len__

Well, in current version I have to add symmetric note for __len__ just because that helper might be used instead of __bool__ in some implicit call to bool().

So that's an argument for not putting a special note in the entry for either __bool__ or __len__.

There is a difference. The __bool__ method in general can return different values for same object; but the reference implementation does assume it can't be in certain conditions.

TeamSpen210 · 2024-09-29T09:43:40Z

Maybe instead of saying something about __bool__(), we could instead say it about and/or. In a chain of logical ops, the interpreter is allowed to calculate the boolean value of any given term only once. Or more broadly, say in any given expression the boolean value of any given sub-expression may be reused if already calculated.

iritkatriel · 2024-09-29T11:25:46Z

I think the docs can just say that __bool__ and __len__ should be idempotent.

skirpichev · 2024-09-29T12:02:47Z

I think the docs can just say that __bool__ and __len__ should be idempotent.

That's a more short version of the current statement, isn't? But I worry it might require explanation of the term. E.g. we don't expect that reader knows about complex numbers.

ncoghlan

Suggested wording added inline. Mutating objects shared between threads is another way of hitting this particular piece of implementation defined behaviour, so I think it's best just to state it that way.

I think "no mutation (of this object, or any other object) " is the right expected invariant to specify, since that's the assumption that gets violated in the multi-threading case.

ncoghlan · 2024-10-06T12:38:14Z

Doc/reference/datamodel.rst

+   Two successive calls to :meth:`!__bool__` on the same object must
+   return same value.


Suggested change

Two successive calls to :meth:`!__bool__` on the same object must

return same value.

Note: to help optimize logical expressions, implementations are permitted to assume

that calls to :meth:`!__bool__` will not mutate that object, nor any other object.

While this expected invariant is not explicitly enforced, failing to abide

by it will result in implementation dependent runtime behaviour. This

implementation dependent behaviour may also be encountered when mutable

objects are shared across threads without appropriate synchronization.

I'm unsure if this needs to be explicitly documented as the behavior is dependent on the user's choice of implementation.

is dependent on the user's choice of implementation.

Rather on bugs in the user code, if someone will implement __bool__(), doing crazy things (see issue). Added docs say that implementation may assume certain behaviour from the user code, just as for the __hash__() method (same hash value for equal objects - the invariant, which we also can't enforce, user code might break this).

PS: I think that part of discussion rather belongs to the issue thread, which has some other arguments on why we want document this. See e.g. this.

Should the note warn against side effects in general (like I/O), not just mutation?

Should the note warn against side effects in general (like I/O), not just mutation?

Side effects, including in fact object mutation - are fine, unless they break idempotence.

But I think we don't loose something practically relevant if just forbid mutation of any objects in __bool__() (a shortened version of @ncoghlan suggestion):

Suggested change

Two successive calls to :meth:`!__bool__` on the same object must

return same value.

Calls to :meth:!__bool__` shouldn't mutate any objects.

ncoghlan · 2024-10-06T12:43:18Z

Doc/reference/datamodel.rst

+   Two successive calls to :meth:`!__len__` on the same object must
+   return same value.


Suggested change

Two successive calls to :meth:`!__len__` on the same object must

return same value.

Note: to help optimize logical expressions, implementations are permitted to assume

that calls to :meth:`!__len__` will not mutate that object, nor any other object.

While this expected invariant is not explicitly enforced, failing to abide

by it will result in implementation dependent runtime behaviour. This

implementation dependent behaviour may also be encountered when mutable

objects are shared across threads without appropriate synchronization.

Here is a typo (__len__ -> __bool__). But if we are going with this lengthly wording - I think it's better just point to the __bool__ docs.

skirpichev · 2025-04-12T10:07:08Z

I think that I can't make progress on this.

pythongh-124285: document assumptions on __bool__/__len__ behaviour

1f28227

skirpichev added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Sep 28, 2024

skirpichev requested a review from willingc as a code owner September 28, 2024 05:59

bedevere-app bot added awaiting review docs Documentation in the Doc dir skip news labels Sep 28, 2024

bedevere-app bot mentioned this pull request Sep 28, 2024

Behavior change for foo and 1 or 2: 3.12 newly converts foo to bool twice #124285

Open

iritkatriel reviewed Sep 28, 2024

View reviewed changes

ncoghlan reviewed Oct 6, 2024

View reviewed changes

skirpichev removed the needs backport to 3.12 only security fixes label Apr 8, 2025

skirpichev closed this Apr 12, 2025

skirpichev deleted the truth-testing-assumptions-124285 branch April 12, 2025 10:07

		Two successive calls to :meth:`!__bool__` on the same object must
		return same value.

-   Two successive calls to :meth:`!__bool__` on the same object must
-   return same value.
+   Note: to help optimize logical expressions, implementations are permitted to assume
+   that calls to  :meth:`!__bool__` will not mutate that object, nor any other object.
+   While this expected invariant is not explicitly enforced, failing to abide
+   by it will result in implementation dependent runtime behaviour. This
+   implementation dependent behaviour may also be encountered when mutable
+   objects are shared across threads without appropriate synchronization.

	Two successive calls to :meth:`!__bool__` on the same object must
	return same value.
	Calls to :meth:!__bool__` shouldn't mutate any objects.

		Two successive calls to :meth:`!__len__` on the same object must
		return same value.

Uh oh!

Conversation

skirpichev commented Sep 28, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skirpichev Sep 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Sep 28, 2024

Uh oh!

AlexWaygood commented Sep 28, 2024

Uh oh!

skirpichev commented Sep 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TeamSpen210 commented Sep 29, 2024

Uh oh!

iritkatriel commented Sep 29, 2024

Uh oh!

skirpichev commented Sep 29, 2024

Uh oh!

ncoghlan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncoghlan Oct 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncoghlan Oct 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skirpichev commented Apr 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

skirpichev commented Sep 28, 2024 •

edited by github-actions bot

Loading

skirpichev Sep 29, 2024 •

edited

Loading

skirpichev commented Sep 29, 2024 •

edited

Loading

ncoghlan left a comment •

edited

Loading

ncoghlan Oct 6, 2024 •

edited

Loading

ncoghlan Oct 6, 2024 •

edited

Loading