gh-121459: Deferred LOAD_GLOBAL by Fidget-Spinner · Pull Request #123128 · python/cpython

Fidget-Spinner · 2024-08-18T14:31:58Z

This implements PEP 703 deferred refcounting for LOAD_GLOBAL in an agnostic way using the cases generator. So it won't block full deferring of references in the future.

For now, we need to write directly to a stackref entry to keep the global alive. Changing the dictionary lookup to stackref variants is also a prerequisite for LOAD_ATTR and friends.

Issue: PEP 703 Deferred reference counts #121459

picnixz

I can only nitpick on things I understand :') But I will, at some point, try to understand the magic of bytecodes.c.

Include/internal/pycore_dict.h

Objects/dictobject.c

Python/bytecodes.c

Python/ceval.c

picnixz · 2024-08-18T15:07:55Z

Tools/cases_generator/generators_common.py



-def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
+def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, allow_unbalanced_parens: bool = False) -> None:


Suggested change

def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, allow_unbalanced_parens: bool = False) -> None:

def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, *, allow_unbalanced_parens: bool = False) -> None:

Objects/dictobject.c

picnixz · 2024-08-18T15:10:13Z

Objects/dictobject.c

+            goto read_failed;
+        }
+
+        if (ix >= 0) {


Maybe put the case if (ix < 0) first to reflect the if (rc < 0) constructions that we usually do?

Objects/dictobject.c

picnixz · 2024-08-18T15:14:16Z

Objects/dictobject.c

+    ix = _Py_dict_lookup(mp, key, hash, &value);
+    *value_addr = value == NULL ? PyStackRef_NULL : PyStackRef_FromPyObjectNew(value);


Can't you use the _Py_dict_lookup_threadsafe_stackref helper here since you acquired the lock? This would save the PyObject *value declaration (and you could consider inlining the helper actually)

Objects/dictobject.c

Tools/cases_generator/generators_common.py

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

…ython into deferred_globals_final

Fidget-Spinner · 2024-08-19T10:28:57Z

Objects/dictobject.c

+            goto read_failed;
+        }
+        if (ix >= 0) {
+            *value_addr = PyStackRef_FromPyObjectNew(DK_ENTRIES(dk)[ix].me_value);


I'm not quite sure I understand the TSAN failures on dictionaries here. @colesbury do you have a clue?

This is not thread-safe it the value is not deferred or immortal.

Wait but doesn't PyStackRef_FromPyObjectNew incref the value if it's not deferred or immortal, which make it thread safe? Or maybe I'm missing something.

Calling Py_INCREF() on a value that might be concurrently freed is not thread-safe.

Ahh okay that makes sense now. So one possible solution would be to introduce another variant that tries to incref and is allowed to fail? Like in https://github.com/python/cpython/blob/main/Objects/dictobject.c#L1435

Or we can check if it's deferred/immortal, and only if it's not we do the TryXGetRef thing, what do you think?

Yeah, do the check that it's deferred/immortal and if not do the TryXGetRef thing.

I think it's best to write it out explicitly first. If there are other uses of it, we can refactor it into a function later.

colesbury · 2024-08-19T17:06:27Z

Objects/dictobject.c

+    dk = _Py_atomic_load_ptr(&mp->ma_keys);
+    kind = dk->dk_kind;
+
+    if (kind != DICT_KEYS_GENERAL) {


This is complicated and duplicates logic elsewhere. I think it might be worth only supporting a a single case or two. For example, unicode-only and non-split dict. Fallback to _Py_dict_lookup_threadsafe + PyStackRef_FromPyObjectSteal for other cases.

I don't think it is worth changing anything. LOAD_GLOBAL is specialized incredibly well, so this only applies to a rare slow path.

It's not specializing at all in the free-threaded build so LOAD_GLOBAL is a scaling bottleneck now that fewer types are immortalized.

I don't think we want to wait for specialization to be made thread-safe, but we can treat this as a temporary measure and structure the code to reflect that.

I changed this case to fall back to _Py_dict_lookup_threadsafe

This looks backwards to me. The common case should be DICT_KEYS_UNICODE for globals dictionaries, but this only handles DICT_KEYS_GENERAL efficiently.

I think this is still too complicated. We only need to handle the common case of unicode keys. Globals are almost never split dictionaries.

I think this should look something like the following. You can refactor it further by extracting a _Py_TryXGetStackRef() that's like _Py_TryXGetRef but returns a _PyStackRef.

PyDictKeysObject *dk = _Py_atomic_load_ptr(&mp->ma_keys); if (dk->dk_kind == DICT_KEYS_UNICODE && PyUnicode_CheckExact(key)) { Py_ssize_t ix = unicodekeys_lookup_unicode_threadsafe(dk, key, hash); if (ix == DKIX_EMPTY) { *value_addr = PyStackRef_NULL; return ix; } else if (ix >= 0) { PyObject **addr_of_value = &DK_UNICODE_ENTRIES(dk)[ix].me_value; PyObject *value = _Py_atomic_load_ptr(addr_of_value); if (value == NULL) { *value_addr = PyStackRef_NULL; return DKIX_EMPTY; } if (_Py_IsImmortal(value) || _PyObject_HasDeferredRefcount(value)) { *value_addr = (_PyStackRef){ .bits = (uintptr_t)value | Py_TAG_DEFERRED }; return ix; } if (_Py_TryIncrefCompare(addr_of_value, value)) { *value_addr = PyStackRef_FromPyObjectSteal(value); return ix; } } } PyObject *obj; Py_ssize_t ix = _Py_dict_lookup_threadsafe(mp, key, hash, &obj); if (ix >= 0 && obj != NULL) { *value_addr = PyStackRef_FromPyObjectSteal(obj); } else { *value_addr = PyStackRef_NULL; } return ix;

markshannon

I don't really see the point of these changes. LOAD_GLOBAL is something like 1 in 5000 of executed instructions. Its performance is irrelevant.

What is STACK_ENTRY for? All outputs are already declared in the stack comment for the op.

Include/internal/pycore_ceval.h

markshannon · 2024-08-19T15:52:27Z

Include/internal/pycore_dict.h

 extern Py_ssize_t _PyDict_LookupIndex(PyDictObject *, PyObject *);
 extern Py_ssize_t _PyDictKeys_StringLookup(PyDictKeysObject* dictkeys, PyObject *key);
-PyAPI_FUNC(PyObject *)_PyDict_LoadGlobal(PyDictObject *, PyDictObject *, PyObject *);
+PyAPI_FUNC(void)_PyDict_LoadGlobalStackRef(PyDictObject *, PyDictObject *, PyObject *, _PyStackRef *);


Suggested change

PyAPI_FUNC(void)_PyDict_LoadGlobalStackRef(PyDictObject *, PyDictObject *, PyObject *, _PyStackRef *);

PyAPI_FUNC(_PyStackRef )_PyDict_LoadGlobalStackRef(PyDictObject *, PyDictObject *, PyObject *);

Likewise

markshannon · 2024-08-19T15:58:38Z

Python/bytecodes.c

-                                            (PyDictObject *)BUILTINS(),
-                                            name);
-                    if (v_o == NULL) {
+                    _PyDict_LoadGlobalStackRef((PyDictObject *)GLOBALS(),


This is inconsistent with PyMapping_GetOptionalItem above.
I think it would be better to change the whole instruction to use stack refs, or leave it as it is.

I removed stackrefs from the instruction to leave it as is.

markshannon · 2024-08-19T16:00:54Z

Python/bytecodes.c

+                    _PyDict_LoadGlobalStackRef((PyDictObject *)GLOBALS(),
+                                               (PyDictObject *)BUILTINS(),
+                                               name,
+                                               STACK_ENTRY(v));


What does STACK_ENTRY do? Does it have any side effects, or is it just equivalent to &v?

Practically equivalent to &stack_pointer[whatever_v_index_is]. It has no side effects.

markshannon · 2024-08-19T16:05:11Z

Tools/cases_generator/generators_common.py



-def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
+def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, *, allow_unbalanced_parens: bool = False) -> None:


Don't we want balanced parentheses? What is the construct that doesn't balance parentheses?

Specifically it's when we replace something in the middle of the line but still want the rest of the line (and all it's parantheses)

Can you give me an example?

E.g. we replace the before foo(STACK_ENTRY(x));, with foo(stack_pointer[-1]);, we want the replacement function to write until the semicolon ;, ignoring all balanced brackets from the replacement point till that semicolon.

Instead of adding allow_unbalanced_parens can we push the logic into the caller (stack_entry)? Like replace up to the matching ) and then consume/emit tokens up to the end of the statement?

markshannon · 2024-08-20T14:48:19Z

Objects/dictobject.c

+    dk = _Py_atomic_load_ptr(&mp->ma_keys);
+    kind = dk->dk_kind;
+
+    if (kind != DICT_KEYS_GENERAL) {


I don't think it is worth changing anything. LOAD_GLOBAL is specialized incredibly well, so this only applies to a rare slow path.

Fidget-Spinner · 2024-08-26T14:42:40Z

@colesbury I think the PR is in better shape now. Can you take a look please?

colesbury

Lets ensure that this actually improves the scalability of LOAD_GLOBAL calls.

colesbury · 2024-08-26T17:12:05Z

Python/bytecodes.c

            PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1);
-            PyObject *res_o = _PyEval_LoadGlobal(GLOBALS(), BUILTINS(), name);
-            ERROR_IF(res_o == NULL, error);
+            _PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, STACK_ENTRY(res));


Two suggestions:

Make _PyEval_LoadGlobalStackRef and _PyDict_LoadGlobalStackRef return int error codes

Use &STACK_ENTRY(res) because it makes it more clear that the value is a pointer and because it means the STACK_ENTRY() transformation preserves the type.

I will apply 2. but not 1. The reasoning is to keep consistency with _PyEval_LoadGlobal (non-stackref version).

colesbury · 2024-08-26T17:14:47Z

Tools/cases_generator/generators_common.py



-def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
+def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, *, allow_unbalanced_parens: bool = False) -> None:


Instead of adding allow_unbalanced_parens can we push the logic into the caller (stack_entry)? Like replace up to the matching ) and then consume/emit tokens up to the end of the statement?

colesbury · 2024-08-26T17:16:36Z

Objects/dictobject.c

+    dk = _Py_atomic_load_ptr(&mp->ma_keys);
+    kind = dk->dk_kind;
+
+    if (kind != DICT_KEYS_GENERAL) {


This looks backwards to me. The common case should be DICT_KEYS_UNICODE for globals dictionaries, but this only handles DICT_KEYS_GENERAL efficiently.

Objects/dictobject.c

Fidget-Spinner · 2024-08-28T12:31:25Z

Main:

object_cfunction MEGA FAILED: 1.1x slower
cmodule_function MEGA FAILED: 1.5x slower
generator FAILED: 2.7x faster
pymethod MEGA FAILED: 1.6x slower
pyfunction MEGA FAILED: 2.3x slower
module_function MEGA FAILED: 1.6x slower
load_string_const MEGA FAILED: 1.6x slower
load_tuple_const PASSED: 3.9x faster
create_closure MEGA FAILED: 2.8x slower
create_pyobject MEGA FAILED: 2.1x slower

This branch:

object_cfunction MEGA FAILED: 1.0x slower
cmodule_function MEGA FAILED: 1.2x slower
generator PASSED: 3.1x faster
pymethod MEGA FAILED: 1.4x slower
pyfunction MEGA FAILED: 2.4x slower
module_function MEGA FAILED: 1.2x slower
load_string_const MEGA FAILED: 1.6x slower
load_tuple_const PASSED: 3.9x faster
create_closure MEGA FAILED: 2.6x slower
create_pyobject PASSED: 4.2x faster

Create_pyobject became a lot faster!

Fidget-Spinner · 2024-08-30T13:11:47Z

Benchmarking results courtesy of Mike: no slowdown on JIT build https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20240828-3.14.0a0-bfd4400-JIT

colesbury

Sorry for the delay in reviewing this. I think we should simplify this further to only focus on the case we care about.

Objects/dictobject.c

colesbury · 2024-09-06T20:34:16Z

Objects/dictobject.c

+    dk = _Py_atomic_load_ptr(&mp->ma_keys);
+    kind = dk->dk_kind;
+
+    if (kind != DICT_KEYS_GENERAL) {


I think this is still too complicated. We only need to handle the common case of unicode keys. Globals are almost never split dictionaries.

I think this should look something like the following. You can refactor it further by extracting a _Py_TryXGetStackRef() that's like _Py_TryXGetRef but returns a _PyStackRef.

PyDictKeysObject *dk = _Py_atomic_load_ptr(&mp->ma_keys); if (dk->dk_kind == DICT_KEYS_UNICODE && PyUnicode_CheckExact(key)) { Py_ssize_t ix = unicodekeys_lookup_unicode_threadsafe(dk, key, hash); if (ix == DKIX_EMPTY) { *value_addr = PyStackRef_NULL; return ix; } else if (ix >= 0) { PyObject **addr_of_value = &DK_UNICODE_ENTRIES(dk)[ix].me_value; PyObject *value = _Py_atomic_load_ptr(addr_of_value); if (value == NULL) { *value_addr = PyStackRef_NULL; return DKIX_EMPTY; } if (_Py_IsImmortal(value) || _PyObject_HasDeferredRefcount(value)) { *value_addr = (_PyStackRef){ .bits = (uintptr_t)value | Py_TAG_DEFERRED }; return ix; } if (_Py_TryIncrefCompare(addr_of_value, value)) { *value_addr = PyStackRef_FromPyObjectSteal(value); return ix; } } } PyObject *obj; Py_ssize_t ix = _Py_dict_lookup_threadsafe(mp, key, hash, &obj); if (ix >= 0 && obj != NULL) { *value_addr = PyStackRef_FromPyObjectSteal(obj); } else { *value_addr = PyStackRef_NULL; } return ix;

Co-Authored-By: Sam Gross <655866+colesbury@users.noreply.github.com>

colesbury

LGTM

markshannon

Can we go back to defining res as an array of one.

markshannon · 2024-09-11T17:06:30Z

Python/bytecodes.c

            PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1);
-            PyObject *res_o = _PyEval_LoadGlobal(GLOBALS(), BUILTINS(), name);
-            ERROR_IF(res_o == NULL, error);
+            _PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &STACK_ENTRY(res));


Suggested change

_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &STACK_ENTRY(res));

_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &res[0]));

Can we go back to the earlier design of making res an array?
Please also add a comment explaining why we res needs to be trated specially (we need pass a pointer to _PyEval_LoadGlobalStackRef).

The code generator already understands arrays, and it avoids confusion as to where it should insert stack spills for fully deferred reference counting.

You'll able to remove def stack_entry below.

Sorry for the back and forth on this.

markshannon · 2024-09-11T17:30:51Z

Tools/cases_generator/generators_common.py

        # unused portions of the stack to NULL.
        stack.flush_single_var(self.out, target, uop.stack.outputs)

+    def stack_entry(


I'm not sure that the stack calculations are correct here, and I don't think it will be able to handle code that writes to multiple results, like UNPACK....

If we go back to using arrays, we can remove this.

bedevere-app · 2024-09-11T17:33:54Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

bedevere-bot · 2024-09-13T16:28:20Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 8810e28.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/146/builds/9494) and take a look at the build logs.
Check if the failure is related to this commit (8810e28) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/146/builds/9494

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 14, done.        
remote: Counting objects:   7% (1/14)        
remote: Counting objects:  14% (2/14)        
remote: Counting objects:  21% (3/14)        
remote: Counting objects:  28% (4/14)        
remote: Counting objects:  35% (5/14)        
remote: Counting objects:  42% (6/14)        
remote: Counting objects:  50% (7/14)        
remote: Counting objects:  57% (8/14)        
remote: Counting objects:  64% (9/14)        
remote: Counting objects:  71% (10/14)        
remote: Counting objects:  78% (11/14)        
remote: Counting objects:  85% (12/14)        
remote: Counting objects:  92% (13/14)        
remote: Counting objects: 100% (14/14)        
remote: Counting objects: 100% (14/14), done.        
remote: Compressing objects:   7% (1/13)        
remote: Compressing objects:  15% (2/13)        
remote: Compressing objects:  23% (3/13)        
remote: Compressing objects:  30% (4/13)        
remote: Compressing objects:  38% (5/13)        
remote: Compressing objects:  46% (6/13)        
remote: Compressing objects:  53% (7/13)        
remote: Compressing objects:  61% (8/13)        
remote: Compressing objects:  69% (9/13)        
remote: Compressing objects:  76% (10/13)        
remote: Compressing objects:  84% (11/13)        
remote: Compressing objects:  92% (12/13)        
remote: Compressing objects: 100% (13/13)        
remote: Compressing objects: 100% (13/13), done.        
remote: Total 14 (delta 4), reused 2 (delta 1), pack-reused 0 (from 0)        
From https://github.com/python/cpython
 * branch            main       -> FETCH_HEAD
Note: checking out '8810e286fa48876422d1b230208911decbead294'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 8810e28... gh-121459: Deferred LOAD_GLOBAL (GH-123128)
Switched to and reset branch 'main'

Could Not Find D:\buildarea\3.x.bolen-windows10\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find D:\buildarea\3.x.bolen-windows10\build\PCbuild\python*.zip

Cannot open file 'D:\buildarea\3.x.bolen-windows10\build\test-results.xml' for upload

Could Not Find D:\buildarea\3.x.bolen-windows10\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find D:\buildarea\3.x.bolen-windows10\build\PCbuild\python*.zip

This reverts commit 8810e28.

Fidget-Spinner added 3 commits August 18, 2024 21:14

deferred globals

803e9f7

fix merge from upstream

b213b6f

use cases generator to generate stack offsets

11d2fb6

Fidget-Spinner requested review from markshannon and methane as code owners August 18, 2024 14:31

bedevere-app bot mentioned this pull request Aug 18, 2024

PEP 703 Deferred reference counts #121459

Closed

bedevere-app bot added the awaiting core review label Aug 18, 2024

Fidget-Spinner requested a review from colesbury August 18, 2024 14:32

Fidget-Spinner added skip news topic-free-threading labels Aug 18, 2024

Fidget-Spinner added 2 commits August 18, 2024 22:46

fix bug

ea8d855

add type annotations

af0c032

picnixz reviewed Aug 18, 2024

View reviewed changes

Eclips4 reviewed Aug 18, 2024

View reviewed changes

Tools/cases_generator/generators_common.py Outdated Show resolved Hide resolved

Fidget-Spinner and others added 5 commits August 19, 2024 18:14

Update Objects/dictobject.c

1a431de

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

Update Objects/dictobject.c

48bebe9

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

Update Objects/dictobject.c

6e948b4

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

Address review

e23567a

Merge branch 'deferred_globals_final' of github.com:Fidget-Spinner/cp…

dbbebf9

…ython into deferred_globals_final

Fidget-Spinner commented Aug 19, 2024

View reviewed changes

colesbury reviewed Aug 19, 2024

View reviewed changes

markshannon reviewed Aug 20, 2024

View reviewed changes

Fidget-Spinner added 3 commits August 22, 2024 18:21

partially address review

dbdee10

simplify

6d176c0

fix some stuff

4ff9f17

colesbury reviewed Aug 26, 2024

View reviewed changes

Fidget-Spinner added 3 commits August 28, 2024 20:25

specialize for str and split str, move out NULL checks

c644ebf

remove allow_unbalanced_parens

64f932a

Add a & for STACK_ENTRY

bfd4400

colesbury reviewed Sep 6, 2024

View reviewed changes

Address review

0c1ee7e

Co-Authored-By: Sam Gross <655866+colesbury@users.noreply.github.com>

colesbury approved these changes Sep 9, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Sep 9, 2024

markshannon requested changes Sep 11, 2024

View reviewed changes

bedevere-app bot added awaiting changes and removed awaiting merge labels Sep 11, 2024

Change to use array instead of STACK_ENTRY

ad33dd1

Fidget-Spinner requested a review from markshannon September 13, 2024 12:46

markshannon approved these changes Sep 13, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting changes labels Sep 13, 2024

Fidget-Spinner merged commit 8810e28 into python:main Sep 13, 2024

bedevere-app bot removed the awaiting merge label Sep 13, 2024

Fidget-Spinner deleted the deferred_globals_final branch September 13, 2024 16:23

zware added a commit to zware/cpython that referenced this pull request Sep 13, 2024

Revert "pythongh-121459: Deferred LOAD_GLOBAL (pythonGH-123128)"

3563dbd

This reverts commit 8810e28.

colesbury added a commit to colesbury/cpython that referenced this pull request Sep 14, 2024

Revert "pythongh-121459: Deferred LOAD_GLOBAL (pythonGH-123128)"

831d5fa

This reverts commit 8810e28.



		def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str) -> None:
		def emit_to(out: CWriter, tkn_iter: Iterator[Token], end: str, allow_unbalanced_parens: bool = False) -> None:

		ix = _Py_dict_lookup(mp, key, hash, &value);
		*value_addr = value == NULL ? PyStackRef_NULL : PyStackRef_FromPyObjectNew(value);

	PyAPI_FUNC(void)_PyDict_LoadGlobalStackRef(PyDictObject , PyDictObject , PyObject , _PyStackRef );
	PyAPI_FUNC(_PyStackRef )_PyDict_LoadGlobalStackRef(PyDictObject , PyDictObject , PyObject *);

	_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &STACK_ENTRY(res));
	_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, &res[0]));

Uh oh!

Conversation

Fidget-Spinner commented Aug 18, 2024 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

picnixz Aug 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markshannon left a comment • edited by colesbury Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fidget-Spinner commented Aug 26, 2024

Uh oh!

colesbury left a comment

Choose a reason for hiding this comment

Uh oh!

Fidget-Spinner commented Aug 18, 2024 •

edited by bedevere-app bot

Loading

picnixz Aug 18, 2024 •

edited

Loading

markshannon left a comment •

edited by colesbury

Loading