gh-139122: Reimplement base UUID type, uuid4(), and uuid7() in C by 1st1 · Pull Request #139123 · python/cpython

1st1 · 2025-09-18T13:14:06Z

The C implementation considerably boosts the performance of the key UUID operations:

------------------------------------
Operation                    Speedup
------------------------------------
uuid4() generation            15.01x
uuid7() generation            29.64x
UUID from string               6.76x
UUID from bytes                5.16x
str(uuid) conversion           6.66x
------------------------------------

Summary of changes:

The UUID type is reimplemented in C in its entirety.
The pure-Python is kept around and is used of the C implementation isn't available for some reason.
Both implementations are tested extensively; additional tests are added to ensure that the C implementation of the type follows the pure Python implementation fully.
The Python implementation stores UUID values as int objects. The C implementation stores them as uint8_t[16] array.
The C implementation supports unpickling of UUIDs created with Python 2 using protocols starting with 0.
The C implementation has faster hash() implementation but also caches the computed hash value to speedup cases when UUIDs are used as set/dict keys.
The C implementation has a freelist to make new UUID object instantiation as fast as possible.
uuid4() and uuid7() are now implmented in C. The most performance boost (10x) comes from overfetching entropy to decrease the number of _PyOS_URandom() calls. On its own it's a safe optimization with the edge case that Unix fork needs to be explicitly handled. We do that by comparing the current PID to the PID of when the random buffer was populated.
Portions of code are coming from my implementation of faster UUID in gel-python [1]. I did use AI during the development, but basically had to rewrite the code it generated to be more idiomatic and efficient.
The benchmark can be found here [2].
This PR makes Python UUID operations as fast as they are in NodeJS and Bun runtimes.

[1] https://github.com/MagicStack/py-pgproto/blob/b8109fb311a59f30f9947567a13508da9a776564/uuid.pyx

[2] https://gist.github.com/1st1/f03e816f34a61e4d46c78ff98baf4818

Issue: Implement UUID functions and type in C #139122

python-cla-bot · 2025-09-18T13:14:11Z

All commit authors signed the Contributor License Agreement.

… in C The C implementation considerably boosts the performance of the key UUID operations: ------------------------------------ Operation Speedup ------------------------------------ uuid4() generation 15.01x uuid7() generation 29.64x UUID from string 6.76x UUID from bytes 5.16x str(uuid) conversion 6.66x ------------------------------------ Summary of changes: * The UUID type is reimplemented in C in its entirety. * The pure-Python is kept around and is used of the C implementation isn't available for some reason. * Both implementations are tested extensively; additional tests are added to ensure that the C implementation of the type follows the pure Python implementation fully. * The Python implementation stores UUID values as int objects. The C implementation stores them as `uint8_t[16]` array. * The C implementation supports unpickling of UUIDs created with Python 2 using protocols starting with 0. That necessitated a small fix to the `copyreg` module (the change is only affecting legacy pickle pathway). * The C implementation has faster hash() implementation but also caches the computed hash value to speedup cases when UUIDs are used as set/dict keys. * The C implementation has a freelist to make new UUID object instantiation as fast as possible. * uuid4() and uuid7() are now implmented in C. The most performance boost (10x) comes from overfetching entropy to decrease the number of _PyOS_URandom() calls. On its own it's a safe optimization with the edge case that Unix fork needs to be explicitly handled. We do that by comparing the current PID to the PID of when the random buffer was populated. * Portions of code are coming from my implementation of faster UUID in gel-python [1]. I did use AI during the development, but basically had to rewrite the code it generated to be more idiomatic and efficient. * The benchmark can be found here [2]. * This PR makes Python UUID operations as fast as they are in NodeJS and Bun runtimes. [1] https://github.com/MagicStack/py-pgproto/blob/b8109fb311a59f30f9947567a13508da9a776564/uuid.pyx [2] https://gist.github.com/1st1/f03e816f34a61e4d46c78ff98baf4818

1st1 · 2025-09-18T13:28:07Z

Note to reviewers:

Please compare uuid7 C and Python implementations side by side
Please review all of the bitshift/magic code closely (Python implementation uses int, i'm using char[16], so low-level operations are very different but have to be equivalent)

zooba · 2025-09-18T13:45:59Z

+
+    *counter = (((uint64_t)(high & 0x1FF) << 32) | (low >> 32)) & 0x1FFFFFFFFFF;
+    *tail = (uint32_t)low;
+    return 0;


I haven't tested thoroughly (mostly looking at GodBolt output), but this ought to generate more efficient code (and there might be more ways to generate the right values directly, I just didn't go that far):

struct { uint16_t high; uint32_t low1; uint32_t low2; } rand_bytes; if (gen_random(state, (uint8_t*)&rand_bytes, sizeof(rand_bytes)) < 0) { return -1; } *counter = (((uint64_t)(rand_bytes.high & 0x1FF) << 32) | (rand_bytes.low1)) & 0x1FFFFFFFFFF; *tail = rand_bytes.low2;

@1st1 for visibility

picnixz · 2025-09-18T17:02:44Z

I would be happy to have a C implementation of UUID but for reviewing purposes, may I suggest that we first focus on implementing the wrapper in C in one PR and have different PRs for each UUID? it would be much easier to focus on the algorithm to review.

picnixz

These are the first round comments I have. I realy want multiple PRs because the module becomes really huge and there are parts that I don't think we need to reimplement in C.

1st1 · 2025-09-19T10:48:34Z

I realy want multiple PRs because the module becomes really huge and there are parts that I don't think we need to reimplement in C.

Sorry, I'm not going to do multiple PRs, I've no time for that. But I will address your comments here for sure!

picnixz · 2025-09-19T12:29:04Z

I would REALLY appreciate smaller PRs. Usually this is our (current) workflow, namely incrementing changes. I think other core devs would agree with me here (@vstinner, @serhiy-storchaka and @encukou).

Note that whatever we do, this is a feature that will only be included in 3.15 so I can also make the small PRs once I am back at home. Changing 3.14 is no more possible unless the RM gives their approval here (cc @hugovk, who could also whether he prefers one or multiple PR) as performance improvements are usually considered as new features (especially if we are adding a C implementation).

1st1 · 2025-09-19T12:52:59Z

I would REALLY appreciate smaller PRs. Usually this is our (current) workflow, namely incrementing changes. I think other core devs would agree with me here (@vstinner, @serhiy-storchaka and @encukou).

The current PR layout:

90% is reimplementing the UUID type
10% is uuid4 and uuid7 implementations which are isolated separate functions from (1); uuid4 is very tiny.

I'm usually also for making smaller PRs, but objectively there's no point for that right here; I don't see how reviewing this would be fundamentally any different if instead of one PR with 1700 lines in uuidmodule.c, you'd have 3 PRs, one with 1600 lines and another with like 100 lines, and yet another with 30. Reviewing time would be the same, no? What's the point?

If you absolutely insist I can do this, but tbqh I really don't understand why you're pushing for this so hard. It would create a lot of work for me and not necessarily make reviewers more happy.

Note that whatever we do, this is a feature that will only be included in 3.15 so I can also make the small PRs once I am back at home. Changing 3.14 is no more possible unless the RM gives their approval here (cc @hugovk, who could also whether he prefers one or multiple PR) as performance improvements are usually considered as new features (especially if we are adding a C implementation).

I don't think this is 3.14 material and I wasn't pushing for that. 3.15 is fine (unless other people but me want this to be in 3.14, in which case I won't say no).

That said I'd like to see my work through and would love to still merge this myself.

1st1 · 2025-09-21T14:47:05Z

@picnixz I'm sorry for my snappiness (I blame 4 days spent deep in C coding! :)) I'm adding some basic machinery to test that C implementation of uuid() functions behave identically. Since this is even more extra code it's now very reasonable to split this into two prs: one for the base UUID type, another for uuid4() and uuid7(). I'll do that later.

This fixes a segfault; regr test added. While there, make the freelist only work with our base UUID type and no subclasses of it.

…ship

read-the-docs-community · 2026-10-19T23:26:59Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #35281390 | 📁 Comparing 9e02f29 against main (395d766)

🔍 Preview build

11 files changed · ± 11 modified

± Modified

…plicit

eendebakpt · 2026-10-20T18:18:24Z

The following reproducer crashes for me:

import gc
import sys
import uuid

_uuid = sys.modules['_uuid']
UUID = _uuid.UUID
assert uuid.UUID is UUID, "C implementation not in use"

# Pass 1: drop the Python module and its NAMESPACE_*/NIL/MAX instances, so no
# base-type instance is left to keep the C type alive.
del sys.modules['uuid']
del uuid
gc.collect()

class Sub(UUID):
    pass

Sub.keep = Sub(int=1)   # instance reachable only from the (younger) subclass

del sys.modules['_uuid']
del _uuid, UUID, Sub
print('final collect')
gc.collect()            # module, base type, Sub and Sub.keep are all garbage
print("survived")

The crash is not due to the freelist, but is related: the crash is due to module state lookup which is there for the freelist.

The _asyncio module also is an extension that uses the cpython freelist. So I would prefer to either keep the freelist out of the PR (and make a followup PR), or use the standard cpython freelist mechanism.

… updates

…ocation

1st1 · 2026-10-20T21:08:18Z

@eendebakpt

The following reproducer crashes for me:

Should be fixed!

picnixz

Commenting first for the Python and the documentation part (some comments I left last year should still be addressed but those are mainly for rendering docs or tests).

picnixz · 2026-10-20T21:38:40Z

   variant and version number set according to :rfc:`9562`, overriding bits in the
   given *hex*, *bytes*, *bytes_le*, *fields*, or *int*.

+   .. versionchanged:: 3.16


Suggested change

.. versionchanged:: 3.16

.. versionchanged:: next

picnixz · 2026-10-20T21:45:36Z

+                self.assertEqual(
+                    self.uuid.UUID(*kwargs.values(), version=None), expected)


Suggested change

self.assertEqual(

self.uuid.UUID(*kwargs.values(), version=None), expected)

u_with_args = self.uuid.UUID(*kwargs.values(), version=None)

self.assertEqual(u_with_args, expected)

picnixz · 2026-10-20T21:45:53Z

+            with self.subTest(input=name):
+                kwargs = dict.fromkeys(inputs)
+                kwargs[name] = value
+                self.assertEqual(self.uuid.UUID(**kwargs, version=None), expected)


Suggested change

self.assertEqual(self.uuid.UUID(**kwargs, version=None), expected)

u_with_kwargs = self.uuid.UUID(**kwargs, version=None)

self.assertEqual(u_with_kwargs, expected)

picnixz · 2026-10-20T21:58:50Z

+            for safety in self.uuid.SafeUUID
+        )
+        with support.swap_item(sys.modules, 'uuid', self.uuid):
+            for mapping in (dict, UserDict, MappingProxyType):


Suggested change

for mapping in (dict, UserDict, MappingProxyType):

for mapping in (dict, frozendict, UserDict, MappingProxyType):

picnixz · 2026-10-20T21:59:01Z

+
+    def test_setstate_missing_int(self):
+        # A missing required key must retain the mapping's KeyError and key.
+        for mapping in (dict, UserDict, MappingProxyType):


Suggested change

for mapping in (dict, UserDict, MappingProxyType):

for mapping in (dict, frozendict, UserDict, MappingProxyType):

picnixz · 2026-10-20T22:20:02Z

+    _c_uuid4 = uuid4
+    _c_uuid7 = uuid7
+
 def main():


Suggested change

_c_uuid4 = uuid4

_c_uuid7 = uuid7

def main():

_c_uuid4 = uuid4

_c_uuid7 = uuid7

def main():

picnixz · 2026-10-20T22:20:22Z

+_py_UUID = UUID
+try:
+    from _uuid import UUID, uuid4, uuid7
+    from _uuid import _install_c_hooks as _install_c_hooks


I'm not sure about the as _install_c_hooks relevance.

picnixz · 2026-10-20T22:20:55Z

+    _c_UUID = None
+    _c_uuid4 = None
+    _c_uuid7 = None


Suggested change

_c_UUID = None

_c_uuid4 = None

_c_uuid7 = None

_c_UUID = _c_uuid4 = _c_uuid7 = None

picnixz · 2026-10-20T22:21:12Z

+_py_uuid4 = uuid4
+_py_uuid7 = uuid7
+_py_UUID = UUID


Suggested change

_py_uuid4 = uuid4

_py_uuid7 = uuid7

_py_UUID = UUID

_py_UUID = UUID

_py_uuid4 = uuid4

_py_uuid7 = uuid7

picnixz · 2026-10-20T22:22:18Z

+Reimplement base UUID type, uuid4(), and uuid7() in C
+


Suggested change

Reimplement base UUID type, uuid4(), and uuid7() in C

:mod:`uuid`: Reimplement :class:`~uuid.UUID` objects, :func:`~uuid.uuid4`,

and :func:`~uuid.uuid7` in C.

Just to get some nice markup in the changelog.

picnixz

Mostly LGTM with a few nitpicks and questions (about __str__/__repr__/__setattr__).

picnixz · 2026-10-20T22:41:00Z

+
+    *counter = (((uint64_t)(high & 0x1FF) << 32) | (low >> 32)) & 0x1FFFFFFFFFF;
+    *tail = (uint32_t)low;
+    return 0;


@1st1 for visibility

picnixz · 2026-10-20T22:44:35Z

+
+/*[clinic input]
+@critical_section
+_uuid._install_c_hooks


Could you add a docstring for this one to say "Only for testing purposes."

picnixz · 2026-10-20T22:52:21Z

+    int i, j;
+
+    // Reimplement `hex = hex.replace('urn:', '').replace('uuid:', '')`
+    if (size > 0 && start[0] == 'u') {


There is a slight difference here where uuid:urn:... is also supported in the pure Python version. So what we could do is reject it in the pure Python version as well as it's not a legitimate change. More precisely, what is allowed are prefixes urn:uuid:, urn: and uuid:.

I would suggest that we change the pure Python implementation so that we only parse those forms and not those that contain uuid in the middle of the string like abcuuid:def (which would be 'abcdef').

Looking at one of your other comment, if you want to retain exact same semantics, you need to really re-implement the substring replacement (not just the prefix replacement).

picnixz · 2026-10-20T22:52:50Z

+    if (size >= 1 && start[0] == '{') {
+        start++;
+        size -= 1;
+    }
+    if (size >= 1 && start[size - 1] == '}') {
+        size -= 1;
+    }


Strictly speaking you need a while loop here for the strip. Either do it with a while loop or change the Python implementation to removeprefix('{') and removesuffix('}') so that we only strip one character and not consecutive ones.

picnixz · 2026-10-20T22:57:08Z

+    if (PyBytes_GET_SIZE(bytes_le) != UUID_SIZE) {
+        PyErr_SetString(PyExc_ValueError,
+            "bytes_le is not a 16-char string");


Suggested change

if (PyBytes_GET_SIZE(bytes_le) != UUID_SIZE) {

PyErr_SetString(PyExc_ValueError,

"bytes_le is not a 16-char string");

if (PyBytes_GET_SIZE(bytes_le) != UUID_SIZE) {

PyErr_Format(PyExc_ValueError,

"bytes_le is not a %d-char string", UUID_SIZE);

picnixz · 2026-10-20T23:11:30Z

+    // UUID string format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (36 chars)
+    char str[36];
+
+    for (int i = 0; i < 4; i++) {


If you unrolled the loops, is it faster?

picnixz · 2026-10-20T23:12:05Z

+    }
+
+    // Get the class name (sadly can't use tp_name -- we don't need the full name)
+    PyObject *cls_name = PyObject_GetAttrString((PyObject *)Py_TYPE(self), "__name__");


@1st1 for visibility

picnixz · 2026-10-20T23:13:31Z

+        return NULL;
+    }
+
+    PyObject *repr = PyUnicode_FromFormat("%U('%U')", cls_name, str_obj);


I don't remember now but if someone subclass the UUID class and has a different __str__, wouldn't it be better if we use %U('%S') then? using PyType_GetName() should also be enought.

picnixz · 2026-10-20T23:18:26Z

+}
+
+static int
+Uuid_setattr(PyObject *Py_UNUSED(self), PyObject *Py_UNUSED(name),


Do we really need to override the __setattr__? aren't we by default read-only here?

picnixz · 2026-10-20T23:25:22Z

+    state->uuid_type = (PyTypeObject *)PyType_FromMetaclass(
+        NULL,
+        module,
+        &Uuid_spec,
+        NULL
+    );


Suggested change

state->uuid_type = (PyTypeObject *)PyType_FromMetaclass(

NULL,

module,

&Uuid_spec,

NULL

);

state->uuid_type = (PyTypeObject *)PyType_FromModuleAndSpec(

module,

&Uuid_spec,

NULL

);

I think it's enough that way.

1st1 requested review from ambv, pablogsal and zooba September 18, 2025 13:14

1st1 requested review from ZeroIntensity and ericsnowcurrently as code owners September 18, 2025 13:14

bedevere-app Bot added the awaiting core review label Sep 18, 2025

1st1 changed the title ~~The C implementation considerably boosts the performance of the key UUID~~ Reimplement base UUID type, uuid4(), and uuid7() in C Sep 18, 2025

1st1 force-pushed the uuid_push branch 2 times, most recently from 2d0a381 to c4363f8 Compare September 18, 2025 13:22

1st1 changed the title ~~Reimplement base UUID type, uuid4(), and uuid7() in C~~ #139122: Reimplement base UUID type, uuid4(), and uuid7() in C Sep 18, 2025

1st1 changed the title ~~#139122: Reimplement base UUID type, uuid4(), and uuid7() in C~~ #gh-139122: Reimplement base UUID type, uuid4(), and uuid7() in C Sep 18, 2025

bedevere-app Bot mentioned this pull request Sep 18, 2025

Implement UUID functions and type in C #139122

Open

1st1 force-pushed the uuid_push branch 2 times, most recently from fa12192 to 4a5c2a2 Compare September 18, 2025 13:24

zooba reviewed Sep 18, 2025

View reviewed changes

StanFromIreland changed the title ~~#gh-139122: Reimplement base UUID type, uuid4(), and uuid7() in C~~ gh-139122: Reimplement base UUID type, uuid4(), and uuid7() in C Sep 18, 2025

picnixz reviewed Sep 18, 2025

View reviewed changes

1st1 force-pushed the uuid_push branch from 4a5c2a2 to d244f95 Compare September 19, 2025 10:47

1st1 force-pushed the uuid_push branch 2 times, most recently from da233b4 to ba9217e Compare September 19, 2025 10:59

picnixz reviewed Sep 19, 2025

View reviewed changes

Comment thread Misc/NEWS.d/next/Library/2025-09-18-14-13-00.gh-issue-139122.m3lp66.rst

eendebakpt reviewed Sep 19, 2025

View reviewed changes

Comment thread Modules/_uuidmodule.c Outdated

1st1 and others added 8 commits October 19, 2026 15:32

Stop importing unused SafeUUID members

39360c1

Drop SafeUUID.unknown

9c31cf7

Drop more things that can be imported dynamically

fe8ce01

Remove circular import; import 'uuid' from '_uuid' lazily

01bb137

Regen files

dff1e4f

Get module state assuming there could be subclasses of UUID

5b8f47f

This fixes a segfault; regr test added. While there, make the freelist only work with our base UUID type and no subclasses of it.

More ergonomic int cast code

cbf7226

pythongh-139122: Fix UUID subclass allocation and freelist type owner…

d43cb28

…ship

1st1 force-pushed the uuid_push branch from b10a1f7 to d43cb28 Compare October 19, 2026 22:35

1st1 requested a review from FFY00 as a code owner October 19, 2026 22:35

bedevere-app Bot added the type-feature A feature request or enhancement label Oct 19, 2026

1st1 added 3 commits October 19, 2026 15:51

pythongh-139122: Match UUID state restoration mapping semantics

436e39d

pythongh-139122: Fix UUID time hook error handling and reference updates

0c8c3a0

pythongh-139122: Align C UUID constructor inputs with Python

4e0388f

1st1 requested a review from AA-Turner as a code owner October 19, 2026 23:23

1st1 added 4 commits October 19, 2026 16:40

pythongh-139122: Make UUID narrowing and entropy lock requirements ex…

e543a0c

…plicit

pythongh-139122: Preserve UUID fields length errors

d09404a

pythongh-139122: Clarify version-specific UUID layouts

68e41aa

pythongh-139122: Apply UUID review cleanup

404ce3b

eendebakpt reviewed Oct 20, 2026

View reviewed changes

Comment thread Doc/whatsnew/3.16.rst

1st1 added 2 commits October 20, 2026 13:56

pythongh-139122: Fix concurrent UUID entropy refills and UUIDv7 state…

d22c571

… updates

pythongh-139122: Avoid subclass module-state lookup during UUID deall…

99ba7f7

…ocation

1st1 removed the stale Stale PR or inactive for long period of time. label Oct 20, 2026

1st1 added 2 commits October 20, 2026 14:09

pythongh-139122: Mark UUID testing hook as an explicit re-export

547e827

pythongh-139122: Document UUID performance improvements in What’s New

9e02f29

picnixz reviewed Oct 20, 2026

View reviewed changes

		self.assertEqual(
		self.uuid.UUID(*kwargs.values(), version=None), expected)

	self.assertEqual(self.uuid.UUID(**kwargs, version=None), expected)
	u_with_kwargs = self.uuid.UUID(**kwargs, version=None)
	self.assertEqual(u_with_kwargs, expected)

	for mapping in (dict, UserDict, MappingProxyType):
	for mapping in (dict, frozendict, UserDict, MappingProxyType):

	Reimplement base UUID type, uuid4(), and uuid7() in C
	:mod:`uuid`: Reimplement :class:`~uuid.UUID` objects, :func:`~uuid.uuid4`,
	and :func:`~uuid.uuid7` in C.

Conversation

1st1 commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot Bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

1st1 commented Sep 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

picnixz commented Sep 18, 2025

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

1st1 commented Sep 19, 2025

Uh oh!

picnixz commented Sep 19, 2025

Uh oh!

Uh oh!

1st1 commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

1st1 commented Sep 21, 2025

Uh oh!

read-the-docs-community Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Uh oh!

eendebakpt commented Oct 20, 2026 • edited by hugovk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

1st1 commented Oct 20, 2026

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

1st1 commented Sep 18, 2025 •

edited

Loading

python-cla-bot Bot commented Sep 18, 2025 •

edited

Loading

1st1 commented Sep 19, 2025 •

edited

Loading

read-the-docs-community Bot commented Oct 19, 2026 •

edited

Loading

eendebakpt commented Oct 20, 2026 •

edited by hugovk

Loading