Building on cpyext

The same C core runs on CPython and on PyPy, with no pure-Python fallback and no second backend. This page covers what had to change to make that true, and the invariant that keeps it from costing CPython anything. What cpyext costs a user is on Interpreters.

Most of the C API surface needs no adaptation. cpyext emulates the PEP 393 layout, so PyUnicode_KIND, PyUnicode_DATA, PyUnicode_READ and PyUnicode_WRITE all work, and the heap types, module state, multi-phase init, and GC slots the module is built on are supported. Py_BEGIN_CRITICAL_SECTION collapses to a brace no-op there through the same #ifndef that CPython 3.10 through 3.12 take, and for the same reason: cpyext holds the GIL across every call into the extension.

The CPython codegen is untouched

src/turbohtml/_c/core/pycompat.h holds every adaptation, and each name in it expands to the CPython spelling on CPython. TH_SEALED becomes Py_TPFLAGS_DISALLOW_INSTANTIATION, TH_SEALED_END becomes {0, NULL}, and th_str_format(...) becomes PyUnicode_FromFormat(__VA_ARGS__). A CPython translation unit’s preprocessed token stream is therefore unchanged, so the profile-guided and link-time-optimized layouts the release wheels depend on cannot shift. Every PyPy-only code path sits behind #ifdef PYPY_VERSION and never reaches the CPython compiler.

The invariant is checkable rather than aspirational: compile every translation unit from main and from a branch at -O3 -g0 -DNDEBUG and compare the normalized objdump -d output. Anything that differs and is not a deliberate behavior change is a regression.

tests/core/test_c_api_portability.py pins the other half. Each guarded spelling may be named only inside pycompat.h; everywhere else the C sources must reach it through the wrapper. Reaching for the raw spelling compiles, passes on CPython, and is silently wrong on PyPy, which no compiler catches.

Sealing the types

Document(), Node(), Token() and eleven siblings raise TypeError, because only a parse builds them and one constructed by hand carries no tree. CPython enforces that with Py_TPFLAGS_DISALLOW_INSTANTIATION. cpyext ignored that flag until PyPy 7.3.21 (pypy#5318), so those types would construct with no tree attached and segfault on first use, letting pure Python code take the interpreter down. 7.3.21 honors the flag and prints a debug line to stdout for every type that sets it (pypy#5388), which corrupts anything reading the CLI’s output.

Neither costs anything to avoid. cpyext also seals a type through an explicit tp_new, the branch the flag would otherwise shadow, so on PyPy the flag comes off and a tp_new that raises goes on. Every supported PyPy then refuses the same constructions with the same message, and 7.3.21 stays quiet because nothing sets the flag it prints for. A subtype declaring its own tp_new overrides the inherited one, so Element("div") and Text("hi") build as they do on CPython.

The three other gaps

PyUnicode_CopyCharacters does not exist in cpyext, so it gets a PyUnicode_READ/PyUnicode_WRITE loop.

PyUnicode_FromFormat returns a string cpyext has not put in canonical form, so PyUnicode_KIND, PyUnicode_DATA, and PyUnicode_GET_LENGTH are all undefined on it, and with NDEBUG their assertions are compiled out, so GET_LENGTH answers one past the code point count (pypy#5524). The core readies every result before touching it.

cpyext cannot take a 2-byte buffer at all (pypy#5525). It materializes one by decoding it as UTF-16, so it eats a leading U+FEFF as a byte-order mark, byte-swaps the rest of the string after a leading U+FFFE, collapses a surrogate pair into the one code point it encodes, and aborts the interpreter on a lone surrogate through the strict error handler. A CPython str carries a lone surrogate fine, and both HTML input and the WHATWG tokenizer have to preserve one. cpyext’s 1-byte and 4-byte paths are exact, so on PyPy any result too wide for Latin-1 is built at 4-byte kind. CPython keeps the narrowest kind, because its str equality compares kind before content and would report a too-wide string as unequal to its own value.

encoding/decode.h calls PyUnicode_New directly and deliberately: every WHATWG decoder replaces an ill-formed sequence with U+FFFD and no legacy encoding maps a byte to a surrogate, so its 2-byte results hold none, and PyUnicode_New pins the byte order where PyUnicode_FromKindAndData does not.

A fourth difference lives outside that header. A negative subscript reaches sq_item unadjusted (pypy#5526) where CPython adds the sequence length first, so dom/node.c does that adjustment itself.