Skip to content

Keep emptied [table]-header tables (incl. out-of-order fragments) in the output - #688

Closed
mayur-dhavan wants to merge 18 commits into
python-poetry:masterfrom
mayur-dhavan:fix-emptied-table-serialization
Closed

mayur-dhavan wants to merge 18 commits into
python-poetry:masterfrom
mayur-dhavan:fix-emptied-table-serialization

Conversation

@mayur-dhavan

@mayur-dhavan mayur-dhavan commented Oct 25, 2026 •

Copy link
Copy Markdown

Fixes the remaining half of #553 (see "Scope" below - #590 covers the dotted-key half).

Problem

After removing the last key of a table that was materialized from [table] headers (not from a dotted key), dumps() silently drops the table even though it is still part of the document:

doc = tomlkit.loads("[x.y]\nz = 1\n")
del doc["x"]["y"]
tomlkit.dumps(doc)  # before: '' - the x table vanishes; in-memory doc is {'x': {}}

The same happens to emptied fragments of an out-of-order table: del doc["bar"]["a"] leaves bar == {} in the document, but dumps() renders nothing for it. The emptied table's body only holds a Null placeholder, so is_super_table() classifies it as a super table and no header is emitted. Empty tables are valid TOML ("Empty tables are allowed and simply have no key/value pairs within them", TOML v1.0.0), and dropping them loses data on round-trip.

Fix

In Container._render_table, when a table with a non-dotted key holds no real items (only whitespace or Null placeholders), emit its header anyway. For fragments of an out-of-order table, a fragment is skipped when a sibling fragment still holds the table's items, so no duplicate headers appear.

Deleting a table itself (del doc["x"]) still removes it from the output entirely; only tables that remain in the document keep their header. This extends the direction of the merged #567 (emptied [[aot]] renders as a = []) from arrays of tables to plain tables.

Scope

#590 fixes the dotted-key half of #553 (emptied tables materialized from x.y.z = 1 style keys, key.is_dotted()), and explicitly leaves out-of-order tables and non-dotted keys untouched. This PR covers exactly that remainder: non-dotted keys, including out-of-order fragments. The two PRs are complementary; neither changes the other's behavior. Happy to rebase if #590 lands first.

Behavior change

test_delete_out_of_order_table_key previously asserted that an emptied out-of-order table vanishes from the output. Since del doc["bar"]["a"] leaves bar == {} in the document, dropping it loses data; the test now expects the empty [bar] header. Round-trip checked with stdlib tomllib: the new output parses back to exactly the in-memory document.

Testing

  • test_delete_out_of_order_table_key (updated expectation) and new test_deleting_last_child_table_keeps_emptied_parent fail on master, pass with this fix.
  • New guards that pass before and after: test_emptied_explicit_table_still_serializes (explicit [x.y] header survives emptying) and test_deleting_table_removes_it_from_output (whole-table deletion still removes it).
  • Dotted-key case verified unchanged from master (renders empty, as today - that is Render the header of emptied super tables created from dotted keys #590's scope).
  • Full suite: 1206 passed on Python 3.13.

This PR was prepared with AI assistance (an automated coding agent acting on behalf of the author).

Tables left with no items after removals now emit their header again instead of vanishing from dumps() (python-poetry#553).
Covers dotted-key and explicit tables emptied by removals, and updates the out-of-order deletion expectation (python-poetry#553).
Copilot AI balanced review requested due to automatic review settings October 25, 2026 18:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@mayur-dhavan

Copy link
Copy Markdown
Author

Prior-art note for reviewers: #590 (open since Aug 26) fixes the same dotted-key super-table half of #553 with a similar approach, and explicitly leaves out-of-order tables untouched. This PR covers that case plus the remaining out-of-order fragment path (an emptied fragment no longer renders when a sibling fragment holds the table's items, and an emptied out-of-order table keeps its [header] since it is still in the document). Happy to rebase or narrow this to only the out-of-order part if #590 lands first. The AoT half of #553 was already handled by the merged #567.

mayur-dhavan and others added 6 commits October 26, 2026 00:31
Tables materialized from dotted keys stay as they were (covered by python-poetry#590); emptied tables from [table] headers keep their header.
Drop the dotted-key regression test (covered by python-poetry#590); keep out-of-order and explicit-table cases plus explicit deletion checks.
@mayur-dhavan mayur-dhavan changed the title Keep emptied tables in the serialized output Keep emptied [table]-header tables (incl. out-of-order fragments) in the output Oct 25, 2026
@mayur-dhavan

Copy link
Copy Markdown
Author

Update: as promised above, this PR has now been narrowed to only the non-dotted-key / out-of-order-fragment half, removing the dotted-key change that overlapped #590. Title and body rewritten to the precise residual scope; the dotted-key case renders exactly as on master. Full suite: 1206 passed.

@dimbleby

Copy link
Copy Markdown
Contributor
doc = tomlkit.parse("[a]\nx=1\n[t.a.b]\nx=1\n")
del doc["t"]["a"]["b"]

The sibling-fragment check looked at the root body, so an unrelated same-named top-level table could suppress the header of an emptied nested table.
Add test for emptied nested table with same named top-level table.
@mayur-dhavan

Copy link
Copy Markdown
Author

Good catch - that case exposed a real bug in the first version of this change. The emptied t.a still vanished (output was [a]\nx=1\n, losing t), because my sibling-fragment check compared against the root container body, where the unrelated top-level [a] collided with the nested a and suppressed the header.

Fixed in the latest commit: the check now compares only within the container body that actually owns the fragment. Your example now renders:

[a]
x=1
[t.a]

which round-trips to exactly the in-memory document ({'a': {'x': 1}, 't': {'a': {}}}). Added test_emptied_nested_table_with_same_named_top_level_table as a regression test; full suite 1207 passed.

@dimbleby

Copy link
Copy Markdown
Contributor

You need to put more effort into critical review of your fixes

doc = tomlkit.parse("[t.a.b]\nx=1\n[t]\na.c=1\n")
del doc["t"]["a"]["b"]
doc = tomlkit.parse("[t.a.b]\nx=1\n[q]\nx=1\n[t.a]\nc=1\n")
del doc["t"]["a"]["b"]

mayur-dhavan and others added 6 commits October 26, 2026 02:56
Emptied tables of an out-of-order (fragmented) parent were serialized
without checking the other fragments, producing invalid TOML: an emptied
fragment could emit its [header] while another fragment still rendered
the same table, defining it twice.

The dedup now considers the root fragment group (tuple entries in the
root container map): an emptied fragment only keeps its header when no
fragment of the group renders real content, and a shared per-render set
guarantees each emptied header is emitted at most once. Super tables can
also keep their header when they would otherwise vanish while still
being present in the document.
Emptied tables of an out-of-order (fragmented) parent were serialized
without checking the other fragments, producing invalid TOML: an emptied
fragment could emit its [header] while another fragment still rendered
the same table, defining it twice.

The dedup now considers the root fragment group (tuple entries in the
root container map): an emptied fragment only keeps its header when no
fragment of the group renders real content, and a shared per-render set
guarantees each emptied header is emitted at most once. Super tables
also keep their header when they would otherwise vanish while still
being present in the document.
Cover the cases where a deleted table's parent has later or interleaved
fragments of the same root table (the emptied fragment must be dropped so
the output never redefines a table), the case where every fragment is
emptied (each emptied sub-table keeps exactly one header), and deletion
of an implicit table's last child (the emptied parent is kept).
@mayur-dhavan

Copy link
Copy Markdown
Author

You're right, and thank you for the probes. Both reproduce on the previous commit and both outputs were invalid TOML (duplicate / conflicting table headers). The root cause: my dedup only compared a table against its same-container siblings, so it completely missed the case that matters here - fragments of the same out-of-order root table live in separate container bodies, keyed as a tuple in the root container's map.

Reworked the logic: when an emptied table would emit its [header], the renderer now looks up the root key in the root container map. If the root key is an out-of-order fragment group, the emptied fragment keeps its header only when no fragment of the group renders real content, and a per-render set guarantees each emptied header is emitted at most once. The same rule covers implicit (super) tables that would otherwise vanish while still present in the document.

Your cases now:

  • tomlkit.parse("[t.a.b]\nx=1\n[t]\na.c=1\n"), del doc["t"]["a"]["b"] → [t]\na.c=1\n (the emptied fragment is dropped; [t] renders the remaining content)
  • tomlkit.parse("[t.a.b]\nx=1\n[q]\nx=1\n[t.a]\nc=1\n"), del doc["t"]["a"]["b"] → [q]\nx=1\n[t.a]\nc=1\n

Both outputs parse back with tomllib and round-trip equal to doc.unwrap(). I've added regression tests for both of your cases plus the all-fragments-emptied case, and ran a broader sweep of out-of-order / deletion scenarios checking validity and round-trip fidelity against tomllib (10/10 clean). Full test suite passes locally (1210 tests). CI here is green apart from the repo-wide ubuntu-latest / 3.9 runner failure that currently affects every PR.

@mayur-dhavan

Copy link
Copy Markdown
Author

Closing this. Independent validation of the latest head (property-based fuzzing over nesting / ordering / deletion combinations) found that the emptied-table handling still produces invalid TOML for some out-of-order fragment shapes - for example [c.b.b]\ny = 1\n[c.b]\n[c]\n with del doc["c"]["b"]["b"] serializes with [c.b] declared twice. Patching fragment dedup at render time is not holding up as an approach, so I am withdrawing this PR rather than asking for more review cycles on it.

Thank you @dimbleby for the time and the sharp probes - they exposed the first round of these problems and made the real difficulty of this corner of the serializer clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants