gh-114563: C decimal falls back to pydecimal for unsupported format strings

[belm0]

When the mpdecimal lib fails to parse a format string, we fall back to the pydecimal format implementation.

Notably this allows complex workarounds for 'z' format support to be removed. (mpdecimal added 'z' support in version 4.0.0, so this becomes efficient in the long term.)

• remove 'z' format implementation from C decimal
• fix 'z' format memory leak
• fix 'z' format applied to type 'F'
• fix missing '#' format support

Suggested and prototyped by Stefan Krah.

Fixes gh-114563, gh-91060

[mdickinson]

LGTM, with one suggestion about the exception handling for the __format__ call.

@skrah Thank you for your help with this. If you happen to have the bandwidth and inclination to take a quick glance at this PR, I'd very much appreciate your input.

[skrah]

If I checkout d96a8cd and then apply my own patches, I reach the exact same state as in this PR.

When comparing the result to 3.9, the diff in the dec_format area is clean and manageable.

I tested my own patches with deccheck (z-format extended version), so I approve!

@serhiy-storchaka's pydec_format changes look correct to me, but I'd have to look up some of the new functions, so I'll stay out of that part.

[serhiy-storchaka]

LGTM.

Is it okay that the PyDecimal instance can have a different context?

[skrah]

Is it okay that the PyDecimal instance can have a different context?

This is an excellent observation! It is the one thing that cannot be caught by deccheck, because the _decimal and _pydecimal contexts are always in sync. _pydecimal.__format__ needs rounding and capitals.

Maybe the easiest thing would be to add another private method to _pydecimal, something like:

def fallback_format(self, specifier, rounding, caps):
    # make a new context with rounding and caps
    # return __format__ with that context

[skrah]

This is a clever use of the _decimal context! It works because ctx.rounding is a string and ctx.capitals is a bool. For others who happen to come by this issue, I want to (pedantically) point out that the _pydecimal and _decimal contexts are not in general interchangeable.

LGTM.

[serhiy-storchaka]

Now please add a NEWS entry (it is a user visible change), and it will be done.

Describe what behavior was changed, not implementation details.

[miss-islington-app]

Thanks @belm0 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[miss-islington-app]

Thanks @belm0 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @belm0 and @serhiy-storchaka, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 72340d15cdfdfa4796fdd7c702094c852c2b32d2 3.12

[miss-islington-app]

Sorry, @belm0 and @serhiy-storchaka, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 72340d15cdfdfa4796fdd7c702094c852c2b32d2 3.11

[serhiy-storchaka]

@belm0, do you mind to create a backport for 3.12?

It will be easier to backport then from 3.12 to 3.11 than directly to 3.11.

[bedevere-app]

GH-115353 is a backport of this pull request to the 3.12 branch.

[bedevere-app]

GH-115384 is a backport of this pull request to the 3.11 branch.