stubtest: error if a dunder method is missing from a stub

[AlexWaygood]

Description

This PR alters stubtest so that it raises an error if a dunder method is missing from a stub. Currently, stubtest does raise an error for a few missing dunders (__call__, __class_getitem__, etc.). However, for the vast majority of them, it does not raise an error.

I have been using this patch to provide a number of fixes to typeshed over the past week or so:

Details

• Add missing dunder overrides in array, tracemalloc and unittest.mock typeshed#7248
• Improve ipaddress dunders typeshed#7244
• stdlib: Add several missing __(deep)copy__ methods typeshed#7242
• stdlib: Add many missing dunder overrides typeshed#7231
• types: Add several missing __(qual)name__ attributes typeshed#7216
• Add a few missing dunders in builtins typeshed#7214
• Add missing itertools.repeat.__length_hint__ method typeshed#7212
• Improve traceback.FrameSummary typeshed#7210
• Improve traceback.FrameSummary typeshed#7210
• collections: Add missing reflected BinOp methods typeshed#7207
• Add EnumMeta.__bool__ typeshed#7206
• Add missing __isabstractmethods__ attributes in abc and functools typeshed#7205
• Add missing descriptor methods in dataclasses and functools typeshed#7203
• stdlib: Add several missing comparison methods typeshed#7202

With these typeshed fixes merged, here are the new errors stubtest reports from checking typeshed, with this patch applied:

Details

+ _collections_abc.Mapping.__eq__ is inconsistent, stub argument "__o" should be positional or keyword (remove leading double underscore)
+ _collections_abc.Mapping.__reversed__ is not present in stub
+ _collections_abc.Set.__eq__ is inconsistent, stub argument "__o" should be positional or keyword (remove leading double underscore)
+ _collections_abc.Set.__rand__ is not present in stub
+ _collections_abc.Set.__ror__ is not present in stub
+ _collections_abc.Set.__rsub__ is not present in stub
+ _collections_abc.Set.__rxor__ is not present in stub
+ _collections_abc.dict_items.__and__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__contains__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__key")
+ _collections_abc.dict_items.__ge__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__gt__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__le__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__lt__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__or__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__rand__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__ror__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__rsub__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__rxor__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__sub__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_items.__xor__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__and__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__contains__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__key")
+ _collections_abc.dict_keys.__ge__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__gt__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__le__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__lt__ is inconsistent, stub argument "s" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__or__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__rand__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__ror__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__rsub__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__rxor__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__sub__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _collections_abc.dict_keys.__xor__ is inconsistent, stub argument "o" should be positional-only (rename with a leading double underscore, i.e. "__value")
+ _weakref.ProxyType.__bytes__ is not present in stub
+ _weakref.ProxyType.__reversed__ is not present in stub
+ builtins.property.__set_name__ is not present in stub
+ builtins.super.__self__ is not present in stub
+ builtins.super.__self_class__ is not present in stub
+ builtins.super.__thisclass__ is not present in stub
+ collections.ChainMap.__bool__ is not present in stub
+ collections.Counter.__missing__ is not present in stub
+ dbm.dumb._Database.__contains__ is inconsistent, stub argument "__o" should be positional or keyword (remove leading double underscore)
+ enum.EnumMeta.__prepare__ is inconsistent, stub argument "__name" should be positional or keyword (remove leading double underscore)
+ enum.EnumMeta.__prepare__ is inconsistent, stub argument "__bases" should be positional or keyword (remove leading double underscore)
+ functools.partial.__vectorcalloffset__ is not present in stub
+ http.cookies.BaseCookie.__str__ is inconsistent, stub does not have argument "attrs"
+ http.cookies.BaseCookie.__str__ is inconsistent, stub does not have argument "header"
+ http.cookies.BaseCookie.__str__ is inconsistent, stub does not have argument "sep"
+ http.cookies.Morsel.__str__ is inconsistent, stub does not have argument "attrs"
+ http.cookies.Morsel.__str__ is inconsistent, stub does not have argument "header"
+ lib2to3.pytree.Leaf.__unicode__ is not present in stub
+ lib2to3.pytree.Node.__unicode__ is not present in stub
+ platform.uname_result.__getitem__ is inconsistent, stub argument "__x" should be positional or keyword (remove leading double underscore)
+ pstats.FunctionProfile.__eq__ is inconsistent, stub argument "__o" should be positional or keyword (remove leading double underscore)
+ pstats.StatsProfile.__eq__ is inconsistent, stub argument "__o" should be positional or keyword (remove leading double underscore)
+ types.FunctionType.__builtins__ is not present in stub
+ types.LambdaType.__builtins__ is not present in stub
+ uuid.UUID.__setattr__ is inconsistent, stub argument "__name" should be positional or keyword (remove leading double underscore)
+ uuid.UUID.__setattr__ is inconsistent, stub argument "__value" should be positional or keyword (remove leading double underscore)
+ weakref.ProxyType.__bytes__ is not present in stub
+ weakref.ProxyType.__reversed__ is not present in stub
+ xml.dom.minidom.Node.__bool__ is not present in stub

Notes on the new errors reported

• We have a few new complaints about discrepancies between typing and collections.abc. These just need to be allowlisted imo, unless and until we find a good general solution to that problem (refs Decouple collections.abc types from typing aliases typeshed#6257)
• There are a lot of complaints about dict_keys, dict_values and dict_items having positional-only differences with typing.KeysView, typing.ValuesView and typing.ItemsView. The way to resolve that would be essentially duplicate every method from typing.{Keys, Values, Items}View into those classes (but with pos-only args instead of pos-or-kw args). I don't really have the stomach to do that. These should probably just be allowlisted.
• weakref.ProxyType, dbm.dum._Database, lib2to3.pytree.Leaf: I don't really know anything about these classes.
• enum.EnumMeta.__prepare__: this will be resolved by Add EnumMeta.__prepare__ typeshed#7243
• super has some really weird dunders.
• property.__set_name__: this is a false positive -- it doesn't exist in real life, just needs to be allowlisted.
• http.cookies has some really weird __str__ methods.
• functools.partial.__vectorcalloffset__: no idea what this does or is for.
• platform.uname_result is a mess in CPython.
• pstats hits: these relate to a mypy bug in dataclasses __eq__ method generation (Mypy incorrectly infers that parameters are positional-only for __eq__ methods autogenerated with @dataclass #12186).
• collections.Counter.__missing__: this always raises an exception, but per Why is collections.ChainMap.__missing__ defined in the stub? typeshed#7188, we could maybe add it to the stub.
• There's a few missing __bool__ overrides stubtest now complains about: we could add these to the stub, though they don't seem particularly useful, since they don't alter the default behaviour of bool(x) at all. (But I don't think __bool__ should be added to the list of ignored dunders: it was useful for this patch to flag the missing __bool__ method fixed in Add EnumMeta.__bool__ typeshed#7206, as that is an override that changes behaviour from the default.)
• uuid.UUID.__setattr__: this always raises an exception, I think it should just be allowlisted.

Test Plan

I've altered the existing tests so they continue to pass, and added two new test cases.

cc. @hauntsaninja: here is the promised follow-up PR!

[AlexWaygood]

This is a fairly arbitrary ignorelist, based on the hits that this patch initially came up with, combined with my judgement as to which were worth fixing and which weren't. I'm happy to add or remove items!

[hauntsaninja]

This is great, thanks for doing this!

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[AlexWaygood]

Thanks! :D