Fixing pickle for nested classes - Utilities

Skip to content
Back to top
View this page
Edit this page

# Fixing pickle for nested classes

As of Python 2.7, names for nested classes are set by Python in a way which is incompatible with the pickling of such classes (pickling by name):

Sage

    sage: class A:
    ....:     class B:
    ....:         pass
    sage: A.B.__name__
    'B'

Python

    >>> from sage.all import *
    >>> class A:
    ...     class B:
    ...         pass
    >>> A.B.__name__
    'B'

instead of more natural `'A.B'`. Furthermore upon pickling and unpickling a class with name `'A.B'` in a module `mod`, the standard cPickle module searches for `'A.B'` in `mod.__dict__` instead of looking up `'A'` and then `'B'` in the result. See: https://groups.google.com/forum/#!topic/sage-devel/bHBV9KWAt64

This module provides two utilities to workaround this issue:

- `nested_pickle()` “fixes” recursively the name of the subclasses of a class and inserts their fullname `'A.B'` in `mod.__dict__`

- `NestedClassMetaclass` is a metaclass ensuring that `nested_pickle()` is called on a class upon creation.

See also `sage.misc.test_nested_class`.

Note

In Python 3, nested classes, like any class for that matter, have `__qualname__` and the standard pickle module uses it for pickling and unpickling. Thus the pickle module searches for `'A.B'` first by looking up `'A'` in `mod`, and then `'B'` in the result. So there is no pickling problem for nested classes in Python 3, and the two utilities are not really necessary. However, `NestedClassMetaclass` is used widely in Sage and affects behaviors of Sage objects in other respects than in pickling and unpickling. Hence we keep `NestedClassMetaclass` even with Python 3, for now. This module will be removed when we eventually drop support for Python 2.

EXAMPLES:

Sage

    sage: from sage.misc.nested_class import A1, nested_pickle

    sage: A1.A2.A3.__name__
    'A3'
    sage: A1.A2.A3
    <class 'sage.misc.nested_class.A1.A2.A3'>

    sage: nested_pickle(A1)
    <class 'sage.misc.nested_class.A1'>

    sage: A1.A2
    <class 'sage.misc.nested_class.A1.A2'>

    sage: A1.A2.A3
    <class 'sage.misc.nested_class.A1.A2.A3'>
    sage: A1.A2.A3.__name__
    'A1.A2.A3'

    sage: sage.misc.nested_class.__dict__['A1.A2'] is A1.A2
    True
    sage: sage.misc.nested_class.__dict__['A1.A2.A3'] is A1.A2.A3
    True

[Python]

    >>> from sage.all import *
    >>> from sage.misc.nested_class import A1, nested_pickle

    >>> A1.A2.A3.__name__
    'A3'
    >>> A1.A2.A3
    <class 'sage.misc.nested_class.A1.A2.A3'>

    >>> nested_pickle(A1)
    <class 'sage.misc.nested_class.A1'>

    >>> A1.A2
    <class 'sage.misc.nested_class.A1.A2'>

    >>> A1.A2.A3
    <class 'sage.misc.nested_class.A1.A2.A3'>
    >>> A1.A2.A3.__name__
    'A1.A2.A3'

    >>> sage.misc.nested_class.__dict__['A1.A2'] is A1.A2
    True
    >>> sage.misc.nested_class.__dict__['A1.A2.A3'] is A1.A2.A3
    True

All of this is not perfect. In the following scenario:

Sage

    sage: class A1:
    ....:     class A2:
    ....:         pass
    sage: class B1:
    ....:     A2 = A1.A2

    sage: nested_pickle(A1)
    <class '__main__.A1'>
    sage: nested_pickle(B1)
    <class '__main__.B1'>
    sage: A1.A2
    <class '__main__.A1.A2'>
    sage: B1.A2
    <class '__main__.A1.A2'>

Python

    >>> from sage.all import *
    >>> class A1:
    ...     class A2:
    ...         pass
    >>> class B1:
    ...     A2 = A1.A2

    >>> nested_pickle(A1)
    <class '__main__.A1'>
    >>> nested_pickle(B1)
    <class '__main__.B1'>
    >>> A1.A2
    <class '__main__.A1.A2'>
    >>> B1.A2
    <class '__main__.A1.A2'>

The name for `'A1.A2'` could potentially be set to `'B1.A2'`. But that will work anyway.

class sage.misc.nested_class.MainClass

  Bases: `object`

  A simple class to test nested_pickle.

  EXAMPLES:

  Sage

      sage: from sage.misc.nested_class import *
      sage: loads(dumps(MainClass()))
      <sage.misc.nested_class.MainClass object at 0x...>

  Python

      >>> from sage.all import *
      >>> from sage.misc.nested_class import *
      >>> loads(dumps(MainClass()))
      <sage.misc.nested_class.MainClass object at 0x...>

  class NestedClass

    Bases: `object`

    EXAMPLES:

    Sage

        sage: from sage.misc.nested_class import *
        sage: loads(dumps(MainClass.NestedClass()))
        <sage.misc.nested_class.MainClass.NestedClass object at 0x...>

    Python

        >>> from sage.all import *
        >>> from sage.misc.nested_class import *
        >>> loads(dumps(MainClass.NestedClass()))
        <sage.misc.nested_class.MainClass.NestedClass object at 0x...>

    class NestedSubClass

      Bases: `object`

      EXAMPLES:

      Sage

          sage: from sage.misc.nested_class import *
          sage: loads(dumps(MainClass.NestedClass.NestedSubClass()))
          <sage.misc.nested_class.MainClass.NestedClass.NestedSubClass object at 0x...>
          sage: getattr(sage.misc.nested_class, 'MainClass.NestedClass.NestedSubClass')
          <class 'sage.misc.nested_class.MainClass.NestedClass.NestedSubClass'>
          sage: MainClass.NestedClass.NestedSubClass.__name__
          'MainClass.NestedClass.NestedSubClass'

      Python

          >>> from sage.all import *
          >>> from sage.misc.nested_class import *
          >>> loads(dumps(MainClass.NestedClass.NestedSubClass()))
          <sage.misc.nested_class.MainClass.NestedClass.NestedSubClass object at 0x...>
          >>> getattr(sage.misc.nested_class, 'MainClass.NestedClass.NestedSubClass')
          <class 'sage.misc.nested_class.MainClass.NestedClass.NestedSubClass'>
          >>> MainClass.NestedClass.NestedSubClass.__name__
          'MainClass.NestedClass.NestedSubClass'

      dummy(x, r=(1, 2, 3.4), *args, **kwds)

        A dummy method to demonstrate the embedding of method signature for nested classes.

class sage.misc.nested_class.NestedClassMetaclass

  Bases: `type`

  A metaclass for nested pickling.

  Check that one can use a metaclass to ensure nested_pickle is called on any derived subclass:

  Sage

      sage: from sage.misc.nested_class import NestedClassMetaclass
      sage: class ASuperClass(object, metaclass=NestedClassMetaclass):
      ....:     pass
      sage: class A3(ASuperClass):
      ....:     class B():
      ....:         pass
      sage: A3.B.__name__
      'A3.B'
      sage: getattr(sys.modules['__main__'], 'A3.B', 'Not found')
      <class '__main__.A3.B'>

  Python

      >>> from sage.all import *
      >>> from sage.misc.nested_class import NestedClassMetaclass
      >>> class ASuperClass(object, metaclass=NestedClassMetaclass):
      ...     pass
      >>> class A3(ASuperClass):
      ...     class B():
      ...         pass
      >>> A3.B.__name__
      'A3.B'
      >>> getattr(sys.modules['__main__'], 'A3.B', 'Not found')
      <class '__main__.A3.B'>

sage.misc.nested_class.modify_for_nested_pickle(cls, name_prefix, module, first_run=True)

  Modify the subclasses of the given class to be picklable, by giving them a mangled name and putting the mangled name in the module namespace.

  INPUT:

  - `cls` – the class to modify

  - `name_prefix` – the prefix to prepend to the class name

  - `module` – the module object to modify with the mangled name

  - `first_run` – boolean (default: `True`); whether or not this function is run for the first time on `cls`

  NOTE:

  This function would usually not be directly called. It is internally used in `NestedClassMetaclass`.

  EXAMPLES:

  Sage

      sage: from sage.misc.nested_class import *
      sage: class A():
      ....:     class B():
      ....:         pass
      sage: module = sys.modules['__main__']
      sage: A.B.__name__
      'B'
      sage: getattr(module, 'A.B', 'Not found')
      'Not found'
      sage: modify_for_nested_pickle(A, 'A', module)
      sage: A.B.__name__
      'A.B'
      sage: getattr(module, 'A.B', 'Not found')
      <class '__main__.A.B'>

  Python

      >>> from sage.all import *
      >>> from sage.misc.nested_class import *
      >>> class A():
      ...     class B():
      ...         pass
      >>> module = sys.modules['__main__']
      >>> A.B.__name__
      'B'
      >>> getattr(module, 'A.B', 'Not found')
      'Not found'
      >>> modify_for_nested_pickle(A, 'A', module)
      >>> A.B.__name__
      'A.B'
      >>> getattr(module, 'A.B', 'Not found')
      <class '__main__.A.B'>

  Here we demonstrate the effect of the `first_run` argument:

  Sage

      sage: modify_for_nested_pickle(A, 'X', module)
      sage: A.B.__name__  # nothing changed
      'A.B'
      sage: modify_for_nested_pickle(A, 'X', module, first_run=False)
      sage: A.B.__name__
      'X.A.B'

  [Python]

      >>> from sage.all import *
      >>> modify_for_nested_pickle(A, 'X', module)
      >>> A.B.__name__  # nothing changed
      'A.B'
      >>> modify_for_nested_pickle(A, 'X', module, first_run=False)
      >>> A.B.__name__
      'X.A.B'

  Note that the class is now found in the module under both its old and its new name:

  Sage

      sage: getattr(module, 'A.B', 'Not found')
      <class '__main__.A.B'>
      sage: getattr(module, 'X.A.B', 'Not found')
      <class '__main__.A.B'>

  Python

      >>> from sage.all import *
      >>> getattr(module, 'A.B', 'Not found')
      <class '__main__.A.B'>
      >>> getattr(module, 'X.A.B', 'Not found')
      <class '__main__.A.B'>

sage.misc.nested_class.nested_pickle(cls)

  This decorator takes a class that potentially contains nested classes. For each such nested class, its name is modified to a new illegal identifier, and that name is set in the module. For example, if you have:

  Sage

      sage: from sage.misc.nested_class import nested_pickle
      sage: module = sys.modules['__main__']
      sage: class A():
      ....:     class B:
      ....:         pass
      sage: nested_pickle(A)
      <class '__main__.A'>

  Python

      >>> from sage.all import *
      >>> from sage.misc.nested_class import nested_pickle
      >>> module = sys.modules['__main__']
      >>> class A():
      ...     class B:
      ...         pass
      >>> nested_pickle(A)
      <class '__main__.A'>

  then the name of class `'B'` will be modified to `'A.B'`, and the `'A.B'` attribute of the module will be set to class `'B'`:

  Sage

      sage: A.B.__name__
      'A.B'
      sage: getattr(module, 'A.B', 'Not found')
      <class '__main__.A.B'>

  [Python]

      >>> from sage.all import *
      >>> A.B.__name__
      'A.B'
      >>> getattr(module, 'A.B', 'Not found')
      <class '__main__.A.B'>

  In Python 2.6, decorators work with classes; then `@nested_pickle` should work as a decorator:

  Sage

      sage: @nested_pickle    # todo: not implemented
      ....: class A2():
      ....:     class B:
      ....:         pass
      sage: A2.B.__name__    # todo: not implemented
      'A2.B'
      sage: getattr(module, 'A2.B', 'Not found')    # todo: not implemented
      <class __main__.A2.B at ...>

  Python

      >>> from sage.all import *
      >>> @nested_pickle    # todo: not implemented
      ... class A2():
      ...     class B:
      ...         pass
      >>> A2.B.__name__    # todo: not implemented
      'A2.B'
      >>> getattr(module, 'A2.B', 'Not found')    # todo: not implemented
      <class __main__.A2.B at ...>

  EXAMPLES:

  Sage

      sage: from sage.misc.nested_class import *
      sage: loads(dumps(MainClass.NestedClass())) # indirect doctest
      <sage.misc.nested_class.MainClass.NestedClass object at 0x...>

  [Python]

      >>> from sage.all import *
      >>> from sage.misc.nested_class import *
      >>> loads(dumps(MainClass.NestedClass())) # indirect doctest
      <sage.misc.nested_class.MainClass.NestedClass object at 0x...>