Persist values for a known set of keys in Python

1. Separation of concerns

There are three main features in this code:

1. Restricting the keys that can be stored in a dictionary.

2. Presenting a dictionary as a namespace.

3. Adding context manager behaviour to shelves in older versions of Python.

The principle of separation of concerns indicates that these should be implemented separately. See §3 below for one way to do this. This makes the code more understandable, testable, and reuseable.

2. Other review points

1. There are no docstrings. What does this code do? How do I use it?

2. The code in __exit__ assumes that Shelf objects have context manager behaviour in Python 3.0 or later. But the documentation says that this behaviour was only added in Python 3.4.

3. Storing _keys as a list means that checking to see if a key is valid takes time proportional to the number of keys. It would be better to store the keys in a set so that checking validity takes constant time.

4. There are implementations of __getattr__ and __setattr__ but not __delattr__.

5. A caller who tried to store a key check_key in the shelve would end up overwriting the method of that name.

3. Revised code

1. Restricting the keys that can be stored in a dictionary.

   import collections.abc
   
   class RestrictedMapping(collections.abc.MutableMapping):
       """A proxy for a mapping that restricts the set of allowable keys.
   
       >>> d = RestrictedMapping({0:0, 1:1}, range(3))
       >>> del d[1]
       >>> d[2] = 2
       >>> d[3] = 3
       Traceback (most recent call last):
           ...
       KeyError: 3
       >>> sorted(d.items())
       [(0, 0), (2, 2)]
   
       """
       def __init__(self, mapping, keys):
           keys = set(keys)
           for key in mapping:
               if key not in keys:
                   raise ValueError('invalid key {}'.format(key))
           self.keys = keys
           self.mapping = mapping
           super(RestrictedMapping, self).__init__()
   
       def _check_key(self, key):
           if key not in self.keys:
               raise KeyError(key)
   
       def __getitem__(self, key):
           self._check_key(key)
           return self.mapping[key]
   
       def __setitem__(self, key, value):
           self._check_key(key)
           self.mapping[key] = value
   
       def __delitem__(self, key):
           self._check_key(key)
           del self.mapping[key]
   
       def __iter__(self):
           return iter(self.mapping)
   
       def __len__(self):
           return len(self.mapping)
   

2. Presenting a dictionary as a namespace.

   class Namespace(object):
       """A proxy for a mapping that provides access to items via attribute
       lookup.
   
       >>> d = dict(a=1, b=2, c=3)
       >>> n = Namespace(d)        # n is now a proxy for d
       >>> n.d = 4
       >>> n.a, n.b, n.c, n.d
       (1, 2, 3, 4)
       >>> del n.b
       >>> sorted(d.items())
       [('a', 1), ('c', 3), ('d', 4)]
   
       """
       def __init__(self, mapping):
           super(Namespace, self).__setattr__('_mapping', mapping)
   
       def __getattr__(self, attr):
           return self._mapping[attr]
   
       def __setattr__(self, attr, value):
           self._mapping[attr] = value
   
       def __delattr__(self, attr):
           del self._mapping[attr]
   

3. Combining restriction on keys, presentation as namespace, and addition of context manager behaviour.

   from contextlib import contextmanager
   import shelve
   
   @contextmanager
   def restricted_shelve(filename, keys):
       shelf = shelve.open(filename)
       try:
           yield Namespace(RestrictedMapping(shelf, keys))
       finally:
           shelf.close()
   

4. Namespace antipattern

Presenting a mapping as a namespace instead of a dictionary is superficially attractive because n.name is three characters shorter than d['name'], but it's an antipattern in Python. (The only place this pattern is used in the standard library is in the argparse module.)

The problems are:

1. Attribute names need to be syntactically valid as Python identifiers, so you have to be very sure you're never going to need an attribute called class or return or 1 or whatever.

2. There will be collisions between the data attributes and the implementation attributes. In my version in §3.2 above you can't have attributes named _mapping or __init__ or __dict__ or __slots__ or ...

3. There's no easy way to get at the list of attributes.

4. There's not much you can do with a namespace-like object: the Python standard library usually expects you to use mappings instead. For example, you can't pass a namespace to itertools.ChainMap, or convert it to JSON using json.dumps, or output it as CSV using csv.DictWriter, and so on.

5. Answers to questions

1. As far as I know, there's no easy way to implement features 1 (restriction on keys) and 2 (presentation as namespace). Feature 3 (adding context manager behaviour) is more easily done using contextlib.contextmanager as shown in §3.3 above.

2. Your design puts all features into one class. This technique is known as the "big ball of mud".

3. Since your code knows exactly when it is assigning a special attribute, it can call super().__setattr__ itself, thus avoiding the need for special handling in the __setattr__ method. See §3.2 above.

4. Your code looks OK to me, but writing __exit__ methods is complicated and so (if you can) it's better to use contextlib.contextmanager which takes care of this complexity for you.

5. It's sensible to define your own exception classes, but it's good practice to inherit directly from Exception only if there is no close match in the exception hierarchy. Here UnexpectedKeyError is a runtime problem caused by an invalid value of the correct type, so it makes sense to inherit from ValueError.

   The BadSpecialKey exception indicates a static programming error, not runtime exceptional behaviour, so I would use an assertion here.

6. It would be better to avoid having special keys altogther: you'll see in §3 above that I didn't need them. But if you must, then @classmethod makes slightly more sense: in theory you might have a subclass with a different value for _SPECIAL_KEYS.