How to avoid carrying around a lock with my variable when multiple threads instantiation in Python?

Question

I created an application using multiple threads defined in multiple class files which read and write into a "shared variable" (a dictionary).

To ensure thread-safety, I am using a Lock, and I pass it as argument to my Thread class next to my variable.

What bothers me is that two objects which are intrinsically linked have to be passed to my class, while ideally only the shared variable should be. This would reduce the code and bring greater certainty while avoiding confusion in the parameters.

Here is how my program looks:

inc_thread.py

from threading import Thread


class IncThread(Thread):

    def __init__(self, shared_dict, lock):
        Thread.__init__(self, daemon=True)
        self._shared_dict = shared_dict
        self._lock = lock

    def run(self):
        while 1:
            with self._lock:
                self._shared_dict['some_value'] += 1

dec_thread.py

from threading import Thread


class DecThread(Thread):

    def __init__(self, shared_dict, lock):
        Thread.__init__(self, daemon=True)
        self._shared_dict = shared_dict
        self._lock = lock

    def run(self):
        while 1:
            with self._lock:
                self._shared_dict['some_value'] -= 1

main.py

import time

from threading import RLock
from dec_thread import DecThread
from inc_thread import IncThread


if __name__ == '__main__':
    lock = RLock()
    shared_dict = {'some_value': 0}

    t1 = DecThread(shared_dict, lock)
    t2 = IncThread(shared_dict, lock)

    t1.start()
    t2.start()

    # Ensure equity regarding wich thread started first
    with lock:
        shared_dict['some_value'] = 0

    while 1:
        print(shared_dict['some_value'])
        time.sleep(1)

Is it the correct way to do it?

What I also thought:

• Subclass dict and override get / set methods with an internal lock, but it becomes complicated when I want to set other datastructures (nested dict, list, set, etc.) as values.

• Using Queue but it does not seem to fit my needs, as it is by default blocking for put / get while my design is more of "fire and forget".

Answer 1

Your concern seems legit. Fusing data that needs thread-safe operations and means of those operations would be nice.

However, this:

Subclass dict and override get / set methods with an internal lock

will not work, because += - which you use - will not be atomic.

Things get worse with nested data structures: we actually have several objects and some of them carry references to others. If there are also some more references to those objects - how do we check that? Example:

a = [1,2,3]
b = {'a' : a} # or some crafty wrapper on top of that

If we make sure that every b['a'] is atomic, a can still be changed directly.

Making a wrapper that accepts mutation function and executes it in a locked context would not really work if you need in-place operations, not replacements of the whole object.

What you can do is take a closer look at context managers and try to create an object that is both dictionary and context manager.

Or simply do something like:

class FlexibleDict(dict):
    pass

shared_dict = FlexibleDict(shared_dict)
shared_dict.lock = lock

and pass that around.