I have a Python + Flask app at a Nginx + gunicorn server with multiple workers.

One of the key app components is an “index”. Every worker and every thread shall have an unique index. At every operation the index is incremented by 1.

To prevent the same “index” to be used by more than one worker/thread, it is stored at Redis and the reading, incrementing and saving index to/from Redis is protected by a Lock().

Index is retrieved and incremented by non-threaded as well as multi-threaded functions.

The problem is that the Lock cannot be pickled by multi-threading.
I tried different arrangements of the two solutions proposed here, but none worked, seems because has to work with both multi-threaded and non-threaded.

The code is (and must be) spread over 3 different .py files.

# rotator.py

from multiprocessing import Lock
lock = Lock()

def increment_index():
    time_before = datetime.datetime.now()
    with lock:
        index = retrieve_index_from_redis()
        index = index + 1
        save_index_at_redis(index)


def rotator_of_index():
    new_index = increment_index()
    # do other stuff, log new index, etc
    return new_index

# salary_worker.py

from rotator import rotator_of_index

def do_work(n):
    index = rotator_of_index()
    # do other stuff
    return something
    
    

# do_work without multithreading:
def do_more_work():
    something = do_work(n)
    # do more stuff

do_with_threading.py

from salary_worker import do_work

from multiprocessing.dummy import Pool as ThreadPool

def do_work_with_threading():
    list_of_n = [1, 2, 3, 4, 5]
    pool = ThreadPool(15)
    list_of_results = pool.map(do_work, list_of_n)
    pool.close()
    pool.join()
    print(list_of_results)

I tried also a ray Pool to replace the multiprocessing Pool, without success.

Any alternative to make it work?