I am trying to get a queue-based application to work.
Basically, the multiprocessing queue is limited to a 32-bit number(32768)
Ubuntu Linux 18.04 does not have this limitation. I am guessing this is a sysctl setting for the OS.
ctx.BoundedSemaphore(maxsize). Does anyone know where the limitation is coming from? I have tried a number of settings in sysctl and rebooted the machie. (etc etc). The limit remains.
colin_bitterfield$ python3
Python 3.7.4 (default, Sep 7 2019, 18:27:02)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> myQ = Queue(64000)
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 102, in Queue
return Queue(maxsize, ctx=self.get_context())
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/queues.py", line 48, in __init__
self._sem = ctx.BoundedSemaphore(maxsize)
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 87, in BoundedSemaphore
return BoundedSemaphore(value, ctx=self.get_context())
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/synchronize.py", line 145, in __init__
SemLock.__init__(self, SEMAPHORE, value, value, ctx=ctx)
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/synchronize.py", line 59, in __init__
unlink_now)
OSError: [Errno 22] Invalid argument
>>> myQ = Queue(32767)On Ubuntu:
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-1051-aws x86_64)python3
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> myQ = Queue(64000)