[ Multiprocessing Queue Negative Max Size ]
There are some examples of code that specify a negative number for the
maxsize of a
multiprocessing.Queue like so
import multiprocessing queue = multiprocessing.Queue(-1)
I could not find anything in the documentation that specifies what a negative value means so I looked at the
multiprocessing.Queue source code.
class Queue(object): def __init__(self, maxsize=0): if maxsize <= 0: maxsize = _multiprocessing.SemLock.SEM_VALUE_MAX self._maxsize = maxsize ...
I can't seem to find the value for
_multiprocessing.SemLock.SEM_VALUE_MAX. Does a negative number create the biggest Queue possible supported by the OS? Is there any documentation that clarifies this magic value?
The docs for
multiprocessing.Queue apparently don't say this directly, but it's strongly implied in two places.
Queue docs say:
Queueimplements all the methods of
And earlier, in Exchanging objects between processes:
And, if you follow the link:
Constructor for a FIFO queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.
So, yes, -1 is documented to mean infinite… although maybe not in the most direct way possible.
You could file a doc bug on this. Maybe all of the classes should explicitly say "The interface is exactly like
<relevant other class, with link> except as specified below", instead of leaving you to find that information elsewhere in the docs?
Since you actually asked two questions, not one, here's where
SEM_VALUE_MAX is defined:
PyInit__multiprocessing, the top-level code for the
_multiprocessing module, it's added dynamically to the
__dict__ of the
The value it gets set to is platform-specific, but it's generally going to be your platform's
INT_MAX. The former is defined in POSIX 2004 as part of
limits.h, and just specifies:
The maximum value a semaphore may have.
The source comments may be a bit confusing.
I don't know about NetBSD, but IIRC, old FreeBSD did the same: the value is provided as a signed integer (actually, it's just a
#define preprocessor macro, but that makes it an
int-typed constant), but the APIs that use it take unsigned values. In C, you can just pass
-1 to a function that takes an
unsigned int and it's the same as passing
UINT_MAX (4294967295). Python doesn't have those casting rules, so the module just converts a -1 to
INT_MAX (2147483647), which is safer, and you're not going to have more than 2.1 billion semaphores. I believe current FreeBSD and its descendants, like OS X, just give you
USHORT_MAX (32767), which makes this unnecessary, but there may be other OS's that do the same.
I can't seem to find the value for _multiprocessing.SemLock.SEM_VALUE_MAX
python -c "import _multiprocessing; import platform; print platform.platform(), _multiprocessing.SemLock.SEM_VALUE_MAX"
Specific values found so far (please edit and update accordingly):
# OSX 10.9.5 Darwin-13.4.0-x86_64-i386-64bit 32767 # Linux Mint Qiana Linux-3.13.0-24-generic-x86_64-with-LinuxMint-17-qiana 2147483647 # Fedora Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-fedora-21-Twenty_One 2147483647 # Ubuntu Trusty Linux-3.13.0-45-generic-x86_64-with-Ubuntu-14.04-trusty 2147483647 # Debian 8 Linux-188.8.131.52-i686-with-debian-8.0 2147483647
This may be a good case to suggest an improvement to the documentation. If you would be so kind.
I could not find anything in the documentation that specifies what a negative value means
It is actually platform dependent, use the above command the get the actual value.
While there does not seem to be documentation on
maxsize specific to
multiprocessing, the standard library's Queue documentation has this to offer:
If maxsize is less than or equal to zero, the queue size is infinite.
and multiprocessing's guide states that
The Queue class is a near clone of Queue.Queue.
Hence I suppose it is fair to assume that your interpretation is correct, i.e. create the biggest Queue possible supported by the OS. Actually, we don't have to revert to assumptions, it's a fact:
The gory details
If you want the gory details of how
_multiprocessing.SemLock.SEM_VALUE_MAX is determined at compile time, read on.
Is there any documentation that clarifies this magic value?
No -- If you really want to know, you have to follow the code... For CPython there are these #include files multiprocessing.h and the platform C library's limits.h, the latter eventually being included by Python.h. In other words,
SEM_VALUE_MAX is set depending on the host platform.
As per Open Group's documentation on limits.h It is is defined as
The maximum value a semaphore may have. Minimum Acceptable Value: _POSIX_SEM_VALUE_MAX
_POSIX_SEM_VALUE_MAX is the minimum which in turn is defined as:
The maximum value a semaphore may have. Value: 32 767