Hi,
Thought I'd stop by to give you some extra information about what caused this problem.
PaX's ASLR is implemented differently from that of the vanilla kernel. The vanilla kernel implements what is referred to as a "detached" heap. Normally the heap begins at the end of the binary image as mapped in memory. The vanilla kernel separated this, leaving a random-sized gap in the middle. The gap is actually security-relevant and one reason why PaX's implementation is different: with allocations above a certain size, glibc will switch to using mmap to satisfy allocation requests instead of extending the brk()-managed heap. Under memory pressure and with such an allocation, glibc could thus be forced under a vanilla kernel to allocate in the gap adjacent to the beginning of the brk()-managed heap. If an overflow were to occur in that allocation, it would then overflow into an area of the heap that would otherwise be inaccessible were it not for vanilla's ASLR implementation.
PaX has never introduced a gap below the heap and instead implemented its heap randomization by simply extending the existing mapping below the heap by a random amount. The randomization (on x86) was applied to bits 4-16 of the address, while vanilla (introducing multiple pages of gaps and needing page-alignment) applied it to bits 12-25.
Since upstream established that entropy up to bit 25th could be withstood by userland, the PaX team decided to increase the amount of randomization provided for the heap by PaX's ASLR, making it apply to bits 4-25. That's 21 bits of entropy compared to the 13 of a vanilla kernel. The way PaX increased its amount was by essentially creating a gap of its own, but then filling it with a NOACCESS mapping, as you can see from some recent maps output:
Code:
0804f000-08050000 rw-p 00007000 08:01 1945892 /bin/cat
08050000-08928000 ---p 00000000 00:00 0
08928000-0894a000 rw-p 00000000 00:00 0 [heap]
Back to the problem at hand here, we found that the courier imap daemon was implementing its own resource limits -- RLIMIT_AS in this case, which is a limit on the amount of address space used. The mapping that filled the gap between the binary image and the randomized base of the heap ended up being counted against this address space limit. After this was discovered, the fix was somewhat simple: just treat the heap "gap" mapping as unusable space and don't count it towards the address space limit.
Sorry for the temporarily problem, but know that you're benefiting now from even greater heap randomization as a result of this work.
Thanks!
-Brad