[klibc] malloc: Fail if block size is out of range for sbrk
sbrk() takes a parameter of type intptr_t. We allow allocating up to
PTRDIFF_MAX (equal to INPTPTR_MAX), and then add a header to that, so
the result fsize can be > INTPTR_MAX. The conversion of fsize to
intptr_t would then result in undefined behaviour (but probably
*lowering* the top of heap). Fail cleanly before that happens.
This is currently a theoretical problem since we actually use mmap()
instead of sbrk() on all architectures.
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
diff --git a/usr/klibc/malloc.c b/usr/klibc/malloc.c
index abda84c..09a596f 100644
--- a/usr/klibc/malloc.c
+++ b/usr/klibc/malloc.c
@@ -171,6 +171,10 @@
fsize = (size + MALLOC_CHUNK_MASK) & ~MALLOC_CHUNK_MASK;
#if _KLIBC_MALLOC_USES_SBRK
+ if (fsize > INTPTR_MAX) {
+ errno = ENOMEM;
+ return NULL;
+ }
fp = (struct free_arena_header *)sbrk(fsize);
#else
fp = (struct free_arena_header *)