[klibc] malloc: Fail if block size is out of range for sbrk

sbrk() takes a parameter of type intptr_t.  We allow allocating up to
PTRDIFF_MAX (equal to INPTPTR_MAX), and then add a header to that, so
the result fsize can be > INTPTR_MAX.  The conversion of fsize to
intptr_t would then result in undefined behaviour (but probably
*lowering* the top of heap).  Fail cleanly before that happens.

This is currently a theoretical problem since we actually use mmap()
instead of sbrk() on all architectures.

Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
diff --git a/usr/klibc/malloc.c b/usr/klibc/malloc.c
index abda84c..09a596f 100644
--- a/usr/klibc/malloc.c
+++ b/usr/klibc/malloc.c
@@ -171,6 +171,10 @@
 	fsize = (size + MALLOC_CHUNK_MASK) & ~MALLOC_CHUNK_MASK;
 
 #if _KLIBC_MALLOC_USES_SBRK
+	if (fsize > INTPTR_MAX) {
+		errno = ENOMEM;
+		return NULL;
+	}
 	fp = (struct free_arena_header *)sbrk(fsize);
 #else
 	fp = (struct free_arena_header *)