bcachefs: Increase size of cuckoo hash table on too many rehashes

Also, improve the calculation of the new table size, so that it can
shrink when needed.

Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>