Two funds at the quant-trading pioneer suffered their worst months ever in October.