The plural of ox is not oxes. It's oxen. So wouldn't it make sense to say foxen? What about moosen? Because it's not goosen. It's geese. Would it be meese? Do you Canadians hunt deer or Caribou? And fishes is a verb. Sheepen?
Their just remains of some past lexical morphologies in present-day english. You can usually derive the word endings you'reself from older forms in, e.g., old english and it's cognates.