I play Rugby for my college. We don't have a football team, so on our campus we are the biggest, fastest, and strongest guys. Most of the players are ex-football players, that had never been introduced to Rugby before, and most of us like it just as much as we did football (as I played football all through HS too). It is a different sport though... rugby players should not be as big (and thereby as strong) as football players, but they should be just as fast in sprint speed (though we measure by the 50 meter), and should be MUCH faster in long distance speed and be able to complete more body weight excercise reps.
Forwards (the bigger guys on the field) are usually linebacker size, there is no need for OLine/DLine sized guys on a rugby field.
I'd say Professional Football has better athletes, but not by the margin I bet most people in America would believe, and this is only due to there being more money in football.
It takes a special person to play Rugby though... Football is a tough sport, but no where near the toughness of Rugby. Imagine running at a college linebacker with no pads expecting to make the big hit, welcome to Rugby. I'd say its closer to MMA and Boxing than football in that regard.