In the rare case of Mike Tyson... yes you do. Tyson spent his entire career ducking a fight with Holyfield. When they finally fought - Tyson lost.
Holyfield was the ONLY top fight Tyson ever went up against. He had the luck/disgrace of fighting during an era where tomato cans like Trevor Berbik and Riddick Bowe were considered contenders.
Tyson neer got to test his metal against a George Foreman or Ali. It will never be known if he was a great fighter in his prime because he NEVER faced great competiton and won.
The greats who got to watch Tyson fight - Holmes, Ali, and the rest of that era have universally said they are not impressed with Tyson and do not think he could have entered the ring as worthy competition.
My personal feeling from having watched Tyson fight - he had one of boxing hardest and fastest punches... but he never had the juice beyond two rounds. He lost his edge way too quickly... and he was never a sophisticated boxer. He couldn't get past Lewis's defense... he couldn't get past Holyfield's... and he never made a good showing against either of them.