There are many factors involved in overtraining.
In order for protein synthesis to occur optimally sufficient ATP and specific enzymes must be present in the muscle (glycogen provides some of the ATP which provides the energy necessary for protein synthesis). Each set you do depletes intramuscular ATP, glycogen and enzyme levels - this depletion inhibits growth. But there is also a prevalent theory of muscle growth that states that this ATP depletion is, itself, a necessary stimulus for growth. Also, each set produces muscular damage. The damaged fiber components are removed from the cell and eliminated (via the nitrogen cycle). In fact, after a workout, protein breakdown is markedly accelerated. This protein degradation is offset by the fact that the cellular damage incurred also causes the body to increase it's protein synthesis rates at the same time. So, the growth process is a balance of protein degradation and synthesis - catabolism and anabolism. Now, it is a fact that the body is capable of increasing protein synthesis rates to only a certain point (limited by substrate and enzyme availability, nuclei number, etc). So, by all of this, there definitely appears to be a point where enough ATP has been depleted to stimulate growth (if you subscribe to that theory) and the body's ability to synthesize protein has been stimulated to it's maximum. Any work beyond this point will contribute further to protein degradation but no further synthesis will be stimulated - the body simply is incapable of it.
Overtraining in a nutshell is - When so much work is performed, that more catabolism takes place than anabolism.