I have no documentation and haven't even read Moldvay, but I would guess it's because normal men get the lowest hit die being used. In OD&D, hirelings and first level clerics and magic-users get 1d6, versus the fighter with the extra +1 at first level; so, fighters have more hit points than normal men. When you switch to the d4/d6/d8 scheme, thieves and magic-users have a d4; if normal men had more than 1d4, that means that a character could potentially start with 5 or 6 hit points, then drop to 4 because they started studying magic or thievery.
I read Moldvay extensively - and I guess Talysman statement is correct. It's also a simplier version than AD&D Dmg, where you have to find a table somewhere in the book and search which category is your normal man. So, I guess it's "Moldvay's sense of efficient rule".
It's not from Holmes. Holmes mentions "Normal Men" in the Saving Throw and Combat Tables (these may have been added by TSR since they differ from OD&D) but doesn't otherwise describe them in the main body of the rules. Bandits have 1 HD (d8), so they would be considered either 1st level fighters or 1 HD monsters. However, in the Sample Dungeon the pirates are described as "normal men (1 6-sided hit die)", so Holmes stuck with the d6 HD from OD&D.