Certainly not.
You do need a sufficient number for a successful conquest.
But you also need enough of them to defend your own territories.