The Armories allow more units to train simultaneously, in the training slots. That means, you have to fill the slots less often and can produce more Units over night. The higher level they are the more you can queue to build, also they give ranking points.
When you get to the orcs chapter each armory will become an orc breeding ground and produce the orcs right out of the armory/breeding ground and not the barracks queue system as well as keeping the unit training size system so they are worth every square they take up now. The Mercenary camp and the training grounds give you other types of units as well as provide ranking points so they are all good to have, I know you can only produce one type of unit to train at a time but by upgrading the armory you build more of them and queue other units to build after.. I agree it would be a good thing if the training grounds and mercenary camps had their own queue system but it is what it is and there are benefits to having them.