You are not forced to fight. Armories do not produce troops. These Orcs are treated as another “resource”. Brace yourself, the next chapter (Woodelves) will bring yet another new resource, though I’m sure you won’t associate that one with fight. And it doesn’t stop there.
As LazyTony above mentioned, there is also craftable building in your Magic Academy called “Orcs nest“. It produces these same Orcs automatically and for free, you just need to collect them. ;-) However, it rarely shows up, so you’ll need luck to be able to craft it any time soon. And to be able to craft enough of them to cover your needs.
It’s safer to build and fully upgrade let’s say 2 armories right now. It will cost you quite some population and culture. If you really hate armories for their association with fight, you can keep looking for the Orcs Nests in your MA, craft them and replace your armories gradually.