i currently have that armory quest. is it worth to do it or should i just decline since i hate fighting
If its a declinable quest and you don't want to do it then just decline, some of the declinable quests are on a loop and so pop up again and some are specials which come up once, the armory quest would be a special.
I think its worth having some troops for tournaments even if you've decided that you'd rather negotiate on the world map.
For what its worth, I have 4 cities of differing sizes and they all negotiate on the world map as I have neither the time not inclination to fight, too many troops, too many choices and I'm not a fighter either. However, I do make troops for tournaments and thats because I can get further fighting than I can be negotiating in all but the smallest city - the one thats the same size as yours - at that level its worth negotiating the first province, and the second if you can. I'm in a solo FS and by negotiating the first province to round 6 and the second to round 3 or 4 (can't remember which it was!) I got the first chest opened. Its cheaper to do those provinces as they just require gold, supplies and a few of your T1 goods (like 7 or 8 for the first province) rather than the world map which requires larger quantities of goods.
Once that city becomes larger I might do a bit of fighting, my main and 2nd cities are 95% fighting with just a few encounters negotiated, one of those cities has reached the end of the tech tree and the other is in orcs.