The guards generally don't seem to care when one cuts down tea trees in the cultivated lands.
Do they care if you cut down a city tree in one of the set pieces?
Bonus question, feel free to ignore: Do the guards care if you mine the rocks next to the rock cutters?
Best Answer
I tested this out with a couple short games.
The guards will get argo'd with the first hit of the axe to the tree in a city set piece. So they aren't worth chopping down.
The rocks in the middle of the rock cutters area will also agro the guards but only on the final hit that destroys the pile. They don't care at all about the trees and rocks outside of the fixed set-pieces. And I think anything you plant is fair game.