Nope, if they are on your territory, they are the aggressor and you have total rights to defend your territory. You might still get the warning about getting excommed when doing so, but the pope never excomms and you never take a relations hit if you do so.