As I understand it you earn points towards guilds by achieving goals and building certain buildings.

If this is true. . .

. . . my question is, if I demolish a couple of barracks in castles I don't want will it hurt my chances of getting a related guild?