I always leave them and then upgrade, unless they're max level dedicated to Ptolemaic dynasty specific gods (in which case I don't get any bonuses from them).