I vote for the fatimids, they're a good match for the Abbassids and represent how the Abbassids lost power when the big kaliphate split up more. Almoravids would just be the same as Al-Andalus - historically the leader of Al-Andalus invited the Almoravids to invade his domains and then take over the fighting against the Christians in the Iberian peninsula. This invasion took only a few years, so it's basically the same as Al-Andalus being transformed into the Almoravids faction if we just put a small script and an event explaining it, and unlocking berber units for Al-Andalus after a few years of economical penalties when the "invasion" happens. With fatimids we'd get a faction which isn't the same as another and that is transformed in that same way.