Invasio Barbarorum (IBFD) installs everything to its own subfolder (.../Rome TotalWar/ibfd/) and deleting that would delete IBFD completely and safely.
The problem you encountered is because your other mods changed files in RTW/data folder that IBFD needs to run properly. If you encountered problems with EB or XGM (are they even compatible together?) then you either installed IBFD incorrectly (by manually installing the files to RTW/data, for example) else it has nothing to do with IBFD as IBFD does not affect any other mods.