Before the unpacker at least, changes were only recognised if run using mod switch (there's a thread for it somewhere).