Hiya, it should work fine unless it modifies files that come with VanillaMod. Even in those cases, some combinations likely can be made.

The approach you could try is to backup your VanillaMod folder first (well, you still have the downloaded archive, so maybe you won't need to ) and then copy all contents of the other mod into the VanillaMod folder, making sure that if the other mod's folder is called say Ornamentum, you don't copy that folder into VanillaMod but only its contents and subfolders so the overall structure remains the same as in VanillaMod. If you are not prompted to overwrite anything, things should be fine. If yaou are prompted for overwriting, press no for now and note down the name of the file(s) and post them here so I can have a look and help you get it to work.