You cannot make a shadow faction your vassal. They'll also not sue for peace when beaten down.

Who breaks alliance with who has some random factor but it mainly based on relations. If your relations with the faction is better than the other's for at least 1 turn, chances are it will break alliance with the person you attacked.