The truth is, no one really knows for certain: there are no descriptions, nor drawings...
By its name "pollice verso", is clear the thumb is "turned", in which direction, nobody knows :P