If there is a preferences.txt file on the Mac, simply try to edit it by hand and set the resolution. That approach works on Windows with various “odd” resolutions; and it might just work on the Mac, too.