Any time period before the invention of clips or rounds would be pretty horrible. You'd have only one shot (if using a musket), or have to get into melee combat. Not my idea of fun. My personal choice would be anytime during the 20th century after the invention of penicillin, but before the baby boom explosion of the 1950s. My choice echoes Jkarinen, I'd go for late 1940s. As horrible as it may sound, Hitler and Stalin contributed to the reduction in world population by a few million, thus reducing the number of potential zombies.
As for the weapons, I prefer accuracy over spray-and-pray. I'd pick the Enfield No.4 MkI. I'm proficient in cold-iron sights, so no scope is needed nor desired for me (gets in the way at close range). For a sidearm, nothing beats the old Browning 1911.