Himeji? Has a fantastic castle, close to Kyoto, and the castle is visible from the shinkansen station. Can do a quick trip there from Kyoto.

Or perhaps Miyajima island, near Hiroshima, a great place. Also has one of the most photographed spots in Japan (the torii in the sea), and a temple which looks like it's floating when the tide is in. You can also stay there at a ryokan.

Nara is a day trip from Kyoto.

Check the shinkansen fares here to see if it's worth getting a JR pass (from memory, either 7, 14 or 21 days unlimited JR travel, inc some buses).

I would recommend staying in a traditional Japanese inn (ryokan) for at least a night, it's a real experience. Normally breakfast and impressive dinner (often many many exquisite seasonal dishes) included.

Search for one here. Be warned that they are often quite expensive, though there are reasonably priced places.

Carry around extra cash there, as quite a few places (including our ryokan, at $150/night x 2 people x 4 nights!) didn't accept cards. Thankfully Japan is generally an extremely safe place.

The JNTO has lots of info

Also I'd recommend arranging for a volunteer goodwill guide at least a few days, who will take you around whichever sites you want, translate, and can answer your million and one questions (and ask a million of their own!). Especially the first day you are there, so they can show you basic things like how to buy a train ticket. We arranged for 3, and all were great. Seems like they are mostly uni students, housewives or retirees. Contact the appropriate group and they will try and find someone to take you around. Generally that means that the guide will speak at least basic English (in your case), and will have some idea about the local sites. All you need to do is feed them and pay for entry fees (sometimes they get in free to museums and such). It's nice if you give them a small gift (gave them a small koala soft toy, which they were thrilled about).

I really want to go back again, I envy you.