That depends on your skin colour and nation of origin. I know how racist that sounds, but I've heard far to much of it to dismiss it. The Japanese have huge prejudcie against Koreans and Chinese, and really don't treat them that well (when you hear about Japanese not wanting to rent to foreingers, they could easily be talking about eastern continental asian guest workers). I've heard tell that Japanese papers try and blame any and all crime on Chinese and Koreans. They love all things USA and French, but they'd still probably find them rude.
For another perspective on life for a guest worker in Japan, check out
Gaijin smash. It's the blog of an African-american english teacher in Japan.