you'd expect cult worshiping religious fanatics to be part of a primitive society in a remote part of the world, so I can't really understand how that makes him racist.
Uh...

He does have several stories about inbreeding and degeneracy among noble families of white folks
Wherein the dread-inducing parts are, for example:

Dutch people are "white trash", and one of their noble families literally become ape-creatures over 200 years.

Anglo man discovers his grandfather actually mated with a Negroid ape-creature, meaning that his own nature is evil and so he struggles between succumbing to his inherent evil and committing suicide.

Anglo communities of New England are corrupted by mariners who intermix too much with evil, subhuman islanders and introduce Cthulhu-worship and whatnot.

New York City is full of demon-worshipping immigrants from Portugal, Italy, and various other sub-Anglo societies. They ruin all the old 19th-century buildings with their foul corruption. They speak not in English but in strange barbarian tongues.

Look back and you will see that most of his early stories are focused on the horrors of racial-mixing. Often, non-Anglos serve as gateway-races that lead to the Anglo protagonist's loss of innocence. The non-Anglos are "lower" and therefore certainly consorting with unspeakable eldritch intelligences and working to lure the protagonists into quasi-sin/damnation.