in cases where X does not guarantee privilege
What I'm pointing it is that from the social justice perspective privilege is guaranteed - by their definition. So challenge the definition rather than equivocating.

I don't see what you are trying to say here.
See above.

just like we normally do not categorise people according to eye colour and talk about 'blue-eyed privilege' or 'green-eyed privilege'.
Importantly, it's perfectly possible - though by-the-by blue-eyed privilege is a thing, since it's typically associated with blonde whites.

There is, because wealth is relative
That's beside the point, since we're not currently preoccupied with privilege through time, its causes and trajectories, etc., excepting the following.

privilege or cultural differences?
Well, privilege per se wouldn't be a cause - it's a symptom. The cause would include culture*, but an important part of the story would be what contributed to both the privilege and the culture, which would be one group's domination over another.

*But I do agree that culture in the present day plays a significant role in the economic performance of underprivileged groups in societies, as per Max Weber. One of my biggest beefs with the social justice movement is that they attribute everything to culture except economic performance, and moreover manage to turn whites into perfect agents (taking agency as a concept for granted momentarily), stemming of the movement's libertarianism, but only when it comes to actions with respect to non-whites, who have no agency in that relationship. In all other cases, apparently, the roles are reversed, with marginalized ethnic groups having the agency. It's a true example of schizodoxia.