There are some traits that change squalor and no buildings, but overall it is determined by the % population of the maximum the current walls can hold, up to 80%.