Growth rate penalty is capped at 25%, which is reached at a population of 61500 (with the Imperial Palace). Updated first post with the info.