I don’t think people deny that human population size is relevant. The problem more rests with the moral dilemmas surrounding proposed solutions addressing human population control. Mostly, people don’t even want to touch the subject with a barge pole (unless you are China) because, as is the case with with many moral discussions, there is no win/win scenario. Almost every proposed solution will invariably end up violating some human right. This also then raises the ethical considerations of inaction around human population control and the indirect violations of human rights that occurs from that. Interesting discussions though!
The trolley problem is a problem for a reason.