I keep reading that it's white people (usually males) afraid of losing their majority.

Remember, when non-whites have a lot of kids it's usually not looked on as favourably.