White people can indeed have it tough, but that tough life is never because of the colour of their skin. Non-white people are held back every day because of their skin colour and that can lead to hardship.
Acknowledging white privilege does not deny that whites are not all comfortable in life; it just acknowledges that their colour wasn’t to blame.