I think it becomes racism when the focus of people's criticism is him as a person and particularly in those areas that he 1.) has no control and 2.) when people are applying a scrutiny that was never applied to any previous president (birthers, schoolers or whatever they are calling themselves, etc).