First off, it isn't racial, it's cultural.
Second, it's unclear is the origins of the term are self identified or imposed from the outside. It first appeared in print in The New Yorker, but the author states it was the term the locals used to describe themselves.
Finally, while it has some derogatory aspects [poor, illiterate] it's also embued with positive attributes [self sufficient, free, hardy].
SNL Standards and Practices - "you can't say 'midget,' It's just like saying the N-word.'
John Mulaney - "No it isn't, because you JUST SAID midget."