The furthest I ever go down is to Nakornsrithamarat but really I don't know anyone here that would ever want to go down that far. I wouldn't say categorically that it is not safe but they clearly have issues. I doubt a a white face would make you a target since the troubles have nothing to do external matters, but there is a lot of underlying latent jealousy towards farangs so personally I think there are better places to go for a holiday. Generally speaking most Thais find their countrymen in the south short tempered, aggressive and on occasion violent.
Doesn't really answer your question but it says something that I live in what we call "Pak Thai" and I don't know anyone who would go to those areas by choice if they don't have family there, Thai or Farang.