I don't think people should have to censor themselves in fear of being ostracized, but commenting on someone's appearance- when they can't help it- can't be justified. For example, making fun of someone for being too thin is outrageous. People have been saying this for years, yet skinny-shaming is so much more common and accepted than fat-shaming is, and will probably ever be. I've come across several youtube videos, songs, and "body-positivity" campaigns where people directly come out and say "size zero is not a size" or "real women have curves." These statements have been normalized because being 'skinny' is seen as the standard of beauty, so making fun of someone for being 'too skinny' is seen as a simple tease, a silly joke, and maybe even a compliment.
However, being called out for being too bony or too thin or too flimsy is not flattering, it's just as insulting as being called too chunky or too big or too unhealthy. I never really paid attention to the issue myself until my friend told me how tired she was of people telling her to gain weight or "eat a burger". She said it was frustrating because she wanted to gain weight, but simply could not put the weight on no matter how much she tried; regardless, she learned to love and embrace her shape. Being thin is not a personal choice; it is the body she was given and the body that she is proud to have, despite the incessant and unnecessary comments.
Another reason people may be underweight is that of medications, stress, depression, and other mental or physical illnesses. These are precisely the same things that people that are overweight may be dealing with. Essentially, we do not know the details of everyone's personal lives, so it is best to refrain from pointing out flaws we find in their physical features.
Basically, skinny-shaming is wrong for the same reasons fat-shaming is wrong. Everyone is dealing with their own personal battles, and adding fuel to the fire is the crappiest thing you could do to another human being.