A few Christmas's ago, my uncle sent me a package in the mail. On the top of the box, he wrote: "do not open until Xmas." I didn't think much of it until I caught wind of a few people debating about the correct usage of the word Christmas. Those arguing for the use of Xmas like to claim that they're not trying to "replace Christ," but just think its easier to write. I don't think I have met anyone who intentionally wanted to take the Christ out of Christmas, but none the less, it is important to consider how your word choice can change the meaning of a word completely.
The word Christmas comes from the Latin word Cristemasse, literally meaning "Christ's Mass." When people choose to take the word "Christ" out of the word, they also remove the meaning of the word entirely. In this aspect, what is the point of calling it Xmas or Christmas at all, if you have no connection to the holiday religiously or historically? This is not to say you should never celebrate Christmas or refer to December 25th as Christmas if you don't identify yourself as a Christian, but you should respect the meaning of the holiday for those who do observe it.
For me, Christmas is a time to celebrate the birth of the Christ, and God's love for his people, that he sent his one and only son to us. When I hear or see someone referring to it as Xmas, I see that as a sign of disrespect, not towards me, but towards an entire group of people who see this time of year as a chance to celebrate their beliefs. You wouldn't think to abbreviate Chanakuh or Ramadan, so why should Christmas be any different? As we're becoming a more and more tolerant world of increasingly diversified beliefs, our words hold more weight both positively and negatively. Refraining from the use of Xmas in your letters or speech, especially when addressing someone who celebrates the holiday, is just another way to show respect for those around you and their values and beliefs, even when they might be different from your own.
To return to an earlier point, all this is not to say that you should give up your celebration of Christmas and the holidays if you're not Christian. Christmas should be everyone's right to celebrate no matter your standings on religion or the like; but don't take the meaning away from those who believe in its origin, by abbreviating and removing the most important part of the word.