If you're a man who's wondering what shirt size he should wear, you need to look at your belly. If you're carrying a fair amount of fat there, you need to be sure to never, ever, wear a shirt that isn't at least one size too big for your neck and chest measurements. Ignore the size charts. Get a BIG shirt.
The reason for that is that a big shirt won't cling around your belly. Depending how how much belly fat you have, a big shirt can pretty much hide your fat. And that's a good thing. Nobody really wants to see your belly pushing so hard against your tee-shirt that they can see your belly button. No. No.
If, however, you've lost weight, and are no longer carrying that large belly, it's time to buy a smaller shirt. And I don't mean you have to be "lean and ripped to the bone", just lacking that big old belly. Try on a smaller size, it will look better on you.
It will feel kinda strange at first, as the extra-large sizes are meant to "float" around your body. A shirt that's the correct size, like a suit that's the correct size, won't "float". It will follow the contours of your body, your shoulders, your chest, your arms. And if you've lost weight and have been working out, those areas will look great.
If you're not sure if you're ready yet, take this test: Try on a smaller tee-shirt. If it looks like you're trying to smuggle a watermelon, or you're the first man ever to become pregnant, then stay with the larger shirt. When your belly is gone, the shirt will fit nicely around your chest, arms and shoulders, and fall against your hips. A tee-shirt should not be tight around your belly. A tee-shirt should fit like a nice suit, following the contours of you body, not looking as if you might burst out of it at any minute.
I hope this helps. You've worked hard to get where you are, and it's time to wear clothes that fit. Most people will never be able to this, but you can. And you'll look great.