I don't think most transgender people would dispute that sex exists—i.e. humans are generally grouped into a bimodal distribution of mainly being born with female traits (born with a uterus, XX chromosomes, etc.) or mainly being born with male traits (born with a penis, XY chromosomes). I say "bimodal" instead of "binary," since there are exceptions, but it's a pretty strong correlation. People are usually born with mostly male traits or mostly female traits, hence the grouping of the sexes. Otherwise there would be no reason to have different terms for "trans women" and "cis women."
And then there's gender. When trans women are referred to as women, this is using the term to refer to gender identity (which is arguably more nebulously defined). I'd wager that for a long time, a lot of people used the term "woman" to refer to sex and didn't know about gender identity at all. Some dictionaries define the word "woman" as referring to sex, but then some don't. Trans people aren't a monolith, of course, but I do know that many trans men with female traits would say that their sex is biologically female but object to being called a woman. In other words, they're fine using the term "female" to refer to sex but not the term "woman."
This might have something to do with the etymology of the word, but it might also have to do with how often it's used. If we has another word like "blargh" that was widely used to refer to humans born with female traits, would most trans men object to being described that way? Since both cis women and trans men are blarghs, would using the term "blargh" be different from saying they were assigned female at birth? Part of me suspects that many people would object to the use of the term "blargh" for the same reason they would object to using the term "woman" to refer to biological sex.
I guess a similar thing could be said for weight, where an overweight person might be fine being described as overweight but object to being described as "fat." On the other hand, I can't think of any woman with brown hair who would object to the term "brunette." It might be the case that gendered terms sometimes being intended as an insult (i.e. "you throw like a girl") makes some people not want to be referred to with that term, similarly to "fat" being used as an insult. So maybe people's opinion of the term "blargh" would depend on whether the word "blargh" had cultural connotations associated with it.
Maybe a trans person would be able to give their take on this, though it probably depends on the person. I did make some assumptions here about what trans people generally prefer to be called, and I don't know how accurate those assumptions are.
