Well, here's the problem with chromosomes: Anywhere between 0.01% and 2% (depends who you ask) have chromosomes that are either not XX/XY, or are XX/XY, but occurs in an otherwise male/female person. Data is hard to come by, because it's difficult to assess the prevalence of these conditions when somewhere between a considerable portion and a majority will live their entire life without knowing about it/getting diagnosed. Yes, these constitute a fiftieth at most (if our data is correct), but they're far from a fringe case that doesn't deserve to influence our understanding.
There are two main reasons chromosomes don't matter. The first is that, just as how a woman being born without a uterus doesn't make her non-binary, one singular sex characteristic cannot be the defining factor. If we checked Usain Bolt's chromosomes and found out he has XX chromosomes, that doesn't suddenly make him a woman, nor does it mean he should be allowed to compete against women unless he undergoes HRT.
"...the main difference between XX and XY chromosomes is the type of gender in which they occur."
It's irrelevant. There's no function to separating people based on their chromosomes when you also remember the previous things I outlined. Yes, chromosomes are unchangeable, but using it to try and make sweeping claims about sex and biology just leaves you with the same issue as when you do the same with every other sex characteristic: you unfairly, unreasonably exclude people in a way that doesn't make sense.
Biological sex is a spectrum of sex characteristics that are associated with either end of the gradient, and chromosomes are one of the hundreds of factors involved in that spectrum (unimportant as they may be to daily life). Especially with the arguments from the debate I linked earlier, it doesn't seem to me to make sense to segregate sports based on chromosomes. The reason sports are divided between male and female is because of their difference in capability due to the nature of their sex characteristics, but trans females and trans men are completely able to adopt the relevant sex characteristics of their preferred gender and thus should be allowed to compete in my opinion.