The fact that extremely rare mutations occur in the form of chromosomal deletions and translocations does not discredit the entire science of genetic mapping. There is a huge amount of research devoted to mapping the human genome. Chromosomal structure actually is holy--or let's say stable--enough to base an entire branch of research off of mapping which genes are located where on them.

These are constant enough that we are able to categorize many different birth defects based on chromosomal abnormalities. Cri du chat syndrome is because of a deletion on chromosome 5, down syndrome a result of an extra chromosome 21, Tay-Sachs a mutation in chromosome 15. Many more illnesses can be linked to mutations in genes in very specific locations, but you don't see anyone wailing on about how chromosomes are unimportant here because in a few incredibly rare cases, someone's down's syndrome is actually a result of a translocated piece of chromosome 21 being put on, say, chromosome 20, rather than existing independently as in the standard presentation of trisomy 21.

Calling chromosomes an "educated guess" of biological sex is an extreme distortion of how powerful a predictor they are. In over 99.9% of cases, they accurately predict sex.

Bringing up animals is some actual fifth grade logic here. We are talking about distinct populations here. Humans are not the same as lizards or clownfish. Intersex people are not the same as transgender people (though there is some overlap). Just because clownfish can change their sex, doesn't mean humans can. Just because some intersex people are essentially XY females or XX males, doesn't mean that transgender people are.

but in the end, sex is a physiological trait, not a genetic feature.

This doesn't make even sense. Physiology is determined by genetics. That's why you're grasping at SRY translocations and deletions.