Dysphoria and attraction

Gender dysphoria and sexual attraction are not mutually exclusive but they can be in direct competition. One can take predominance over the other and occupy much space in the brain. 

For example, some androphilic transsexuals go through a period where they attempt to be gay men only to realize that something is wrong. Their attraction moves in the right direction but their dysphoria is acting as a force of incongruence which blocks them. 

It can be the same for gynephilic transgender people who try to lead normal lives but the unresolved dysphoria acts as a deterrent. Their potential sexual psychology has been impacted likely before they even reached puberty when they may have detected something wasn't quite right. 

Attraction to being female (not just expressing as one) was confusing their circuitry. 

Blanchard in the 80's tried to sell this confusion as sexual deviance for both orientations through his two-type taxonomy model when in reality it was the gender dysphoria behaving as agent of chaos. 

Needless to say it can work the same for male to female transgender people.

Here is where non-dysphoric gender variant people have an advantage and can more readily compartmentalize their cross gender expression since there isn't a roadblock to sexual functioning.