How others perceive us is of little importance compared to how we perceive ourselves. If self esteem and self respect depended on public opinion we would be in trouble.
Most of us wear some sort of mask to get by in the world except that with age it should become thinner as we are increasingly rooted in authenticity. We begin to care much more about respecting our autonomy and preserving our well being than following societal whims.
How others view this detachment is entirely their problem and not ours.
Finding this sense of liberation comes at the expense of opinions that are irrelevant; from people who don't know us very well and certainly don't live in our skin. We disregard them while remaining polite and cordial.
We simply have stopped dancing to the same music that they do.