It is not he who gives abuse that affronts, but the view that we take of it as insulting; so that when one provokes you it is your own opinion which is provoking.
— Epictetus
The point
It is our own judgment of an insult, not the insult itself, that provokes us.
