Average is also a word used outside the context of mathematics. Would you make this argument if someone said “I’m average looking”? No, no, you can’t be average looking, because there is no such thing as an average of appearances! Come on.

average to what?

to average is to compare. to compare you need a set. to have a set you need data.

an average is an equilibrium of a set of data.