I dived in a bit, and I will concede that sexual use is not the main reason, since they evolved from primates and previous species in order to produce milk. But all other species don’t have them constantly enlarged. Studies show that there are many factors at play, such as providing more area of touch for the human infant and perhaps a more constant source of food since human infants take longer to mature. But they did develop the use for sexual signalling too when we became upright, if for nothing else than for showing sexual maturity. The human brain has probably evolved along with those changes to select which characteristics evolved more pronounced.

So they have multiple uses. The very fact that they are a subconscious trigger for the male has to mean something.