The fact being that you have had several deer with this characteristic tells me it is genetics. I've heard for years that an injury causes abnormal antler growth on the side that has been injured.
Here is a little experiment I'm watching this season on my cameras. This little guy showed up last December with a nasty wound from a shotgun slug. He lost his lower leg in late January.
He survived the winter and is sporting antlers this summer. Both sides look very similar thus far. We will watch and see if any abnormal growth appears over this summer.
This photo of him bedded shows his antlers pretty well.