The history of Azhagar Kovil dates back to the Sangam period. In ancient times, the region was ruled by the Pandyas, who were great patrons of art and architecture. The temple received significant contributions from various dynasties, including the Cholas and the Vijayanagara kings. The temple's architecture reflects the influences of these different periods. Legend has it that Lord Vishnu, as Azhagar, arrived late for his sister Meenakshi's wedding to Lord Shiva in Madurai. Disappointed, he stayed at the banks of the Vaigai River before returning to his abode in Alagar Hills. This event is celebrated annually during the Chithirai Festival, a grand spectacle that commemorates the divine journey.