As per Hindu legend, once there was an argument between Saraswathi, the consort of Brahma and Lakshmi on superiority. They went to Indra, the king of celestial deities. Indra judged Lakshmi as superior and not satisfied with his argument, Saraswathi went to her husband, Brahma. He also chose Lakshmi to be the superior one. Saraswathi was unhappy with the decision and decided to stay away from Brahma.
Brahma did a severe penance praying to Vishnu and did an Aswametha Yagna. Saraswathi was still angry that the yagna, which usually is done along with consorts was done alone by Brahma. She tried to disrupt the penance in various ways, including as the river Vegavathi/Vega, but Vishnu interfered in all her attempts. After all the demons were killed by Vishnu, who were sent by Saraswati to destroy the yagna(penance) done by Brahma, she finally sent a ferocious Sarpam(snake).
Lord Vishnu took the form of Ashtabhuja Perumal holding eight different weapons to kill the snake. The snake is found on the Vaayu end of the yaaga sala as "Sarabeswaran" in this temple
Another legend associates this temple to Gajendra moksham given by Vishnu to the elephant king Gajendra. As per the legend, the elephant Gajendra, used to worship Vishnu with the lotus fetched from the temple tank every day. Once while picking up lotus, a crocodile caught the leg of Gajendra, who started calling the name of Vishnu for help. Vishnu sent his discus to punish the crocodile and relieve the elephant. The presiding deity is addressed by various names like Adikesava Perumal, Gajendra Varadhan and Chakradhar.
The temple is originally believed to have been built during the rule of Pallavas. The temple has inscriptions from the period of Kulothunga Chola I (1070 - 1120) indicating gifts of land to the temple. There are contributions mentioned in the same inscription indicating contributions from the local people for perpetual lighting of the temple.
There are inscriptions in the Southern walls of the temple from Kulothunga I indicating sale of lands by Rajasundari Chaturvedimangalam to meet the kitchen expenses of the temple. Another inscription from the same period on the same walls indicates gift of tax free land in Chola Chaturvethimangalam to the temple for two years and levying tax of not more than 2 paise per veli subsequently. There are lot of gifts endowed to the temple during the Pallavan kings.