As per Hindu legend, Vishnu wanted to propitiate the sins caused by slaying Mahabali during Vamana avatar. He prayed Shiva for the same at various Shiva temples. His sins were cleared while worshipping Shiva. There is an image of Vishnu in a sculpture showing the legend.
As per another legend, Sambandar, the 7th century Saiva saint Nayanmar, visited the place and sung praises of Atulya Natheswarar. The temple was locked down earlier by Jain monks and was believed to have been brought to worship practises after Sambandar. He was planning to go to Tiruvannamalai, but was stopped by a divine force. He set up Annamalaiyar as Lingam in this place and started worshiping.
It is believed that Samndar's foot steps are seen at the place and the Annamalaiyar Temple at Tiruvannamali can be viewed from the point. Arai indicates rock while ani indicates beauty - since Shiva appears beautifully over the rock, the presiding deity of the temple is called Araiyanitheeswarar, which is Sankritised version of Atulya Nadheswarar.

Image of the Annamalaiyar and Amman shrine of the temple

The original complex is believed to have been built by Pallavas, with later expansion by Cholas, while the present masonry structure was built during the Vijayanagara Empire during the 16th century. There are three rock-cut caves from the Pallava regime during the 10th century. The current Draupadi Amman shrine is believed to have been a Murugan temple.
There are inscriptions from lot of medieval and later Chola emperors like Rajaraja Chola I (985–1014), Kulothunga Chola I (1070–1120), Vikrama Chola (1118–1135), Kulothunga Chola II (1133–1150), Rajadhiraja Chola II (1166–1178), Kulothunga Chola III (1178–1218), Rajaraja Chola III (1216–1256) and Rajendra Chola III (1246–1279).
There are inscriptions from Pandya kings like Maravarman Sundara Pandyan (1216–1238) and Maarvarman Vikrama Pandiyan. The contributions of the kings of the Vijayanagara Empire like Krishna Deva Raya (1509–1529), Sadasiva Raya (1542–1570) and Kampanna Udayar are also found in the inscriptions.

Sculptures on the temple walls
There are also inscriptions from the Sambuvaraya period, during the regime of Sagaloga Chadravarathi Rajanarayana Sambuvarayar.
In all, there are a total of 96 inscriptions in the temple compiled in the Annual Reports on South Indian Epigraphy for the year 1902 in numbers 386–391, year 1905 number 26 and year 1935 number 111–195.
Rajanarayana Sambuvaraya was a chieftain of Medieval Cholas whose contributions are documented in his inscriptions across various temples in modern-day Villupuram, Cuddalore, Tiruvannamalai and Kanchipuram districts and also in his Sanskrit work Madhuravijayam.
He repaired, revived the services and inaugurated festivals of the temple.[6] In modern times, the temple is maintained and administered by the Hindu Religious and Charitable Endowments Department of the Government of Tamil Nadu.