This temple stands at the place where Rama is said to have performed the Ashvamedha Yajnya. About 300 years ago, the Raja of Kulu built a new temple here, which was improved by Ahalyabai Hokar of Indore during 1784, at the same time the adjoining Ghats were also built.
The initial idols in black sandstone were recovered from Saryu and placed in the new temple, famous as Kaleram-ka-Mandir.