The ancient church of Arndilly stood where Arndilly House now stands above the eastern bank of the River Spey. A symbol stone found at the site is built into the west gable and shows the ancient connections of this site, which may have been one of the earliest churches in the district.

The house was built by David Macdowall Grant in 1750. In 1850, the house was enlarged in Neo-Jacobean style by Thomas Mackenzie (1814-54). His eldest son, William M’Dowall Grant of Arndilly, died in 1849, and the estate passed to his brother, Hay M’Dowall Grant, 7^th of Arndilly (1806-1870). The last of the family to hold Arndilly was Marjory Alexandrina McDowall Grant, eldest daughter of William M’Dowall Grant.