The Macdonalds supported the Jacobite cause and after 1715 and 1745 the retribution for this support led to their decline. The castle gradually fell into disrepair, probably more due to the frequent absence of the chief rather anything else.
More appealing reasons are given however. Their departure is sometimes attributed to the ghostly harassing by their ancestor Hugh. Another reason involves the tragic story of the baby which somehow fell from his nurses arms through a window onto the rocks below. The nurse was either thrown to her death after him, or tied up and set adrift in a boat, depending on the story, but her ghostly cries are still said to be heard in the castle and might provide some motive for the occupants to move elsewhere.