To the south of North Berwick on a rocky ridge stands the surprisingly intact ruin of Fenton Tower. The site consists of a long L-plan towerhouse with key marks on it's north facing wall suggesting the possibility of an enclosing barmkin wall, though there is little suggestion of it elsewhere.

During the reign of King David I of Scots (1124-1153) the lands of 'Fentoun' along with the lands of Gullane, Archerfield, Dirleton and Fidra island were all held by the Anglo-Norman De-vaux family. Who later gifted 'Fentoun' to their English kin the De-vauxs of Lanercost Priory. In the mid 12th century the De-vauxs built a 'castri' on the Fidra, a chapel at Gullane, a tower of 'Eilbote' at Archerfield (which must have been a place of some importance since King David signed a charter for the lands of Carberry witnessed at 'El bottle' ) and finally a stone castle at Dirleton itself during the reign of King Alexander II of Scots (1214-1249).

The present tower of Fenton though was not built by the De-vauxs, whose lineage merged with the Halyburton family in the mid 14th century, but was possibly built by Sir John Carmichael in the 1570's.As the tower datestone of 1577 bears his initials. Although he and his wife Margaret Douglas don't appear to have come into procession of Fenton until 1587 when the lands were forfeited by the Whytelaws. Locally the Carmichaels also held the ancient Hepburn stronghold of Waughton castle, near Whitekirk, for a time when in 1569 it was attacked by "Robert Hepburn" who "came to the house of Waughton and brake the stables and took out sixteen horses, the laird of Carmichael being captain and said keeper of Waughton. Then issued out the house and slew three of them, and divers were hurt of both parties". This attack may have caused the Carmichaels to build Fenton as Waughton eventually passed to the Hepburns by legal means.

The Carmichaels originally took their name from their barony in Lanarkshire,as 'John de Carmychell' was given a charter to the lands of 'Carmychell' by William 1st Earl of Douglas of Tantallon castle in the late 14th century. One of the Fenton Carmichael's most famous ancestors was 'John Carmichael of Douglasdale' who in 1421 at the battle of Bauge in France broke his lance while unhorsing the Duke of Clarence (King Henry V of England's brother). The French account (Les Cahiers du Baugeois') makes interesting reading, "Carmichael of Douglasdale spurred his mount on with such dash that he shattered his lance on the breast plate of the Duke who was unseated by the blow." Clarence was then killed and the English army routed. The Carmichael badge today bears an armoured hand holding aloft a broken lance in memory of this victory.

In 1591 while staying at Falkland Palace King James VI (1567-1625)was surrounded by a rebel army led by Stewart Earl of Bothwell. Fortunately the local towns people rallied to the King's support allowing him to flee with his life. Instead of heading to Stirling castle James caught the ferry to North Berwick and stayed with the Carmichaels at Fenton Tower. Which showed he had great trust in them as Fenton was not as strong as the likes of Stirling or Edinburgh castles. James was plagued with many abduction attempts throughout his life in 1589 Bothwell had plotted to seize the King at Hatton castle and in 1600 in what became known as the 'Gowrie conspiracy' the Ruthvens of Dirleton castle tried to take the King hostage at Perth. In that same year Sir John Carmichael, while carrying out his duties as March Warden was killed during peace talks on the border.

Fenton appears to have escaped the wrath of Cromwell's army during the 1650's sacking of Lothian castles ,and it has escaped the destruction of the stone robbers who quarried at such sites in the 1700's.Though they may have stole the tower's enclosing Barmkin wall. The remains today are surprisingly intact and ideal for any would be restorer.