The original farm on which the Waite Potter house was located included 200 acres on both sides of Dartmouth's Main Road and as far east as the Noquochoke River. The house built by Thomas Waite in 1677 was a one room, one and one-half story building, with a large stone fireplace incorporated into its interior side. Known as a ‘’Rhode Island Stone Ender,’’ there are only a few remaining in Rhode Island today.

In 1728, Thomas Waite’s son Benjamin sold a portion of parcel to Robert Kirby, who in 1760 added one room to the opposite side of the stone chimney. After the Kirbys stopped living in the house, it was used as a farm building and pig pen. By 1945, the building was in disrepair, and in 1954, Hurricane Carol ripped the roof off the structure, leaving it open to rot and ruin. By 1962, the chimney had started to topple as the main support, the large oak lintel that held up the throat of the chimney, had rotted away and by 2006, everything had collapsed in the front. Although the house no longer stands, its image is on Westport's town seal.