Following the governor's death in 1770, Martha married Michael Wentworth, a retired British army colonel and accomplished musician. They made the mansion a hospitable social center and entertained George Washington when he visited Portsmouth in 1789. They had one daughter, Martha, who inherited the estate from her widowed mother in 1805. She and her husband, John Wentworth, remained on the property until 1816 when they sold the house and the 113 acres to the successful merchant, Charles Cushing of Roxbury, Massachusetts.

Cushing, his wife, and seven children lived permanently at Little Harbor and continued operating the estate as a working farm. After Cushing's death in 1849, the property eventually passed to his nephew, William P. Israel, in 1860. Israel actively promoted the property to tourists, making the house one of the first historic dwellings in the United States to be opened to the public. In 1886, he sold about 15 acres with various buildings, known as "the Governor Wentworth Estate," to John Templeman Coolidge, III, of Boston.