Waller Holladay began to acquire plantation land in 1803 with an inheritance from his half brother General Lewis Littlepage, a European adventurer and court representative during the early years of the American Republic. First apprenticed to the household of John Jay, envoy to the court of Spain, Littlepage concluded his career in the diplomatic service of Stanislaus Augustus II, the last king of Poland. Littlepage's colorful life brought him in contact with such luminaries as Washington, Franklin, Jefferson, Lafayette, Potemkin, and Empress Catherine of Russia.
General Littlepage's legacy enabled Waller and his wife Huldah to engage Spotswood Dabney Crenshaw to build the main house at Prospect Hill in 1811-1812, where they raised thirteen children. Their youngest son, James Minor Holladay, was captured at the Civil War battle of Five Forks, only eight days before Lee's surrender at Appomattox. Months later, after his release from the Federal prison camp at Point Lookout Maryland, James walked home to Prospect Hill, his wife Lucy and their two small toddlers. James and Lucy brought the farm back to successful operation, and in the 1880's added a large wrap-around porch, the only significant addition to Prospect Hill until 1991.
In 1991-1996 Prospect Hill was carefully restored by the Holladay family members of the fifth and sixth generations. It has been chosen as a recipient of the Annual Great American Home Award (Interior Category) by the National Trust for Historic Preservation. In addition to the manor house, seven dependencies have been restored, with suites available in the east barn, the granary, and the plantation office.
Now this old family home welcomes guests as the Littlepage Inn.