Built in the form of an eighteenth century mansion, the Ordinary is a two-story rectangular structure with a facade of five bays. The front wall is laid in all-header bond with rusticated stone quoins at the corners and a fine rusticated stone doorway. It is believed that the Ordinary is the only building left in Virginia with all-header bond construction.
The window openings are spanned by flat stone arches with superimposed keystones. The basement wall is of random order ashlar below a molded stone water table. The house is crowned by a fully molded wood cornice with modillions and hipped roof. Also features on the house are four interior end chimneys.
Neither a construction date nor a builder for the Ordinary is known. It has been suggested that it was built around 1765 by James Wren because of the many stylistic parallels between it and the Christ Church in Alexandria, on of Wren's best-known works. Also, the affinity between the Ordinary and many Annapolis houses suggests a connection with William Buckland.
During the colonial times, the Ordinary was known as Williams Ordinary. Over the years it was also known as Love's Tavern, the Stage Coach Inn, and others. During the Civil War, the building was used as a Confederate Headquarters during the blockade of D.C. along the Potomac River. Today it is a private residence being restored.