In 1988 the Morgan acquired the Morgan House. Situated on the corner of 37th Street and Madison Avenue, to the north of the Annex, it is the sole survivor of the complex of three residences originally built in 1852/53 for the Phelps, Stokes, and Dodge families, of Phelps-Dodge copper fame. In 1881, Pierpont Morgan purchased 219 Madison Avenue, the southernmost brownstone on the corner of Madison Avenue and 36th Street, as his own residence (demolished in 1928 to make way for the Annex). Morgan bought the central brownstone in 1903, which was then razed to make space for a garden, and a year later he purchased the northernmost house, at 231 Madison, for his son, Jack Morgan. With forty-five rooms, including twelve bathrooms, the house was one of the most impressive residences of its day.
The Morgan family lived in this house until Jack died in 1943, and then sold the brownstone to the Lutheran Church in America to serve as its headquarters. In 1957 the Church constructed a five-story office adjacent to the mansion. In the spring of 1988, it was purchased by the Library. During its forty-four year occupancy, the Lutheran Church carefully maintained the mansion's exterior structure, as well as the interior walls and adornments.
The Morgan House, a rare freestanding brownstone in Manhattan, is a New York City Landmark.
In 1991 the Morgan opened the Garden Court—a glass-enclosed conservatory, constructed to link the Morgan House with the Annex. Architects for the project were Voorsanger & Associates, uniting the Morgan House with the Annex and the original McKim building.