During his collecting years, Morgan maintained several residences. At the time of his father's death in 1890, he inherited a town house at No. 13 Princes Gate, London, where, through the early 1900s he kept most of his collections, partly because of the 1897 U.S. Government Revenue Act, which imposed a 20 percent tariff on imported works of art. To accommodate his growing collections in England, he bought No. 14 in 1904 and opened the walls between the two buildings, converting the first two floors into galleries.

His burgeoning New York collections were initially housed in a little locked treasure room in the basement of his house at 219 Madison Avenue. It soon became apparent, however, that other arrangements had to be made. In 1902 he commissioned the noted architectural firm of McKim, Mead & White to construct a separate building adjacent to his home in New York. Charles McKim, who undertook the project himself, was told quite clearly by his client that his collections were to be housed in a magnificent structure. Morgan told the architect, "I want a gem."