In 1865, just as the Civil War came to an end, Pierpont married again, this time to Frances Louisa Tracy, a woman from his New York social circle. The two proved incompatible in temperament and had widely different instincts and tastes. Morgan loved New York City, hard work, a hectic social life, adventurous travel, and sumptuous luxury in art, houses, interior decor, clothing, and yachts. Fanny preferred a quieter, more domestic life with her children and a few intimate friends.
Pierpont and Fanny had four children. Louisa, the first and favorite daughter, was devoted to her father and traveled with him frequently before her marriage to Herbert L. Satterlee. J. P. Morgan, Jr., called Jack, succeeded his father as head of the house of Morgan. Juliet married a descendant of Alexander Hamilton, William Pierson Hamilton, who joined his father-in-law's banking firm. Anne, the only free spirit among the Morgan children, went on to found a major relief agency in postwar France.
In 1882, the family moved to a brownstone mansion at 219 Madison Avenue, at 36th Street, which remained the Morgans' New York City residence throughout Pierpont's life. He spent several months of each year abroad and was frequently at his father's London town house at Princes Gate or at Dover House, the family estate at Roehampton in South west London.