A year later, in January 1862, he finally received the news that he would be replaced by Robert Pruyn (1815-1882) as United States envoy to Japan, and that at last Harris would be able to return to the land of his fathers. Before he left, Moriyama, a senior Japanese diplomat, paid him a call and expressed his countrymens sentiments regarding Harris, demonstrating that he had won the trust of the Japanese:

We will miss you. You have been more than a friend. You have been our benefactor and teacher. Your sprit and memory will live forever in the history of Japan.

During his retirement in New York City, Harris was always eager to receive anyone at his club who had news of Japan. Harris would always ask these visitors how he was regarded in Japan. Harris had a keen interest in the affairs of the nation where he resided for six years, integrating this formerly isolated nation into the larger world.

Harris rejoined civic affairs in New York. In 1874 he was chair of the committee of the American Society for the Prevention of Cruelty to Animals, founded in 1866 by Henry Burgh (1813-1888).

Townsend Harris died on February 25, 1878, and is buried in Green-Wood Cemetery in Brooklyn. In 1986 City College accepted a gift from the Japanese people of a refurbished gravesite, which includes paving stones, a stone lantern, a cherry tree, a dogwood tree, and two commemorative stones.

The family of Townsend Harris honored the institution that he founded when, in 1905, his niece Bessie A. Harris, donated the Townsend Harris Papers to City College. This set of diplomatic correspondence and the journal he kept of his voyage to Japan and his first two years there is the core of the Harris collection in the City College Archives and is the source for most of the material in this exhibition.