His love of the country and faith in rural influences were too genuine for him not to be deeply interested in the improvement of cities by means of public parks and gardens. Not only for their sanitary use, but for their elegance and refining influence, he was anxious that all our cities should be richly endowed with them. He alluded frequently to the subject in the columns of his magazine, and when it was resolved by Congress to turn the public grounds in Washington, near the Capitol, White House, and Smithsonian Institute, into a public garden and promenade, Downing was naturally the man invited by the President, in April, 1851, to design the arrangement of the grounds and to superintend their execution. All the designs and much of the work were completed before his death. This new labor, added to the rest, while it increased his income, consumed much of his time. He went once every month to Washington, and was absent ten or twelve days.
He was not suffered to be at peace in this position. There were plenty of jealousies and rivalries, and much sharp questioning about the $2500 annually paid to an accomplished artist for laying out the public grounds of the American Capital, in a manner worthy the nation, and for reclaiming many acres from waste and the breeding of miasma. At length the matter was discussed in Congress. On the 24th March, 1852, during a debate upon various appropriations, Mr. Jones, of Tennessee, moved to strike out the sum of $12,000, proposed to complete the improvements around the President's house; complained that there were great abuses under the proviso of this appropriation, and declared, quite directly, that Mr. Downing was overpaid for his services. Mr. Stanton, of Kentucky, replied: - "It is astonishing to my mind - and I have no doubt to the minds of others - with what facility otherwise intelligent and respectable gentlemen on this floor can deal out wholesale denunciations of men about whom they know nothing, and will not inform themselves; and how much the legislation of the country is controlled by prejudices thus invoked and clamor thus raised." After speaking of the bill under which the improvements were making, he continued: "The President was authorized to appoint some competent person to superintend the carrying out of the plan adopted.
He appointed Mr. Downing. And who is he? One of the most accomplished gentlemen in his profession in the Union; a man known to the world as possessing rare skill as a ' rural architect' and landscape gardener, as well as a man of great scientific intelligence. * * * * I deny that he has neglected his duties, as the gentleman from Tennessee has charged. Instead of being here only three days in the month, he has been here vigilantly discharging his duties at all times when those duties required him to be here. He has superintended, directed, and carried out the plan adopted, as fully as the funds appropriated have enabled him to do. If all the officers of the Government had been as conscientious and scrupulous in the discharge of their duties as he has been since his appointment, there would be no ground for reproaches against those who have control of the Government".
Mr. Downing was annoyed by this continual carping and bickering, and anxious to have the matter definitely arranged, he requested the President to summon the Cabinet. The Secretaries assembled, and Mr. Downing was presented. He explained the case as he understood it, unrolled his plans, stated his duties, and the time he devoted to them, and the salary he received. He then added, that he wished the arrangement to be clearly understood. If the President and Cabinet thought that his requirements were extravagant, he was perfectly willing to roll up his plans, and return home. If they approved them, he would gladly remain, but upon the express condition that he was to be relieved from the annoyances of the quarrel. The President and Cabinet agreed that his plans were the best, and his demands reasonable; and the work went on in peace from that time.
The year 1852 opened upon.Downing, in the garden where he had played and dreamed alone, while the father tended the trees; and to which he had clung, with indefeasible instinct, when the busy mother had suggested that her delicate boy would thrive better as a drygoods clerk. He was just past his thirty-sixth birthday, and the Fishkill mountains, that had watched the boy departing for the academy where he was to show no sign of his power, now beheld him, in the bloom of manhood, honored at home and abroad - no man, in fact, more honored at home than he. Yet the honor sprang from the work that had been achieved in that garden. It was there he had thought, and studied, and observed. It was to that home he returned from his little excursions, to ponder upon the new things he had seen and heard, to try them by the immutable principles of taste, and to test them by rigorous proofs. It was from that home that he looked upon the landscape which, as it allured his youth, now satisfied his manhood. The mountains, upon whose shoreward slope his wife was born under the blossoming locusts on the very day on which he was born in the Newburgh garden, smiled upon his success and shared it. He owed them a debt he never disavowed.
Below his house flowed the river of which he so proudly wrote in the preface to the "Fruit-Trees" - "A man born on the banks of one of the noblest and most fruitful rivers in America, and whose best days have been spent in gardens and orchards, may perhaps be pardoned for talking about fruit-trees." Over the gleaming bay which the river's expansion at Newburgh forms, glided the dazzling summer days; or the black thunder-gusts swept suddenly out from the bold highlands of West Point; or the winter landscape lay calm around the garden. From his windows he saw all the changing glory of the year. New York was of easy access by the steamers that constantly passed to and from Albany and the river towns, and the railroad brought the city within three hours of his door. It brought constant visitors also, from the city and beyond; and scattered up and down the banks of the Hudson were the beautiful homes of friends, with whom he was constantly in the exchange of the most unrestrained hospitality. He added to his house the working-room communicating with the library by the mysterious door, and was deeply engaged in the planning and building of country-houses in every direction.