What some people conveniently set aside is that slavery was not solely a southern institution, but a national one. Even so called abolitionist states had resident slave owners. Ulysses S. Grant, the general who defeated Lee, didn't release his slaves until 1865, but no one is talking about destroying Grant's tomb.

The publication of the Emancipation Proclamation caused widespread desertion in the Union army, rioting in New York City, with hangings of black residents, and at least one attempt to overthrow a Union state (Indiana) government and join the Confederacy. The Indiana Copperheads failed in their attempt, but were still strongly in favor of white supremacy (some would say they still are, since one of remaining KKK bastions is in this state).

My family owned the piece of land called the "surrender triangle" at Appomattox where the defeated Confederates stacked their arms. One of my ancestors (great-great uncle) rode with J.E.B. Stuart, my great-great grandfather was a Confederate artilleryman, and the two other brothers who fought in the war died in Union prison camps. Nonetheless I agree with Lee that memorials should not have been erected, and in fact most of them were put up in the 20th century.

My reason for not tearing these monuments down and renaming myriad schools, streets and the like has nothing to do with nostalgia nor southern heritage. I object to the expense of all this destruction. When the nation and nearly every state is sweating over how to fund building and repairing infrastructure (just one example), why on Earth spend taxpayer dollars on an effort not supported by most in the nation?

Incidentally, Walter Williams, the renowned black economist, agrees with Lee's observation that the slaves were better off than their relatives that remained in Africa. Williams points out that the life expectancy of an American slave in the 1800s was better than the current African life expectancy, and that the descendants of American slaves are better off materially, better educated, and with more freedom than a citizen of any African state today. This does not mean Williams thinks slavery was a defendable institution, but that the consequences had unexpected positive outcomes.

While Robert E. Lee acknowledged the evil of slavery,that out-take must be put into context.

Letter to his wife on slavery (selections; December 27, 1856)
"The steamer also brought the President's message to Cong; & the reports of the various heads of Depts; the proceedings of Cong: &c &c. So that we are now assured, that the Govt: is in operation, & the Union in existence, not that we had any fears to the Contrary, but it is Satisfactory always to have facts to go on. They restrain Supposition & Conjecture, Confirm faith, & bring Contentment: I was much pleased with the President's message & the report of the Secy of War, the only two documents that have reached us entire. Of the others synopsis [sic] have only arrived. The views of the Pres: of the Systematic & progressive efforts of certain people of the North, to interfere with & change the domestic institutions of the South, are truthfully & faithfully expressed. The Consequences of their plans & purposes are also clearly set forth, & they must also be aware, that their object is both unlawful & entirely foreign to them & their duty; for which they are irresponsible & unaccountable; & Can only be accomplished by them through the agency of a Civil & Servile war. In this enlightened age, there are few I believe, but what will acknowledge, that slavery as an institution, is a moral & political evil in any Country. It is useless to expatiate on its disadvantages. I think it however a greater evil to the white man than to the black race, & while my feelings are strongly enlisted in behalf of the latter, my sympathies are more strong for the former. The blacks are immeasurably better off here than in Africa, morally, socially & physically. The painful discipline they are undergoing, is necessary for their instruction as a race, & I hope will prepare & lead them to better things. How long their subjugation may be necessary is known & ordered by a wise Merciful Providence. Their emancipation will sooner result from the mild & melting influence of Christianity, than the storms & tempests of fiery Controversy. This influence though slow, is sure. The doctrines & miracles of our Saviour have required nearly two thousand years, to Convert but a small part of the human race, & even among Christian nations, what gross errors still exist! While we see the Course of the final abolition of human Slavery is onward, & we give it the aid of our prayers & all justifiable means in our power, we must leave the progress as well as the result in his hands who sees the end; who Chooses to work by slow influences; & with whom two thousand years are but as a Single day. Although the Abolitionist must know this, & must See that he has neither the right or power of operating except by moral means & suasion, & if he means well to the slave, he must not Create angry feelings in the Master; that although he may not approve the mode which it pleases Providence to accomplish its purposes, the result will nevertheless be the same; that the reasons he gives for interference in what he has no Concern, holds good for every kind of interference with our neighbors when we disapprove their Conduct; Still I fear he will persevere in his evil Course. Is it not strange that the descendants of those pilgrim fathers who Crossed the Atlantic to preserve their own freedom of opinion, have always proved themselves intolerant of the Spiritual liberty of others?"
http://fair-use.org/robert-e-lee/lett...

Robert E. Lee opposed Confederate memorial statues.

Overarching article from "Task and Purpose" military support website
http://taskandpurpose.com/gripknife-r...
includes links to previous research.

"Sender: Robert E. Lee
Recipient: Thoms L. Rosser

Lexington VA 13 Dec - r 1866
My dear Genl
I have considered the questions in your letter of the 8th Inst: & am unable to advise as to the efficacy of the scheme proposed for the accomplishment of the object in view. That can be better determined by those more conversant with similar plans than I am.
As regards the erection of such a monument as is contemplated; my conviction is, that however grateful it would be to the feelings of the South, the attempt in the present condition of the Country, would have the effect of retarding, instead of accelerating its accomplishment; & of continuing, if not adding to, the difficulties under which the Southern people labour. All I think that can now be done, is to aid our noble & generous women in their efforts to protect the graves & mark the last resting places of those who have fallen, & wait for better times.
I am very glad to hear of your comfortable establishment in Baltimore & that Mrs. Rosser is with you. Please present to her my warm regards. It would give me great pleasure to meet you both anywhere, & especially at times of leisure in the mountains of Virginia; but such times look too distant for me to contemplate, much less for me now to make arrangements for -
Very truly yours
R E Lee
Genl Thos: L. Rosser"
Notes: Lee PapersUniversity of Virginia Archives
http://leefamilyarchive.org/papers/le...

“When Lee surrendered at Appomattox he also signed a parole document swearing upon his honor not to bear arms against the United States or to ‘tender aid to its enemies.’ "
Like all of the army and navy officers who abandoned their oaths to the union and the Constitution in joining the southern revolt, Lee already broke his word once. But let us accept that this time he really meant it.

Virginia certainly did not merely petition Congress or bring suit before the Supreme Court. The war started with the bombardment of Fort Sumter in Charleston harbor. What did not happen was that a southern state following rule of law was invaded by federal troops or federal marshals who arrested the government of that state. The elected representatives of Virginia and the other secessionists in other states then joined the attack on the United States.

Yes, many people in the south knew that slavery was (1 )morally wrong and (2) economically doomed. They still clung to it, often arguing from their own alleged benevolence for the so-called "colored race."

Read the Constitution of the United States. Importation of slaves was forbidden after 1807. Everyone knew the end of slavery was inevitable and for another lifetime - 60 years - the southerners did nothing. In fact, they passed even harsher laws and engaged in worse punishments. People of African ancestry, free and slave alike, in America in 1835 were worse off than they had been in 1789.

The northerners who were opposed to slavery compromised time and again with the southern representatives. The Kansas-Nebraska Act nullified the Compromise of 1850. Compromise only delayed the inevitable and did nothing to bring the slave owners into the industrial age.

Moreover, it is curious that the same conservatives who distrust democracy as mobocracy excuse the mobs in the southern states. We all know that just because "everyone" votes for something does not make it right.

That Lee and the other traitors got off so lightly speaks deeply to magnanimity of the American culture, the American spirit. We are nice to our enemies.