What is the origin of the phrase "There goes the neighborhood" and does it have racial connotations?

It can definitely be used both ways. I'm not certain of the origin, but the ngram is steady at zero until the early 1960s.

It seems to have originated during and because of the American Civil Rights Movement and desegregation, so I believe it does have racial connotations.

I agree with everything in choster's answer but the 'originated' bit. There goes the neighbourhood surely comes from the snowclone (http://en.wikipedia.org/wiki/Snowclone) "There goes X", as in

There goes our plan.

There goes the country (down the drain).

There goes all our money.

There goes his dignity.

etc.

While there is nothing racist or classist intrinsic in a neighborhood "going" (maybe a flood just washed over it), the connotation of "there goes the neighborhood" is indeed racist and/or classist. Tread with care.

As choster's answer indicates, "There goes the neighborhood" is intimately associated with racial integration of neighborhoods in the United States during the 1960s, and with the demise of formal and informal deed restrictions and neighborhood covenants that prohibited the sale of homes in certain white neighborhoods to nonwhites. When—through individual defiance of the restriction or covenant or by some other means—a neighborhood ceased to be white only, the result was often white flight and a significant fall in the value of houses in the neighborhood. Hence, "There goes the neighborhood."

But "There goes the neighborhood" is in one way an extraordinary catch-phrase: Although it undoubtedly originated as a genuine expression of concern (or regret or horror) by a homeowner that a newcomer to the speaker's neighborhood was likely to lower the value of houses there, it quickly became an ironic punchline—so quickly, in fact, that the first four confirmable occurrences of the phrase in a Google Books search—all from 1967—are clearly satirical.

From AFL-CIO Committee on Political Education, Memo from COPE (1967) [snippet]:

Sen. Robert Kennedy recently moved his Senate office. It now adjoins offices of Dixiecrat John McClellan and Dixiegop Strom Thurmond. His invasion of this southern preserve prompts Washington line: "Well, there goes the neighborhood."

From syndicated columnist Art Buchwald, Son of the Great Society (1967) [combined snippets]:

When the flying saucers were sighted over Ann Arbor, Mich., a few weeks ago, the first reaction from one of the residents was, "Dammit, there goes the neighborhood."

It has been reliably reported that occupants of all flying saucers are little green men, and this raises a serious problem. Do we want our children to go to school with little green children? What happens to real estate values when the little green people start moving in on the block? Will the green people be responsible for a rise in the crime rate? These questions have to be answered before there is a mass invasion of them and we have a situation that will make Watts [the August 1965 Watts riots in Los Angeles] look like a tea party.

From a review of "Funny, You Don't Look It or (How Can You Say The Whole World Isn't Jewish When Even The Sun's Name Is Sol?)" (released in 1966), in American Record Guide, volume 33, issue 2 (1967) [combined snippets]:

So Paul Revere rouses a lady citizen with his warning that the British are coming. "Oy," she responds, "there goes the neighborhood!" Both Revere and his kindled compatriot, you see, sport burly Jewish accents, as does every other character in this series of would-be hilarious "historical" recreations. The episode is fairly typical of the base-ore humor in store for you in this latest assault on a fun-buff's sensibilities. But you will have to hear the record for yourself (an experiment I am not recommending) to believe its abysmal taste.

From a review of Pete Seeger, Waist Deep in the Big Muddy and Other Love Songs [released August 1, 1967], in Hi Fi/Stereo Review, volume 19 (1967) [combined snippets]:

With a chorus of friends and neighbors from his home town of Beacon, N.Y., he offers a swinging treatment of his own number "Oh, Yes, I'd Climb the Highest Mountain," as well as spirituals and traditional folk tunes. In the course of "Seek and You Shall Find" he pauses for a couple of entertaining anecdotes, including one about an Indian who hears Columbus greeting his tribe in Spanish and mutters, "Well, there goes the neighborhood." He plays a calypso number on his banjo, offers two moving ballads without accompaniment, and winds up the show with a well-intentioned if bathetic number about the civil-rights trio murdered in Mississippi.

All four unique Google Books matches for "There goes the neighborhood" from 1968 are joke punchlines, too. More-recent usage of the catch-phrase seems divisible into three categories: humorous use in a joke setting, as in the 1967–1968 examples; sarcastic use as a view attributed to someone else; and catch-phrase-recognition-based use in display type (chapter titles, section subheads, and the like), where the point is the familiarity of the phrase, which may be only tangentially relevant to what's going on in the text. An instance of the first kind is Andrew Greeley, The Bishop Goes to the University: A Bishop Ryan Novel (2010):

"Half of the Div School students are Catholic and almost half the faculty."

"There goes the neighborhood."

She thought that was very funny.

An example of the second type occurs in Evan Hughes, Literary Brooklyn: The Writers of Brooklyn and the Story of American City Life (2011):

Some see the uniqueness of today's Brooklyn as being under threat or on the wane. Whenever a Starbucks opens, or a chain store, or another real estate broker, you hear the refrain: there goes the neighborhood. A culturally vital neighborhood is a neighborhood where a lot of people want to be, and a place where a lot of people want to be, and a place where a lot of people want to be is a place that's more expensive. Gentrification has its costs, and they are serious.

An example of the third type is this chapter title and subtitle from Anna Soffee, Nerd Girl Rocks Paradise City: A True Story of Faking It in Hair Metal L.A. (2006):

[Chapter] 7. There Goes the Neighborhood

The Smell of Hairspray Gives Way to Teen Spirit

Given its history, "There goes the neighborhood" has always been freighted with racial overtones. But it has also been freighted, from a very early date, with the weight of ironic and facetious usage. Neither form of baggage is benign, in my opinion. If you plan to use the catch-phrase, be aware that it has a peculiar and complicated past, and that it remains loaded today.