What is the origin of the slang term "get out of here" to mean "you're kidding"?

It is slang and written as "get outta here!" There is a 1967 "We Gotta Get Outta this Place" by Charles Wayne "Chuck" Day

get outta here! (Slang) you are kidding!, you must be kidding!; stop lying

English contemporary dictionary. 2014.

As in 'you're kidding':

I found "outta here" in a a 1920 book title SOME NEPHEW! Goldilocks threatenin' the popularity of Robin. I tells him all about it, and instead of bein' worried, he shows his brain work by laughin' like it was the funniest joke he ever heard. "Ed," he yells, "you is wastin' your time runnin' a movie picture studio. They is more managers on Broadway than they is fish in the sea, only lookin' for comedians like you to make rich and famous!" He opens the door, just as I gets up to my feet. "Get away from outta here before I hit you with the desk! You is so thick that what I have just told you must be like a fairy tale to a child!" The door closes, and I hear him still laughin'. Even when I looks outta my office window, and get a slant at him walkin' through the rain, his shoul- ders is still shakin'.

& as in 'go away':

from Beef, iron and wine, by Jack Lait 1916: Gene, you step aside an lemme beat it outta here peaceful an quiet.

I haven't found a suitably early occurrence of "get outta here" in the sense you have in mind, but I suspect it originates in comedy routines where one member of the comedy team pretends to be a drunk, a heckler, or a clueless interloper blundering onto stage and ruining the supposed smooth running of the routine; eventually after the intruder "steals" a big laugh, the straight man demands that the intruder "get outta here." Comedy duos have undoubtedly been using versions of this schtick since long before the birth of vaudeville.

For an odd variation on the setup, consider the lyrics to the Velvet Underground's 1968 song "Temptation Inside Your Heart," where Lou Reed is supposedly trying to sing a serious version of the song while an unwelcome intruder in the recording studio interrupts with comments like "You can talk during this" and "You don't look like Martha and the Vandellas," and Lou responds with "Get outta here" and "Somebody get her outta here." (The "she" in question never leaves during the entire length of the recording.)

Another form of the expression is "Get out!" as used by "Bob and Doug Mackenzie" in the recurring "Great White North" comedy sketches on SCTV which began in 1980; the Mackenzies used the catchphrase regularly when one of the characters disagreed vehemently with something (usually an exaggeration, insult, or bald-faced lie) that the other had just said.

Another possibly related phrase is "Go on with you," in the sense of "You don't really mean it." Here, for instance, is an exchange in Gerard Majella Murray, Career Angel (1944):

Kurt: Suppose we just forget about it? You know, sometimes I feel that I'd like to stay here the rest of my life.

Brother Gregory: Go on with you. Your eye's on another target.

Kurt: Just the same, I mean it, Brother.

This term may have originated from several places.

First, the sport of baseball.

The usage may have come from: "It's outta here!" that was used during the occurrence of a home-run. From this basis and with the help of the radio, this terminology might have caught on. Anything that seems out of the ordinary was deemed 'out of here' and when a tall tale or something mind-boggling was stated, the term must have spread.

There are numerous phrases that have originated from sports that we use today, and this could be another one of those.

Second, from a person about to be fired telling a tall tale or making a elaborate excuse to keep his job.

This might have evolved during the stock market crash of the 1920 where jobs were starting to get hard to come by and the country was on its way to it poverty stricken status of the 1930s. The boss might have listened to story after story and told each one to 'get out of here', which might have led to this statement being what it is.

Third, is has been adopted from a foreign dialect. Spanish is the most likeliest as they have an expression that translates quite nicely, specifically ¡vamos! which is now adopted into the English vocabulary as vamoose.

Fourth, really bad books. There is a book called Gus the Bus and Evelyn, the exquisite checker published in 1917 that uses this phrase in its literal form. It implies that the person is nutty (see image below).

Fifth, Vaudeville. The stage was a place where all sorts of people could watch all sorts of plays, acts, etc. Bad acts were told to get off the stage, which lends to the whole (in)famous Vaudeville Hook shtick. A bad act was considered outrageous because it was a waste of time and money for the audience. This could have evolved from this point to mean that something is outrageous (as in a tall tale).

Again, these are all speculative answers and I cannot find enough concrete evidence to fully throw in with one or the other.

I'm not sure about the origin of the phase, but in chapter 12 of "The War of The Worlds" by H.G. Wells, a character identified as the lieutenant says "get out [...] what confounded nonsense" in response to a description of the Martians. This book was published in 1898 making it a very strong contender for being the first recorded of the saying.