Has a stereotypical "mad scientist" ever made a significant discovery?

André Bloch is an extreme case: He murdered three of his family members. Being institutionalised for the rest of his life, he wrote influential mathematical papers (on complex analysis). Quote from Wikipedia:

On 17 November 1917, while on convalescent leave from service in World War I, Bloch killed his brother Georges, and his aunt and uncle. Several conjectures about the motives for Bloch's crime exist among mathematicians. [...] Bloch told [the medical head of his asylum] that the murders were a eugenic act, in order to eliminate branches of his family affected by mental illness.

[...]

After the murders, Bloch was committed to the asylum at Charenton in Saint Maurice, a suburb of Paris. Bloch continued his mathematical career while confined. All of his publications, including those relating to Bloch's constant, were written while he was committed. Bloch corresponded with several mathematicians [...] giving his return address as only "57 Grande rue, Saint-Maurice", never mentioning that this was a psychiatric hospital. Several of his correspondents were thus unaware of his situation.

Kurt Gödel achieved some of the most important breakthroughs in logic in the 20th century. As far as mental instability is concerned, quote from Wikipedia:

Later in his life, Gödel suffered periods of mental instability and illness. He had an obsessive fear of being poisoned; he would eat only food that his wife, Adele, prepared for him. Late in 1977, she was hospitalized for six months and could no longer prepare her husband's food. In her absence, he refused to eat, eventually starving to death. He weighed 29 kilograms (65 lb) when he died.

Sophus Lie probably qualifies, being both a great mathematician and a patient of mental institutions.

According his Wiki, he largely created the theory of continuous symmetry and applied it to the study of geometry and differential equations.

I think Paul Dirac would have certain characteristics which fit this category. Often called as "theorist's theorist", Dirac was science's archetypal loners,taciturn and devoid of empathy. He has an unhappy childhood, but doesn't mention it for 50 years; learns to speak French, German and Russian, but becomes famous for his long silences; embarks on the wrong career; gets interested in mathematics and ends up at Cambridge, where he becomes famous for his even longer silences; hears about Einstein and gets into advanced physics; and then goes to Copenhagen to meet Niels Bohr, who grumbles to Ernest Rutherford, "This Dirac, he seems to know a lot of physics, but he never says anything". A lonely boy who becomes a lonely man driven by the concept of mathematical beauty

Georg Cantor is well known as someone who was admitted to a mental asylum several times in his lifetime, mainly it seems because of the opposition to his ideas on the characterising mathematical infinities; a theory, that today passes without much comment. In fact, Kronecker

perceived him as a "corrupter of youth" for teaching his ideas to a younger generation of mathematicians.

One might then view that his sickness was not within, but without; that is a sick society. This is after all, one of the main themes of romanticism, which arose as a reaction to a sick society and fed into science fiction and fantasy.

The most commonly mentioned name in this context is Nicola Tesla, he is even featured as such in some fiction (e.g. Tomorrowland). He was brilliant, eccentric, kept to himself, and had some wacky ideas, especially later in life, including persistent search for the ether, see Was Nikola Tesla right about his ether theory? Conspiracy theories around his late life and work were fueled by FBI seizing his belongings when he died. However, and this is were real life diverges from fiction, if the "mad" scientist is supposed to be successful in his "mad" activities, it is not how it worked out with Tesla. He was most successful, indeed brilliant, in his less "mad" early years. Hartsfield gives an interesting take titled Mad Scientists of the Modern Age: Nikola Tesla:

"Edison's light bulb is trumpeted as the signal achievement beginning the electrical era. However, it was Tesla who devised most of the alternating current (AC) electrical systems that the world's power grids are modeled upon. He single-handedly patented AC electrical generators, transformers, power lines, and motors. The electrical system formed by these devices proved so much superior to Edison's direct current (DC) electrical system that it was adopted universally across the globe... Tesla was in many ways the opposite of Thomas Edison. Edison was never known as a brilliant man, but as a methodical, determined one. "Genius is 99 percent perspiration and one percent inspiration," he once said. Tesla was a nearly unparalleled genius, but lacked the business intuition and common sense of Edison. Edison died rich and one of most famous men in the world, while Tesla died penniless and alone, feeding pigeons.

[...] After years of wildly successful invention, Tesla began to lose his way, making grandiose, eccentric and sometimes completely impossible claims. He fervently spoke of building a "teleforce weapon" capable of melting steel wirelessly from 200 miles away. Upon his death, a box supposedly containing a death ray part was found to contain only a basic piece of old equipment. Tesla believed that he could create a camera to project a brain's thoughts onto a wall. He built a 187 foot-tall tower to try to wirelessly convey electric power across the planet, using the earth itself as an enormous electrical conductor. He wanted to harness the entire sphere of the Earth as a power line! Upon picking up odd signals at one of his labs, Tesla believed that they could have been sent from Mars by using that planet similarly as a conductor.

Celibate his entire life, Tesla became lonely and slightly paranoid in his elder years. He claimed to keep his ideas inside of his brain to keep them from being stolen. He was suspicious of twentieth century physics, as explained by Einstein and others... An entire conspiracy theory movement has formed around Tesla's "hidden" or "suppressed" inventions and ideas. The United States government also unwittingly contributed to Tesla's legacy of mad genius. Shortly after his death in 1943, the FBI seized all of his belongings."

Real Clear Science has a whole series on Mad Scientists of the Modern Age, they name Vladimir Demikhov (experimental surgeon known for intrathoracic transplantation), Jack Parsons (jet and rocket propulsion whiz), Lynn Margulis (the founder of endosymbiotic theory), Barry Marshall (discovered the cause of ulcers), and Josef Mengele (Nazi death doctor). On the general correlation between brilliance and "madness" see Why Are Genius and Madness Connected? on Live Science:

"Kay Redfield Jamison, a clinical psychologist and professor at Johns Hopkins University School of Medicine, said the findings of some 20 or 30 scientific studies endorse the notion of the "tortured genius." Of the many varieties of psychosis, creativity appears to be most strongly linked to mood disorders, and especially bipolar disorder, which Jamison suffers from herself. For example, one study tested the intelligence of 700,000 Swedish 16-year-olds and then followed up a decade later to learn which of them had developed mental illnesses. The startling results were published in 2010. "They found that people who excelled when they were 16 years old were four times as likely to go on to develop bipolar disorder... "People with bipolar tend to be creative when they're coming out of deep depression," Fallon said. When a bipolar patient's mood improves, his brain activity shifts, too: activity dies down in the lower part of a brain region called the frontal lobe, and flares up in a higher part of that lobe. Amazingly, the very same shift happens when people have bouts of creativity. "There [is] this nexus between these circuits that have to do with bipolar and creativity"".

Besides the other excellent answers, maybe Georg Cantor could fit your description, with some serious caveats. Although the details of his life have been extremely romanticized (e.g. the widespread story that he was driven crazy "after seeing infinity"), he did suffer from chronic depression in the last years of his life, to the point to which he was hospitalized several times.

One of the main contributors to set theory and the back then shocking concept of transfinite numbers. Saying that his discoveries are significant is kind of an understatement.

Kari Mullis, inventor/discoverer of the polymerase chain reaction (PCR) for fast analysis of DNA samples, was into wild LSD rides and espoused near-Scientology-level beliefs in supernatural goings-on.

I might suggest Linus Pauling, but all evidence suggests his nuttiness came well after all his discoveries, so he was "of sound mind" while doing the work which led to his Nobel Prize.

I would say that Henry Cavendish (1731–1810) fits this description. A hugely rich man (at the time of his death he was the largest depositor in the Bank of England) he was also a loner in a huge scale. According to Wikipedia:

He could only speak to one person at a time, and only if the person were known to him and male. He conversed little, always dressed in an old-fashioned suit, and developed no known deep personal attachments outside his family. Cavendish was taciturn and solitary and regarded by many as eccentric. He only communicated with his female servants by notes. By one account, Cavendish had a back staircase added to his house to avoid encountering his housekeeper, because he was especially shy of women.

He was also, of course, one of the greatest scientists of all time. For instance, he discovered that water is not an element, but rather the result of a chemical reaction between oxygen and hydrogen (and he was the one who discovered hydrogen). He was also the first person who computed the bending of light while passing near a star.

I almost hesitate to offer this answer, because I don't want to suggest in any way that its subject was really a 'mad scientist' at all.

But he did show a few of the other characteristics suggested in the question, so his name is arguably not out of place here.

For a time in the 1960s something of a loner in his research, even a maverick, he carried out essentially self-funded work in a historic mansion that he had restored from a burnt-out shell -- used partly for his family home, and partly for a private research laboratory. During part of this period he also maintained a lone scientific position against the claims of researchers from three established research centers who thought they had disproved his theories. Some of his results appeared in two little books for which his scientific competitors made fun of him by calling them the 'thoughts of Chairman M'.

But his story ended well, he turned out to be essentially in the right, and this was eventually well-recognized. His work and results overcame the initially great reluctance from many in his field to accept them; he was awarded a Nobel Prize, and his theory and results have now become part of the orthodox foundation of knowledge in his field.

Readers of this answer who are familiar with biochemistry and cell biology will probably already have recognized that these selected glimpses come from the life and work of Peter Mitchell (1920-1992). His contribution is often not well described in brief accounts (wikipedia for example has "discovery of the mechanism of ATP synthesis" which arguably misses the main point). The issue was not the synthesis of adenosine triphosphate (ATP) altogether, it was about the harnessing and deployment of the chemical energy needed and used for most of the biochemical building processes especially within the mitochondria of eukaryotic cells. This involves a kind of recycling, in which the ATP participates in any of a wide range of biosynthetic reactions, 'spending' its chemical energy by giving up its third phosphate group, and leaving behind the corresponding diphosphate ADP (or occasionally the monophosphate AMP), compound(s) of lower chemical activity. The ADP (or AMP) then has to be 'recharged' by reconversion to ATP. A big question before Mitchell's work was accordingly, how does the cell carry out this 'recharge', to reconvert those lower-energy residual molecules by adding on the extra phosphate groups -- which takes much energy -- so as to replenish the supply of the ATP molecules that then make the chemical energy available to drive biosynthetic processes? Mitchell's correct answer was that the recharging process does not rely solely on a chain of biochemical reactions (which was the previous orthodox view), it involves subcellular membranous compartments within the mitochondria, into which ionic intermediates are pumped against concentration gradients, and then in turn drive the replenishment of the ATP as they pass out - hence 'chemi-osmotic' theory. Further details can safely be left to the references:

Mitchell's 1961 statement of his chemi-osmotic hypothesis;

Mitchell's 1978 Nobel lecture;

Mitchell's 1992 obituary;

A recent summary of the chemiosmotic theory.