Newton's evaluation of $1 + frac{1}{3} - frac{1}{5} - frac{1}{7} + frac{1}{9} + frac{1}{11} - cdots$

Nick Mackinnon gives what appears to be the story in an article that appeared in the Mathematical Gazette in March 1992 (Vol. 76, No. 475), entitled "Newton's Teaser." He writes that Newton conjured up the poser, in fits and starts, in response to Leibniz's series

$$ 1 - frac13 + frac15 - frac17 + cdots = fracpi4 $$

The other answers here are not off the mark, really. Newton was able to evaluate—determine areas for—among other things, expressions of the form

$$ int frac{dx^{eta-1}}{e+fx^eta+gx^{2eta}} $$

(Actually, he used $z$ instead of $x$, but I'll use the more usual $x$, because that's what Mackinnon does in most of his exposition other than direct cites of Newton.) He pointed out that by letting $eta = 1$, $e = g = 1$, and $f = 0$, then the result can be used to evaluate Leibniz's series, and he further suggested that setting $eta = 1$, $e = g = 1$, and $f^2 = 2eg$ (i.e., $f = sqrt2$) enables the evaluation of the series in question:

$$ 1 + frac13 - frac15 - frac17 + frac19 + frac{1}{11} - cdots $$

Following Newton's suggestion, we have

begin{align} int_{x=-1}^1 frac{dx}{1+sqrt2x+x^2} & = int_{x=-1}^1 frac{dx}{left(x+frac{1}{sqrt2}right)^2 +left(frac{1}{sqrt2}right)^2} \ & = left. sqrt2 arctan left( frac{x+frac{1}{sqrt2}}{sqrt2}right) right]_{x=-1}^1 \ & = sqrt2 left[ arctan left( frac12 + frac{1}{sqrt2} right) - arctan left( frac12 - frac{1}{sqrt2} right) right] \ & = sqrt2 left( frac{3pi}{8} + fracpi8 right) \ & = frac{pi}{sqrt2} end{align}

Newton apparently recorded in his worksheets the factorization

$$ 1+x^4 = (1+sqrt2x+x^2)(1-sqrt2x+x^2) $$

so evidently he means to evaluate the integral alternatively as

begin{align} require{cancel} int_{x=-1}^1 frac{dx}{1+sqrt2x+x^2} & = int_{x=-1}^1 frac{1-sqrt2x+x^2}{1+x^4} , dx \ & = int_{x=-1}^1 frac{1+x^2}{1+x^4} , dx - cancel{int_{x=-1}^1 frac{sqrt2x}{1+x^4} , dx} qquad text{because $sqrt2x$ is odd} \ & = 2int_{x=0}^1 frac{1+x^2}{1+x^4} , dx qquad text{because this is even} end{align}

Combining these gives us

begin{align} frac{pi}{2sqrt2} & = int_{x=0}^1 frac{1+x^2}{1+x^4} , dx \ & = int_{x=0}^1 frac{dx}{1+x^4} + int_{x=0}^1 frac{x^2,dx}{1+x^4} \ & = int_{x=0}^1 1-x^4+x^8-x^{12}+cdots , dx + int_{x=0}^1 x^2-x^6+x^{10}-x^{14}+cdots , dx \ & = left. x-frac{x^5}{5}+frac{x^9}{9} -frac{x^{13}}{13}+cdots right]_{x=0}^1 + left. frac{x^3}{3}-frac{x^7}{7}+frac{x^{11}}{11} -frac{x^{15}}{15}+cdots right]_{x=0}^1 \ & = 1+frac13-frac15-frac17+frac19+frac{1}{11}-frac{1}{13}-frac{1}{15} + cdots end{align}

Mackinnon adduces some circumstantial evidence that strongly suggests Leibniz never cracked Newton's little chestnut.

Looking at the text mentioned, we see something interesting on page $156$, in note $(48)$. Transcribing it (you can find a screencap of the text here):

On observing that $$1 + frac 1 3 - frac 1 5 - frac 1 7 + frac 1 9 + text{etc.} = int_0^1 frac{1+x^2}{1+x^4}dx$$ by expanding the integrand as an ascending series in $x$, we may suppose that Newton obtained his result by integrating the identity $$frac{1}{e+fz+gz^2} + frac{1}{e-fz+gz^2} = frac{2e + 2gz^2}{e^2 + g^2 z^4} text{ (on putting } 2eg = f^2 text{)}$$ in two ways. Put $z sqrt g = x sqrt e$ and $-xsqrt e$, respectively, in the two fractions on the left, and integrate from $0$ to $1$ with regard to $x$. Then, on combining the terms on the left, the identity gives $$frac 1 2 int_{-1}^1 frac{dx}{1 + sqrt 2 x + x^2} = int_0^1 frac{1+x^2}{1+x^4}dx$$ By putting $1 + x sqrt 2 = tan theta$, and integrating $theta$ from $- frac 1 8 pi$ to $frac 3 8 pi$, the left side gives $pi/2 sqrt 2$, and the right is equal to the series. (Cf. Hofmann, p. $175$.) The series, when written $1 + (frac 1 3 - frac 1 5) - frac 1 7 - frac 1 9) + text{etc.}$, leads to the next result in the letter.

Granted, I'm not sure if this is how Newton actually calculated it, and I haven't actually read this book thoroughly enough to say if it's what you're looking for (I just skimmed until I found what seemed relevant). Hopefully it's enlightening though.

Although the question appears to be about how Newton historically did it, I'll convert a popular comment to an answer showing how techniques from his era, similar to those that handle the Gregory series, evaluate the series above: $$begin{align}sum_{nge0}left(frac{1}{8n+1}+frac{1}{8n+3}-frac{1}{8n+5}-frac{1}{8n+7}right)&=sum_{nge0}int_{0}^{1}x^{8n}left(1+x^{2}right)left(1-x^{4}right)dx\&=int_{0}^{1}frac{1+x^{2}}{1+x^{4}}dx\&=int_{0}^{1}frac{1+x^{2}}{left(1-xsqrt{2}+x^{2}right)left(1+xsqrt{2}+x^{2}right)}dx\&=frac{1}{2}sum_{pm}int_{0}^{1}frac{dx}{1pm xsqrt{2}+x^{2}}\&=frac{1}{sqrt{2}}sum_{pm}left[arctanleft(xsqrt{2}pm1right)right]_{0}^{1}\&=frac{arctanleft(sqrt{2}+1right)+arctanleft(sqrt{2}-1right)}{sqrt{2}}\&=frac{pi}{2sqrt{2}}.end{align}$$