Looking for closed-forms of $int_0^{pi/4}ln^2(sin x),dx$ and $int_0^{pi/4}ln^2(cos x),dx$

A more self-contained solution

First note that

$$I=int_0^1frac{ln xln(1+x)}{1+x^2}dx=-sum_{n=0}^infty(-1)^nH_nint_0^1 x^{2n}ln xdx=sum_{n=0}^inftyfrac{(-1)^nH_n}{(2n+1)^2}.$$

We have here

$$int_0^1frac{x^{2n}}{1+x}dx=ln2+H_n-H_{2n}$$

$$=ln(2)+H_n-H_{2n+1}+frac1{2n+1}$$

Multiply both sides by $frac{(-1)^n}{(2n+1)^2}$ then $sum_{n=0}^infty$ we get

$$text{G}ln(2)+sum_{n=0}^inftyfrac{(-1)^nH_n}{(2n+1)^2}-sum_{n=0}^inftyfrac{(-1)^nH_{2n+1}}{(2n+1)^2}+underbrace{sum_{n=0}^inftyfrac{(-1)^n}{(2n+1)^3}}_{pi^3/32}$$

$$=int_0^1frac{1}{1+x}left(sum_{n=0}^inftyfrac{(-1)^nx^{2n}}{(2n+1)^2}right)dx=int_0^1frac{1}{1+x}left(Imfrac{text{Li}_2(ix)}{x}right)dx$$

$$int_0^1frac{1}{1+x}left(Imint_0^1-frac{iln y}{1-ixy}dyright)dx=int_0^1frac{1}{1+x}left(int_0^1-frac{ln y}{1+x^2y^2}dyright)dx$$

$$overset{xy=t}{=}int_0^1int_0^xfrac{ln(x/t)}{x(1+x)(1+t^2)}dtdx=int_0^1frac{1}{1+t^2}left(int_t^1frac{ln(x/t)}{x(1+x)}dxright)dt$$

$$=int_0^1frac{1}{1+t^2}left(text{Li}_2(-t)+frac12ln^2t+ln(2)ln t+frac12zeta(2)right)dt$$

$$=int_0^1frac{text{Li}_2(-t)}{1+t^2}dt+frac{pi^3}{32}-text{G}ln(2)+frac{pi^3}{48}$$

Therefore

$$sum_{n=0}^inftyfrac{(-1)^nH_n}{(2n+1)^2}=sum_{n=0}^inftyfrac{(-1)^nH_{2n+1}}{(2n+1)^2}+int_0^1frac{text{Li}_2(-t)}{1+t^2}dt-2text{G}ln(2)+frac{pi^3}{48}tag1$$

where

$$sum_{n=0}^inftyfrac{(-1)^nH_{2n+1}}{(2n+1)^2}=Imsum_{n=1}^inftyfrac{i^nH_{n}}{n^2}=-frac{pi}{16}ln^2(2)-frac12text{G}ln(2)+Imoperatorname{Li}_3(1+i)tag2$$

and

$$int_0^1frac{text{Li}_2(-t)}{1+t^2}dt=int_0^1frac{1}{1+t^2}left(int_0^1frac{tln x}{1+tx}dxright)dt$$

$$=int_0^1ln xleft(int_0^1frac{t}{(1+t^2)(1+tx)}dtright)dx$$

$$=int_0^1ln xleft(frac{pi}{4}frac{x}{1+x^2}+frac{ln(2)}{2}frac{1}{1+x^2}-frac{ln(1+x)}{1+x^2}right)dx$$

$$=-frac{pi^3}{192}-frac12text{G}ln(2)-int_0^1frac{ln xln(1+x)}{1+x^2}dx$$

Substitute $$int_0^1frac{ln xln(1+x)}{1+x^2}dx=3Imoperatorname{Li}_3(1+i)-frac{5pi^3}{64}-frac{3pi}{16}ln^2(2)-2text{G}ln(2)$$

we get

$$int_0^1frac{text{Li}_2(-x)}{1+x^2}dx=frac{7pi^3}{96}+frac{3pi}{16}ln^2(2)+frac32text{G}ln(2)-3Imoperatorname{Li}_3(1+i)tag3$$

Plug $(2)$ and $(3)$ in $(1)$ we finally get

$$I=sum_{n=0}^inftyfrac{(-1)^nH_n}{(2n+1)^2}=frac{3pi^3}{32}+frac{pi}8ln^2(2)-text{G}ln(2)-2Imoperatorname{Li_3}(1+i)$$

The strategy in this post will be included in another paper.

---

A solution (in large steps) by Cornel Ioan Valean

In my opinion, this is a very magical & powerful way that manages to circumvent the necessity of using the already famous method proposed by Random Variable which I think most posts on MSE use it for such integrals. It's time for a new way to come in place and join the existing one!

In this post, we magically prove that $$int_0^1frac{log xlog(1+x^2)}{1+x^2}textrm{d}x=-frac{pi}{16} log ^2(2) - log (2)G-frac{pi ^3}{64}+2Imbiggr {operatorname{Li}_3left(frac{1+i}{2}right)biggr },$$ by wisely combining a result from the book, (Almost) Impossible Integrals, Sums, and Series, namely the special Fourier series (see eq. 3.284, page 244, and eq. 3.288, page 247), begin{equation} begin{aligned} small sum_{n=1}^{infty} (-1)^{n-1}left(psileft(frac{n+1}{2}right)-psileft(frac{n}{2}right)-frac{1}{n}right)sin(2nx)&small=sum_{n=1}^{infty} (-1)^{n-1}left(int_0^1 t^{n-1}frac{1-t}{1+t} textrm{d}tright)sin(2nx)\ &=-cot(x)log(cos(x)), end{aligned} end{equation} where $displaystyle 0< x<frac{pi}{2}$, and the Cornel's integral,

$$int_0^{pi/2} xfrac{log(cos x)}{sin x}textrm{d}x=2log(2)G-frac{pi}{8}log^2(2)-frac{5}{32}pi^3+4Imleft{text{Li}_3left(frac{1+i}{2}right)right},$$ already calculated in this post How can you approach $int_0^{pi/2} xfrac{ln(cos x)}{sin x}dx$.

Proof: We differentiate both sides of the Fourier series that leads to $$2 sum_{n=1}^{infty} (-1)^{n-1}left(int_0^1 t^{n-1}frac{1-t}{1+t} textrm{d}tright)ncos(2nx)=1+frac{log(cos(x))}{sin^2(x)},$$ and if we multiply both side by $x sin(x)$ and integrate from $x=0$ to $x=pi/2$, we arrive at $$int_0^{pi/2} xsin(x)textrm{d}x+int_0^{pi/2}xfrac{log(cos(x))}{sin(x)}textrm{d}x$$

$$=2 log (2)-1+2 log (2)underbrace{int_0^1 frac{log (x)}{1+x^2}textrm{d}x}_{displaystyle text{Trivial}}+frac{1}{2}underbrace{int_0^1 log (x) log left(1-x^2right)textrm{d}x}_{displaystyle text{Trivial}}$$ $$+frac{1}{2}underbrace{int_0^1frac{log (x) log left(1-x^2right)}{x^2}textrm{d}x}_{displaystyle text{Trivial}}-2underbrace{int_0^1frac{ log (x) log left(1-x^4right)}{1-x^4}textrm{d}x}_{displaystyle text{Beta function in disguise}}$$ $$+2underbrace{int_0^1frac{x^2 log (x) log left(1-x^4right)}{1-x^4}textrm{d}x}_{displaystyle text{Beta function in disguise}}+2color{blue}{int_0 ^1 frac{log (x) log(1+x^2)}{1+x^2}textrm{d}x},$$ from which the desired result follows.

Note the following values of the Beta function forms in disguise:

$$int_0^1 frac{log (x) log left(1-x^4right)}{1-x^4} textrm{d}x=frac{1}{16}int_0^1 frac{log(x)log (1-x)}{ x^{3/4}(1-x) } textrm{d}x$$ $$=frac{7 }{4}zeta (3)+frac{pi ^3}{32}-frac{3}{16}log (2)pi ^2-frac{pi }{4}G-frac{3}{2}log(2)G,$$ and $$int_0^1 frac{x^2log (x) log left(1-x^4right)}{1-x^4} textrm{d}x=frac{1}{16}int_0^1 frac{log(x)log (1-x)}{x^{1/4}(1-x)} textrm{d}x$$ $$=frac{7}{4} zeta (3)+frac{3}{2} log (2)G-frac{1}{4} pi G-frac{3}{16}log(2)pi^2-frac{pi ^3}{32}.$$

A note: this method can also be adjusted to extract other very difficult integrals, which is possible by further exploiting and developing ideas like the ones in the paper A symmetry-related treatment of two fascinating sums of integrals by C.I. Valean.

End of story

my approach to problem $(3)$: begin{align} I&=int_0^1frac{ln xln(1+x^2)}{1+x^2} dx=-2int_0^{pi/4}ln(tan x)ln(cos x) dx\ &=-2int_0^{pi/4}ln(sin x)ln(cos x) dx+2int_0^{pi/4}ln^2(cos x) dx\ &=-int_0^{pi/2}ln(sin x)ln(cos x) dx+2int_0^{pi/4}ln^2(cos x) dx\ &=-left(frac{pi}{2}ln^22-frac{pi^3}{48}right)+2left(frac7{192}pi^3+frac5{16}piln^22-frac12ln2~G-text{Im}operatorname{Li_3}(1+i)right)\ &=frac3{32}pi^3+frac{pi}8ln^22-ln2~G-2text{Im}operatorname{Li_3}(1+i) end{align}

note that we evaluated the first integral using the derivative of beta function and as follows: begin{align} J&=int_0^{pi/2}ln(sin x)ln(cos x) dx=frac18frac{partial^2}{partial{a}partial{b}}beta(a,b)Biggrvert_{ato1/2,~bto1/2}\ &=frac18beta(a,b)left(left(psi(a)-psi(a+b)right)left(psi(b)-psi(a+b)right)-psi^{(1)}(a+b)right)Biggrvert_{ato1/2,~bto1/2}\ &=frac18beta(1/2,1/2)left((psi(1/2)-psi(1))^2-psi^{(1)}(1)right)\ &=frac{pi}8left(4ln^22-zeta(2)right)\ &=frac{pi}2ln^22-frac{pi^3}{48} end{align}

we can prove, using the same strategy of Random Variable, the following equality:

$$int_0^{pi/4}ln^2(cos x) dx=frac7{192}pi^3+frac5{16}piln^22-frac12ln2G-text{Im}operatorname{Li_3}(1+i)$$ proof :
begin{align*} ln(1+e^{2ix}) &= ln (e^{-ix}+e^{ix}) + ln(e^{ix}) \ &= ln(2cos x) + ix end{align*}

squaring both sides and integrating, we get

$$int_0^{pi/4}ln^2(1+e^{2ix}) dx=int_0^{pi/4}(ln(2cos x)+ix)^2 dx$$ equating the real parts on both sides and rearranging the terms, we have:

begin{align*} int_0^{pi/4}ln^2(cos x) dx&=int_0^{pi/4}(x^2-ln^22) dx-2ln2int_0^{pi/4}ln(cos x) dx+text{Re}int_0^{pi/4}ln^2(1+e^{2ix}) dx\ &=frac{pi^3}{192}-frac{pi}{4}ln^22-2ln2left(frac12G-frac{pi}{4}ln2right)+text{Re}int_0^{pi/4}ln^2(1+e^{2ix}) dx\ &=frac{pi^3}{192}+frac{pi}{4}ln^22-ln2G+text{Re}int_0^{pi/4}ln^2(1+e^{2ix}) dx tag{1}\ end{align*} Evaluating the last integral: begin{align*} I&=text{Re}int_0^{pi/4}ln^2(1+e^{2ix}) dx=frac12text{Im}int_1^ifrac{ln^2(1+x)}{x} dx\ &=frac12text{Im}left(ln(-i)ln^2(1+i)+2ln(1+i)operatorname{Li_2}(1+i)-2operatorname{Li_3}(1+i)right)\ &=frac{pi^3}{32}+frac{pi}{16}ln^22+frac12ln2G-text{Im}operatorname{Li_3}(1+i)tag{2} end{align*} Plugging $(2)$ in $(1)$ we get our result.

note that we used: $$ln(-i)=-frac{pi}{2}i$$ $$ln(1+i)=frac12ln2+frac{pi}{4}i$$ $$operatorname{Li_2}(1+i)=frac{pi^2}{16}+left(frac{pi}{4}ln2+Gright)i$$ which give us: $$ln(-i)ln^2(1+i)=frac{pi^2}{8}ln2+left(frac{pi^3}{32}-frac{pi}{8}ln^22right)i$$ $$ln(1+i)operatorname{Li_2}(1+i) =-frac{pi}{4}G-frac{pi^2}{32}ln2+left(frac12ln2G+frac{pi^3}{64}+frac{pi}{8}ln^22right)i$$

$$int_0^fracpi4Big(lnsin xBig)^2~dx~=~dfrac{23}{384}cdotpi^3~+~dfrac9{32}cdotpicdotln^22~+~underbrace{beta(2)}_text{Catalan}cdotdfrac{ln2}2~-~Imbigg[text{Li}_3bigg(dfrac{1+i}2bigg)bigg].$$

$$int_0^fracpi4Big(lncos xBig)^2~dx~=~dfrac{-7}{384}cdotpi^3~+~dfrac7{32}cdotpicdotln^22~-~underbrace{beta(2)}_text{Catalan}cdotdfrac{ln2}2~+~Imbigg[text{Li}_3bigg(dfrac{1+i}2bigg)bigg].$$

---

$$S=int_0^fracpi4Big(lnsin xBig)^2~dx~+~int_0^fracpi4Big(lncos xBig)^2~dx=I+J.$$

But, by a simple change of variable, $t=dfracpi2-x,~J$ can be shown to equal $displaystyleint_fracpi4^fracpi2Big(lnsin xBig)^2~dx$,

in which case $I+J=displaystyleint_0^fracpi2Big(lnsin xBig)^2~dx=dfrac{pi^3}{24}+dfracpi2ln^22.~$ So we know their sum! Now all

that's left to do is to find out their difference, $D=I-J.~$ Then we'll have $I=dfrac{S+D}2$ and

$J=dfrac{S-D}2$.

---

$$D=I-J=int_0^fracpi4Big(lnsin xBig)^2~dx-int_0^fracpi4Big(lncos xBig)^2~dx=int_0^fracpi4Big(ln^2sin x-ln^2cos xBig)~dx$$

$$=int_0^fracpi4Big(lnsin x-lncos xBig)~Big(lnsin x+lncos xBig)~dx=int_0^fracpi4lnfrac{sin x}{cos x}~lnbig(sin x~cos xbig)~dx=$$

$$=int_0^fracpi4lntan xcdotlnfrac{sin2x}2~dx=frac12int_0^fracpi2lntanfrac x2cdotlnfrac{sin x}2~dx=int_0^1ln tcdotlnfrac t{1+t^2}cdotfrac{dt}{1+t^2}$$

where the last expression was obtained by using the famous Weierstrass substitution, $t=tandfrac x2$

$$=int_0^1frac{ln tcdotBig[ln t-ln(1+t^2)Big]}{1+t^2}dt~=~int_0^1frac{ln^2t}{1+t^2}dt~-~int_0^1frac{ln t~lnbig(1+t^2big)}{1+t^2}dt~=~frac{pi^3}{16}-K,$$

where $~K=2~Imbigg[text{Li}_3bigg(dfrac{1+i}2bigg)bigg]-dfrac{pi^3}{64}-dfracpi{16}ln^22-underbrace{beta(2)}_text{Catalan}ln2.~$ It follows then that our two

definite integrals possess a closed form expression if and only if $~text{Li}_3bigg(dfrac{1+i}2bigg)$ has one as well. As

an aside, $~Rebigg[text{Li}_3bigg(dfrac{1+i}2bigg)bigg]=dfrac{ln^32}{48}-dfrac5{192}~pi^2~ln2+dfrac{35}{64}~zeta(3).~$ Also, $~K=displaystylesum_{n=1}^inftyfrac{(-1)^n~H_n}{(2n+1)^2}$.

By setting $x=arctan t$ we have: $$int_{0}^{pi/4}log^2(cos x),dx = frac{1}{4}int_{0}^{1}frac{log^2(1+t^2)}{1+t^2}.$$ Attack plan: get the Taylor series of $log^2(1+t^2)$ and integrate it termwise.

Since $$-log(1-z)=sum_{n=1}^{+infty}frac{z^n}{n}$$ it follows that $$[z^n]log^2(1-z)=sum_{k=1}^{n-1}frac{1}{k(n-k)}=2frac{H_{n-1}}{n},$$ $$log^2(1+t^2)=sum_{n=2}^{+infty}2frac{H_{n-1}}{n}(-1)^n t^{2n}.tag{1}$$ If now we set $$mathcal{J}_m = int_{0}^{1}frac{t^{2m}}{t^2+1},dt $$ we have $mathcal{J}_0=frac{pi}{4}$ and $mathcal{J}_{m+1}+mathcal{J}_m = frac{1}{2m+1}$, hence: $$mathcal{J}_m = (mathcal{J}_m+mathcal{J}_{m-1})-(mathcal{J}_{m-1}+mathcal{J}_{m-2})+ldotspm(mathcal{J}_1+mathcal{J}_0)mpmathcal{J}_0,$$ $$mathcal{J}_m = sum_{j=0}^{m-1}frac{(-1)^j}{(2m-2j-1)}+(-1)^mfrac{pi}{4}=(-1)^m sum_{jgeq m}frac{(-1)^j}{2j+1}.tag{2}$$ From $(1)$ and $(2)$ it follows that: $$int_{0}^{pi/4}log^2(cos x),dx=frac{1}{2}sum_{n=2}^{+infty}frac{H_{n-1}}{n}sum_{rgeq n}frac{(-1)^r}{2r+1},tag{3}$$ and summation by parts gives:

$$int_{0}^{pi/4}log^2(cos x),dx=frac{1}{4}sum_{n=2}^{+infty}(H_n^2-H_n^{(2)})frac{(-1)^n}{2n+1}.tag{4}$$

UPDATE: the question is now set in an answer to another question. This site (many thanks to @gammatester) is devoted to the evaluation of sums like the one appearing in the RHS of $(4)$. Through Euler-Landen's identity (see the line below $(608)$ in the linked site) it is not too much difficult to see that the RHS of $(4)$ depends on $operatorname{Li}_3left(frac{1+i}{2}right)$ as stated in the @Lucian's answer.