Deriving Klein-Gordon from Hamilton's equations for fields using functional derivatives

Your evaluation of the functional derivative in your third equation is wrong. It should read $$ dot phi(x)= frac{delta H}{delta pi(x)}= pi(x) $$ There is no integral. The rest of what you have wirtten has the same problem. Read up on how to define functional derivatives.

Let's actually evaluate the functional derivatives: $$ delta H = int d^dxleft (pi(x) delta pi(x) + nabla phi(x)cdot nabla delta phi(x) + m phi(x)delta phi(x)right) = int d^dxleft( pi(x) delta pi(x) - left((nabla^2 phi(x)right) delta phi(x) + m phi(x)delta phi(x)right). $$ We have integrated by parts and discarded the integrated out term. Compare with the definition of the functional derivative via the variation $$ delta H = int d^dxleft( frac{delta H}{delta pi(x)} delta pi(x) + frac {delta H}{delta phi(x)} delta phi(x) right) $$ which is the generalization of $$ delta H(phi_1,phi_2,ldots phi_n)= sum_{n=1}^n frac{partial f}{partial phi_n} delta phi_n $$ to the case were the discrete "$i$"s are replaced by the continuous label $x$ and the sum is replaced by an integral and $$ frac{partial f}{partial phi_n}to frac{delta f}{delta phi(x)} $$ Mking the comparison read off that $$ frac{delta H}{delta pi(x)} = pi(x) $$ and $$ frac {delta H}{delta phi(x)}=-nabla^2 phi(x)+m^2phi(x). $$

For a more basic example consider a one dimensional calculus-of-variations functional $$ F[y]= int_a^b f(y,y',y'') dx $$ under variations $y(x)to y(x)+delta y(x)$ with $delta y(a)=delta y(b)= delta y'(a)=delta y'(b)=0$ so that we can freely integrate by parts, then $$ frac{delta F}{delta y(x)} = frac{partial f}{partial y(x)} - frac {d}{dx}left( frac{partial f}{partial y'(x)}right)+ frac {d^2}{dx^2}left( frac{partial f}{partial y''(x)}right) $$ in which expression the ordinary partials treat $y(x)$, $y'(x)$ $y''(x)$ as independent variables.

OP's question is not about the Klein-Gordon equation per se, but rather about the very definition of the functional/variational derivative (FD). OP's eq. (4) is just plain wrong.

To complicate matters, OP's eqs. (8) refers to a misleading 'same-space' FD notation, which when applied naively, leads to a mismatch of integral signs in OP's eqs. (3), (7) & (9).

For more details, see e.g. this, this & this related Phys.SE posts.

You are most likely confused because methods from functional calculus are often obscured by hacks to make the calculations easier to understands but it also hides a lot of the intuition. So starting with functionals. Functionals are objects that take in functions and spit out a real (complex) number. They are denoted by square brackets to indicate that they are functionals. The simplest functional is probably a plain integral: $$F[varphi]=int dx,varphi(x)$$ Functional calculus extends the idea of taking derivatives to functionals. This allows you to do things like finding the function that minimizes the functional given some constraint (like the action!). Let's say you want to do this naively by taking the partial derivative. I will take $F$ to be a sum to allow us to take the partial derivative. $$F[varphi]=sum_ivarphi(x_i)Delta x$$ Here $x_i$ are evenly spaced x-values from some interval $[a,b]$. Now we can take the partial derivative if we consider each $varphi(x_i)$ as an independent parameter $$frac{partial F[varphi]}{partialvarphi(x_j)}=frac{partial}{partialvarphi(x_j)}sum_ivarphi(x_i)Delta x=delta_{ij}Delta x$$ You might see the issue here. The result depends on $Delta x$ and if we transform this to an integral we always get zero. Now if we define the functional derivative as $$frac{delta F[varphi]}{deltavarphi(y)}=lim_{epsilonrightarrow 0}frac{F[varphi(x)+epsilon delta(x-y)]-F[varphi(x)]}{epsilon}$$ then we get a definition that doesn't vanish. You can show for yourself that $$frac{delta }{deltapi(y)}int dx,pi(x)^2=2pi(y).$$ In general the functional derivative of an integral will give the derivative of the integrand. This is what you try to state in (4). Now to get back to your problem define $$F[varphi,nablavarphi]=int d^3xleft[(nablavarphi(x))^2+m^2varphi(x)^2right]$$ We could vary $F$ with respect to only its first argument but we are interested the case where the second argument is dependent on $varphi$, similar to a total derivative. Since $nabladelta(x)$ is kind of a nasty object I will use a better, more general definition of the functional derivative that doesn't use delta functions. $$int dx,frac{delta F}{deltavarphi}(x)rho(x)=lim_{epsilonrightarrow 0}frac{F[varphi+epsilon rho]-F[varphi]}{epsilon}tag{1}label{definition}$$ With $rho$ some arbitrary test function. Let's name $rho$ to be $rho(x)=deltavarphi(x)$ to make the notation nicer. To calculate $frac{delta F}{deltavarphi}$ we need to evaluate the right hand side of $$int dx,frac{delta F}{deltavarphi}(x)deltavarphi(x)=lim_{epsilonrightarrow 0}frac{F[varphi+epsilondeltavarphi,nablavarphi+epsilonnabladeltavarphi]-F[varphi,nablavarphi]}{epsilon}$$ After some work you can show that this becomes $$lim_{epsilonrightarrow 0}int d^3xleft[2nablavarphinabladeltavarphi+epsilon(nabladeltavarphi)^2+2m^2varphideltavarphi+epsilon m^2deltavarphi^2right] =int d^3xleft[2nablavarphinabladeltavarphi+2m^2varphideltavarphiright]$$ To match the left hand side of (1) we must use partial integration to remove the $nabla$ from $deltavarphi$. This gives us $$int d^3xleft[-2(nabla^2varphi)deltavarphi+2m^2varphideltavarphiright]$$ Now reading of (1) gives us $$frac{delta F}{deltavarphi}=-2nabla^2varphi+2m^2varphi$$

---

Final note: if you plug in $rho(x)=delta(x-y)$ in (1) you get the first definition of the functional derivative.