Commutator of Lorentz transformations on a scalar field

I’d like to calculate the commutator of the Lorentz transformations on a scalar field and I’ve already done it years ago, but recently I suspect my calculation.
Let me fix some notations.
The representation matrix of the Lorentz transformation on a scalar field is given by
begin{align}
phi(x) rightarrow phi'(x) = U(Lambda)phi(x) = phi(Lambda x)
end{align}
with $phi'(x’) = phi(x)$ (hence $phi(x)$ is a scalar field) and
begin{align}
U(Lambda)=e^{-frac{1}{2} lambda^{rho sigma}L_{[rho sigma]}}.
end{align}
Here, the Lorentz transformation on position is
begin{align}
x^{mu} = Lambda^{mu}_{nu} x’^{nu}
end{align}
and the generator $L_{[rho sigma]}$ is defined as
begin{align}
L_{[rho sigma]} = x_{rho} partial_{sigma} – x_{sigma} partial_{rho}.
end{align}
Actually, I use the notation in the [textbook][1] of supergravity written by Freedman and Van Proeyen.

Then, let me focus on the commutator $[U(Lambda_2), U(Lambda_1)]$.
My thought is just expand $U(Lambda)$ and calculate the commutator. This results that $[U(Lambda_2), U(Lambda_1)]$ becomes a Lorentz transformation with parameters $[lambda_1, lambda_2]$. More concretely,
begin{align}
[U(Lambda_2), U(Lambda_1)] &= left[left(1 – frac{1}{2}lambda_2^{rho sigma}L_{[rho sigma]} + frac{1}{4}lambda_2^{rho sigma}L_{[rho sigma]}lambda_2^{mu nu}L_{[mu nu]}right),left(1 – frac{1}{2}lambda_1^{rho sigma}L_{[rho sigma]} + frac{1}{4}lambda_1^{rho sigma}L_{[rho sigma]}lambda_1^{mu nu}L_{[mu nu]}right)right] + {cal O(lambda^3)}
&= frac{1}{4} [lambda_2^{rho sigma}, lambda_1^{mu nu}]L_{[rho sigma]}L_{[mu nu]},
end{align}
where is already expected since we construct the finite Lorentz transformation via the exponential mapping. It is enough to consider the following term:
begin{align}
lambda_2^{rho sigma} lambda_{1}^{mu nu} L_{[rho sigma]}L_{[mu nu]} = lambda_2^{rho sigma} lambda_{1}^{mu nu} (x_rho partial_sigma -x_{sigma}partial_{rho})(x_mu partial_nu -x_{nu}partial_{mu}).
end{align}
For later convenience, I divide it into the following two piece:

• "non-interference" term
  begin{align}
  lambda_2^{rho sigma} lambda_{1}^{mu nu} (x_rho x_mu partial_{sigma} partial_{nu} – x_rho x_nu partial_{sigma} partial_{mu} – x_{sigma}x_{mu} partial_{sigma}partial_{nu} + x_{sigma}x_{nu} partial_{sigma}partial_{mu}) = 4 lambda_2^{mu nu} lambda_{1}^{rho sigma} x_{mu}x_{rho}partial_{nu}partial_{sigma},
  end{align}
  where this "non-interference" term vanishes in the commutator since the term is symmetric under $1leftrightarrow 2$ (i.e. $mu nu rightarrow rho sigma$).
• "interference" term
  begin{align}
  lambda_2^{rho sigma} lambda_{1}^{mu nu} (x_rho delta_{sigma mu} partial_{nu} – x_rho delta_{sigma nu} partial_{mu} – x_{sigma}partial_{rho mu} partial_{nu} + x_{sigma} delta_{rho nu} partial_{mu}) = 4 {lambda_{1}^{nu}}_{mu} {lambda_{2}^{mu}}_{rho} x^rho partial_nu,
  end{align}
  where this leads to the term ${[lambda_{1}, lambda_{2}]^nu}_{rho}x^rho partial_{nu}$ via the commutator.

Since
begin{align}
-frac{1}{2} lambda^{mu nu}L_{[mu nu]} phi(x) = {lambda^{mu}}_{nu} x^nu partial_{mu} phi(x),
end{align}
the latter "interference" term seems to give the correct result that states $[U(Lambda_2), U(Lambda_1)]$ becomes a Lorentz transformation with parameters $[lambda_1, lambda_2]$.

This result is also shown in the supergravity text book (as Problem 1.7 in page 15), but the following notice is there:

Note that the second transformation acts on $phi^i(x)$, and not on the $x$-dependent factor

which means there is no contribution from the "interference" term by the definition of the sequential tranformations. However, the "non-intereference" term vanishes in my calculation and indeed its includes second derivatives of the field, though I need the first derivative term.

As I mentioned, recently I found this basic sentence and confused with it so much.
Please clarify the problem in my thought.

[1]: D. Z. Freedman and A. Van Proeyen,
"Supergravity", https://www.cambridge.org/core/books/supergravity/B7EEC3E37A39AB6E6625850857B96AA7.