Solving a HJB with additional constraints on control and state variables

I am trying to solve a Hamiltonian-Jacobi-Bellman equation with additional constraints on the state and control variables, but I am a bit confused on how to do that.

In Intrilligator 2002, it is written that

"Dynamic programming can also be used to treat the constrained
calculus of variations (…). For example, for the isoperimetric
problem, where the constraint is $$int_{t_{0}}^{t_{1}} G(mathbf{x},
dot{mathbf{x}}, t)=c$$, Bellman’s equation takes the form
$$-frac{partial J^{*}}{partial t}=max_{{dot{x}}}left[I(mathbf{x}, dot{mathbf{x}}, t)+frac{partial J^{*}}{partial mathbf{x}} dot{mathbf{x}}+frac{partial
J^{*}}{partial c} G(mathbf{x}, dot{mathbf{x}}, t)right]$$ which
yields the same conditions as the calculus of variations formulation
since the Lagrange multiplier is $$y=frac{partial J^{*}}{partial
c}$$ i.e., the variation in the optimal value of the functional with
respect to the constant c of the constraint."

However, I am not sure how to apply this formulation to the following maximization problem:

1. A firm maximizes its discounted profits $max_{i_t} int_0^{infty} e^{-rt}pi_t dt$ where $pi_t=f(q_t)-i_t-tau q_t$. Assume $f(q_t)$ is increasing and concave in $q_t$. The two control variables are $i_t$, the investment decision, and $q_t$, the level of installed capital used
2. This maximization program is subject to the following constraints

• A law of motion of capital $dot{k} = i_t – delta k_t$ where $delta$
  is a parameter defining the rate of depreciation of capital.
• Investment is irreversible $i_tgeq 0$
• A physical constraint on $q_t$: $q_tleq k_t$, which simply states that the firm cannot use more capital that it has installed.

I know how to solve this problem using a Hamiltonian, but I would like to find how to do it using dynamic programming, and thus a HJB.

With a current-value Hamiltonian, I would write
begin{align*}
H = f(q_t)-i_t-tau q_t + lambda_t(i_t-delta k_t)+ psi_t i_t +beta_t (k_t-q_t)
end{align*}
Which would yield the following FOC:
begin{align*}
-1+lambda_t + psi_t = 0 \
f'(q_t)-tau – beta_t = 0 \
-delta lambda_t + beta_t = rlambda_t – dot{lambda}_t
end{align*}

However, I wonder how I could solve this problem using dynamic programming and a Hamilton-Jacobi-Equation. Using the formulation of Intrilligator, I would include the additional constraints into the HJB

$$ rV = max_{q,i} left{f(q_t)-i_t-tau q_t + V_{k}(i_t-delta k_t) + V_{i}i_t+V_{q}(k_t-q_t) right} $$

But I am not sure if this is right (I write $V_k$ for the derivative of the value function with respect to $k$).

Following Walde 2012, I take the FOCs
begin{align*}
-1+V_k+V_i = 0 \
f'(q_t)-tau – V_q=0
end{align*}

However, I have more difficulty with the envelope conditions. Is it ok to write
begin{align*}
rV_{k} =V_{k,k}(i_t-delta k_t) – delta V_k +V_{i,k} i_t + V_{q,k}(k_t-q_t)+V_{q}?
end{align*}
Then, I would write
$$ dot{V_k}=V_{k,k}(i_t-delta k_t)$$
and substituting using the envelope condition to find
$$ dot{V_k}=V_k(r+delta)- V_{i,k} i_t – V_{q,k}(k_t-q_t)-V_q$$

But I am not sure that this is the desired result because of those extra cross-derivatives $V_{i,k}$ and $V_{q,k}$…

My questions are:

1. Do you need to include the additional constraints in the HJB, and how do you do it?
2. How do you take into account those additional constraints in the envelope condition?
3. Is it ok not to include the constraints into the HJB but to solve the maximization problem with a Lagrangian? That is, to write only $rV=max_{q,i} left{f(q)-i_t-tau q_t + V_k (i_t-delta k_t)right}$ but to solve this maximization with the following $$ L =f(q)-i_t-tau q_t + V_k (i_t-delta k_t)+psi_t i_t+beta_t(k_t-q_t) $$
   Thank you for your help!

Note: I posted this question originally on mathSE, but it didn’t receive a lot of attention and I thought it might be more appropriated here.