How can I prove that these definitions of curl are equivalent?

Let's formulate the definition of curl slightly more precisely in the form of a definition/theorem. I'll also not use boldface objects, simply for ease of typing

Definition/Theorem.

Let $Asubset Bbb{R}^3$ be open, $F: A to Bbb{R}^3$ be $C^1$. Then, there is a unique continuous function $H:A to Bbb{R}^3$, such that for every $pin A$, for every $epsilon > 0$, for every "nice" surface $Ssubset A$, there is an open neighbourhood $U$ of $p$ in $S$ such that for every "nice" oriented surface $M$ with $pin Msubset U$, with outward normal vector field $n(cdot)$ (which is simply the restriction of the outward normal of $S$ to $M$), boundary $partial M$ and tangent vector field $tau(cdot)$ on $partial M$, we have: begin{align} left|dfrac{1}{|M|}int_{partial M}langle F, taurangle, dl - langle H(p), n(p)rangleright| < epsilontag{1} end{align} (this is the more precise meaning of the limit you're talking about) In this case, because $H$ is unique, we can give it the name $text{curl}(F)$. In fact, we can show that begin{align} H = left(frac{partial F_z}{partial y} - frac{partial F_y}{partial z}right) boldsymbol{hatimath} + left(frac{partial F_x}{partial z} - frac{partial F_z}{partial x} right) boldsymbol{hatjmath} + left(frac{partial F_y}{partial x} - frac{partial F_x}{partial y} right) boldsymbol{hat k} tag{2} end{align}

In the above paragraph, "nice oriented surface" means nice enough so that Stoke's theorem can be applied; for example a smooth two-dimensional, oriented manifold-with-boundary (or however much you wanna weaken the hypothesis... because each book presents it with varying levels of generality... just as long as Stoke's theorem can be applied).

Note that for the above definition of curl to make sense, we have to first show the existence and uniqueness of such a vector field $H$. We shall start by showing the uniqueness of $H$. So, we assume such a $H$ exists, and then prove its components are determined according to the formula $(2)$; this will complete the proof of uniqueness.

---

Proof of Uniqueness

I'll carry out the computation in detail for proving $H_x = dfrac{partial F_z}{partial y} - dfrac{partial F_y}{partial z}$, and leave the other two to you (it's simply a matter of renaming $x,y,z$). We prove this equality pointwise of course. So, fix a point $p in A$; then for any $delta > 0$ such that the closed cube $C_{p,delta} = p + [-delta,delta]^3$ (which is the closed cube centered at $p$ of sidelength $2delta$) lies entirely inside $A$ (note that since $A$ is open, there are infinitely many such $delta>0$), we define $M^{delta} := {p_1}times [p_2-delta, p_2 + delta]times [p_3-delta, p_3 + delta]$. This is a piece of a plane which we shall orient so that it has a constant outward normal vector field $n = e_1 equiv boldsymbol{i}$. Now, we calculate: note that $partial M^{delta}$ has $4$-pieces, and the unit tangent vector along these boundary paths is constant, so (if you're very careful with signs... which I hope I didn't make any sign mistakes), we get begin{align} dfrac{1}{|M^{delta}|} int_{partial M^{delta}}langle F, tau rangle, dl &= dfrac{1}{4delta^2} bigg[int_{-delta}^{delta} F_2(p_1, p_2 + y, p_3-delta) - F_2(p_1, p_2 + y, p_3+delta) , dybigg]\ &+dfrac{1}{4delta^2}bigg[ int_{-delta}^{delta} F_3(p_1, p_2+delta, p_3+z) - F_3(p_1, p_2-delta, p_3+z), dzbigg] end{align} For each term, we apply the mean-value theorem for integrals (which we can use since everything is continuous), to obtain some $eta in [p_2-delta, p_2+delta]$ and some $zetain [p_3-delta, p_3+delta]$ such that

begin{align} dfrac{1}{|M^{delta}|} int_{partial M^{delta}}langle F, tau rangle, dl &= dfrac{1}{4delta^2}bigg[2delta cdot F_2(p_1, eta, p_3-delta) - 2delta cdot F_2(p_1, eta, p_3+delta)bigg] \ &+ dfrac{1}{4delta^2}bigg[2delta cdot F_3(p_1, p_2+delta, zeta) - 2delta cdot F_3(p_1, p_2-delta, zeta)bigg] \\ &=dfrac{F_3(p_1, p_2+delta, zeta) - F_3(p_1, p_2-delta, zeta)}{2delta} -dfrac{F_2(p_1, eta, p_3+delta) - F_2(p_1, eta, p_3-delta)}{2delta} \ &= dfrac{partial F_3}{partial y}(p_1, alpha, zeta) - dfrac{partial F_2}{partial z}(p_1, eta, beta), end{align} for some $alphain [p_2-delta, p_2+delta], betain [p_3-delta, p_3+delta]$, using the mean-value theorem for derivatives (which can certainly be applied since we assumed $F$ is $C^1$).

Quick summary: what we showed so far is that for every $pin A$ and every $delta>0$ such that the cube $C_{p,delta}$ lies inside $A$, if we define $M^{delta}$ as above to be the plane centered at $p$ with normal pointing in $e_1$ direction, then, there exist points $a_{p,delta},b_{p,delta} in M^{delta} subset C_{p,delta}$ inside the surface (in particular inside the cube) such that

begin{align} dfrac{1}{|M^{delta}|}int_{partial M^{delta}}langle F, taurangle , dl &= dfrac{partial F_3}{partial y}(a_{p,delta}) - dfrac{partial F_2}{partial z}(b_{p,delta}) end{align} From here, it is a simple matter of using the continuity of partial derivatives. Here's the full $epsilon,delta$ argument to finish it off: let $pin A$ and $epsilon> 0$ be arbitrary. By our hypothesis of $(1)$, there is an open $U$ such that blablabla. Now, for this given $epsilon > 0$, let's choose $delta > 0$ small enough so that

• the cube $C_{p,delta}$ lies inside $U$
• the $delta$ "works" for the continuity of $dfrac{partial F_3}{partial y}$ and $dfrac{partial F_2}{partial z}$ at the point $p$

(so really we have to take a minimum of several $delta$'s). Then, we choose the oriented plane $M^{delta}$ as defined above (this plane lies inside $U$ by construction, because of how small $delta$ is). begin{align} left|left(dfrac{partial F_3}{partial y}(p) - dfrac{partial F_2}{partial z}(p)right) - H_1(p)right| &= left|left(dfrac{partial F_3}{partial y}(p) - dfrac{partial F_2}{partial z}(p)right) - langle H(p), n(p)rangleright| \\ &leq left|dfrac{partial F_3}{partial y}(p) - dfrac{partial F_3}{partial y}(a_{p,delta})right| + left|-dfrac{partial F_2}{partial z}(p) + dfrac{partial F_2}{partial z}(b_{p,delta})right|\ &+ left|left(dfrac{partial F_3}{partial y}(a_{p,delta}) - dfrac{partial F_2}{partial z}(b_{p,delta})right) - dfrac{1}{|M^{delta}|}int_{partial M^{delta}}langle F, taurangle, dl right| \ &+ left|dfrac{1}{|M^{delta}|}int_{partial M^{delta}}langle F, taurangle, dl - langle H(p), n(p) rangle right| \\ &leq 4epsilon end{align} (each absolute value is $leq epsilon$ based on everything I've said above, and by the choice of $delta$). Since the point $p$ and $epsilon > 0$ are arbitrary, the inequality above shows that begin{align} H_1 &= dfrac{partial F_3}{partial y} - dfrac{partial F_2}{partial z} end{align}

As a recap of the proof idea: choose a small plane $M^{delta}$ with outward normal pointing along $e_1$; it is the flatness of the plane (which is inherently adapted to cartesian coordinates), along with the ease of boundary parametrization which makes the resulting line integral easy to calculate. Then, simply calculate everything, and use the mean-value theorems for derivatives and integrals (this is one way to fill in the gaps for the arugments you typically see in physics texts, which say "let's keep things up to first order only" and where they use $approx$ everywhere); finally we complete it off with a standard $epsilon,delta$ continuity argument.

Some remarks is that for this argument to work I've had to assume $F$ is $C^1$, so that I can apply the mean-value theorems twice, and finally finish it off with a continuity argument. I'm not sure if this proof can be strengthened so that we only have to assume $F$ is differentiable (rather than $C^1$).

---

Proof of Converse

Now we show the existence of such a vector field $H$; for this we'll show that $text{curl}F$, defined by equation $(2)$ satisfies the conditions of $(1)$. Like I mentioned in the comments, I'm not sure how to do this without already appealing to Stokes theorem. With Stokes' theorem, this becomes quite simple.

Let $pin A$, $epsilon > 0$ and let $Ssubset A$ be any "nice surface". Since $langletext{curl}(F), nrangle$ is a continuous function on $S$, there is an open neighbourhood $U$ around $p$ in $S$ such that for all $qin U$, begin{align} left|langle text{curl}F(q), n(q)rangle - langle text{curl}F(p), n(p)rangleright| & leq epsilon end{align} Now, for any "nice surface" $Msubset U$ (with unit normal being the restriction of the one already on $S$), we have by Stokes theorem: begin{align} left|dfrac{1}{|M|}int_{partial M}langle F,taurangle , dl - langle text{curl} F(p), n(p) rangleright| &= dfrac{1}{|M|}left|int_M langle text{curl}F, nrangle , dA - int_M langle text{curl} F(p), n(p) rangle, dAright| \ &leq dfrac{1}{|M|}int_M left|langletext{curl }F, nrangle - langletext{curl }F(p), n(p)rangle right| , dA \ & leq dfrac{1}{|M|} epsilon |M| \ &= epsilon. end{align} This completes the proof of existence.

The basic geometric object here is that of a (piecewise) smooth closed curve $c$ bounding a smooth 2D surface $S$ in ${Bbb R}^3$. Let the curve be parametrized as $r(t)=(x(t),y(t),z(t))$, $0leq tleq 1$, with $r(0)=r(1)$ since closed.

Projecting $S$ to e.g. the $xy$-plane yields a 2D domain which has an area denoted $S_z$. Choosing a sign convention we may calculate the area as $$ S_z = oint_c x ,dy = int_0^1 x(t) frac{dy}{dt} dt .$$ It also equals $-oint y dx$ since $d(xy)=x dy + y dx$ is a differential of a smooth function, whence integrates to zero along $c$. The area-vector associated with $S$ (or $c$) is then the vector consisting of the areas (with fixed sign conventions) of the 3 natural projections. It is given by: $$ vec{S}(c) = left( oint_c y , dz, oint_c z, dx, oint_c x,dz right) = left( -oint_c z , dy, -oint_c x, dz, -oint_c z,dx right).$$ We also write is as $vec{S}(c) = vec{n} |S|$ with $vec{n}$ a unit vector. One may think of $vec{n}$ as a normal to the surface $S$ (certainly true if $c$ and $S$ lies in a 2D hyperplane). Now, let $vec{F}=(F_x,F_y,F_z)$ be a smooth vector field that we wish to integrate along $c$. Thus, we want to calculate $oint_c vec{F}cdot dvec{r}$ and relate the result to the above area-vector $vec{S}(c)$. For this we need some kind of approximation. So assume first that $vec{F}$ is a linear function of the form $vec{F}=(0,0,F_z)$ with $F_z = {it const} + ax +by + cz$. Then $$ oint_c vec{F}cdot dvec{r} = oint_c (ax, dz + by, dz + cz,dz) = a (-S_y) + b S_x + 0= -frac{partial F_z}{partial x} S_y + frac{partial F_z}{partial y} S_x.$$ We have obtained $ oint_c vec{F}cdot dvec{r}= (nabla times vec{F}) cdot vec{n} |S|$, which remains valid when adding contributions also from $F_x$ and $F_y$ (writing up all details is cumbersome, however). This formula is exact for $F$ linear. As $c$ shrinks towards a point $O$ you would like to show that the non-linear contribution disappears in the limit so you have to be careful about how the shrinking takes place. It suffices e.g. if the ratio of $|S|$ to the square-length of $c$ stays uniformly bounded from below under the shrinking process. I'll leave this part aside.

The natural framework is really differential forms and stokes formula (in any dimension). I personally like the book of V.I.Arnold, Mathematical methods of Classical Mechanics, but there is a vast literature on the subject.