Extracting a parameter from a probability problem

Characterize the event like this: "getting $3$ redheaded kids in the group" means the random sample comprises $3$ of the $A$ kids (Group 1) and $40-3=37$ of the $1000-A$ other kids (Group 2). There are $binom{A}{3}$ equally probable ways to form Group 1 and $binom{1000-A}{37}$ equally probable ways to form Group 2. Because the choices in these groups are (assumed) independent, the total number of such samples is obtained by multiplying these two numbers. Since the chance of any sample of size $40$ is $1/binom{1000}{40},$ we obtain

$$f(A)=Pr(3 text{ redheads}) = frac{binom{A}{3},binom{1000-A}{37}}{binom{1000}{40}}.$$

Your question now can be framed as

Find the largest positive integer $A$ for which $frac{binom{A}{3},binom{1000-A}{37}}{binom{1000}{40}} le 0.1.$

The problem is small enough that a brute force search will work fine. A mindless but effective search in R, which will spit out the answer, is

A <- 1:1000; A[which.max(exp(lchoose(A,3) + lchoose(1000-A,37) - lchoose(1000,40)) >= 0.1) - 1]

---

If I had to solve this by hand (without using any advanced techniques such as Stirling's formula), I would first overestimate $binom{1000-A}{37}$ as $binom{1000}{37}$ to make the dependence on $A$ simpler, thereby yielding a lower bound for the answer. The approximate relation is

$$frac{A(A-1)(A-2)}{6} = binom{A}{3} le 0.1 frac{binom{1000}{40}}{binom{1000}{37}} = 0.1 frac{(963)(962)(961)}{(40)(39)(38)}approx 1500.$$

Multiplying both sides by $6$ and approximating the left hand numerator as $(A-1)^3$ gives

$$A-1 approx sqrt[3]{(6)(1500)}approx 21,$$

indicating $A$ must be $22$ or larger.

That means the approximation of $binom{1000-A}{37}$ by $binom{1000}{37}$ is a factor at least $(1000)/(1000-22))^{37}$ too high, a value that will approximately be $exp(22/978times 37) approx 2.3$ too big. Consequently this lower bound needs to be multiplied by a number greater than $sqrt[3]{2.3}approx 1.3,$ so let's adjust the bound from $22$ to $22times 1.3 approx 29.$

At this point it's probably easiest just to compute the exact chances for $A=29, 30, 31, ldots$ until finding the last value for which $f(A)le 0.1.$ I leave that exercise to you, but will observe that $f(29) approx 0.08$ is still a little low, as we ought to expect.