A bus goes to 3 bus stops, at each stop 3/4 of the people on the bus get off and 10 get on. what is the minimum number of people to start on the bus?

Let the bus have x passengers in the beginning

At first stop: No. of people left in the bus= x/4 + 10

At second stop: No. of people left on the bus= x/16 + 50/4

At third stop: No. of people left on the bus= x/64 + 210/16

Hence, the number of people left on the bus after the bus stops three times is (840 + x)/64

Obviously, the number of people can neither be fractional, nor negative

Hence, 840+x has to be a multiple of 64.

The minimum possible value of x for which 840+x is a multiple of 64 is 64*14-840

Hence x=56

You found out that when $n_0geq1$ passengers are on the bus at the start then after three stops there are $$n_3={840+n_0over64}$$ people on the bus. As $n_3$ has to be an integer the smallest $n_0$ that would qualify is $n_0=56$, making $n_3=14$. In order to make sure we have to check that for this $n_0$ the intermediate numbers $n_1$ and $n_2$ are integers as well.

By the way: When $x_k$ is the number of passengers after $k$ stops then we have the recursion $$x_{k+1}={1over4} x_k+10 .$$ The "Master Theorem" gives the general solution $$x_k=ccdot 4^{-k}+{40over3}qquad(kgeq0) ,$$ but this expression does not care about integerness. Therefore we really have to go through the cases.

Let $R equiv {0, 1, 2, cdots }, ;S equiv {1, 2, 3, cdots }.$

Initially, there are $x_0$ people.
After the 1st stop, there are $x_1$ people.
After the 2nd stop, there are $x_2$ people.
After the 3rd stop, there are $x_3$ people.

(1) $;x_0$ goes to $(1/4)x_0 + 10 = x_1.$
(2) $;x_1$ goes to $(1/4)x_1 + 10 = x_2.$
(3) $;x_2$ goes to $(1/4)x_2 + 10 = x_3.$

It is immediate that $x_0, x_1, x_2$ are all multiples of 4 $;Rightarrow$
$exists ;a,b,c ,in ,S ;ni $ $; x_0 = 4a, ; x_1 = 4b, ; x_2 = 4c.$

(4) By (2), $;b + 10 = 4c ;Rightarrow; c geq 3 ;Rightarrow; exists ;k ,in ,R ;ni c = 3 + k ;Rightarrow $
$x_2 = (12 + 4k) ;Rightarrow$

[by (2)] $;b = (x_2 - 10) = 2 + 4k ;Rightarrow $
$x_1 = (8 + 16k) ;Rightarrow$
[by (1)] $;(1/4) x_0 = a = (x_1 - 10) = [(8 + 16k) - 10] = 16k - 2 ;Rightarrow$

$k geq 1;$ and $;x_0 = 4a = 64k - 8 ;Rightarrow$ The min value for $x_0$ is 56.

Addendum Originally, I thought that the answer was 40.
Then I realized that I was misreading the query. That is, at each stop, 3/4 of the people get off (before 10 get on), not 1/4 of the people.

Addendum-1 A fair criticism of my answer is that I didn't try to focus on the OP's work, and guide his work to a solution. I neglected to try, because with a problem like this I'm only comfortable taking baby steps, so I'm uncomfortable trying to critique a sophisticated approach.

Now expand your third expression to put everything over a common denominator. For example at the third stop you have $frac{stuff}4+7=frac {stuff+28}7$ $stuff$ still has fractions in it, so unpack them. See what the denominator comes out to be and see what the smallest $n$ is to make the fraction an integer.