What statistical tests should be performed to test effect of trials on response times where each subject has different number of data-points?

I am running a behavioral experiment where a number of subjects (=20 in my case) perform a simple cognitive task. The experiment consists of a fixed number of trials (say, 40 in my case). During each trial, the participant executes a single key-press and the response time (RT) is recorded.

So my recorded data looks like this:

subject-1   -> [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)
subject-2   -> [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)
...        ...       ...
subject-20  -> [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)

Now based on some RT criterion, a few trials are removed for each subject. This resulting data looks like this:

subject-1   -> [Trial-1 RT, Trial-3 RT ... Trial-40 RT]              # (32 trials)
subject-2   -> [Trial-1 RT, Trial-2 RT ... Trial-38 RT]              # (36 trials)
...        ...       ...
subject-20  -> [Trial-3 RT, Trial-8 RT ... Trial-40 RT]              # (28 trials)

The removal of a few trials result in a non-uniform number of data-points for each subject. Eg. subject-1, subject-2 and subject-20 have 32, 36 and 28 trials respectively.

Now I want to test the effect of trials on response times, what statistical methods should I use?

I know that when I am not removing the data, I have a nice 20*40 (subjects * trials) data matrix on which I can perform repeated measures ANOVA (within-subject) to see the effect of trials on response times. But how should I go about it if I am removing a couple of trials?