What statistical tests should be performed to test effect of trials on response times where each subject has different number of data points?

LMM is a good suggestion, second that.

Consider also fitting a log-normal to the RT's as recommended by Haines & friends https://psyarxiv.com/xr7y3/download/?format=pdf

You will have more or less uncertainty depending on how many trials you got, but it won't matter that people did different numbers of trials. I find the arguments in this paper for 'one stage' inference (as opposed to two-stage summary) very compelling, see what you think, your mileage may vary. One reason to like it is that it goes very naturally with hierarchical approaches, so it's a natural ally of that LMM suggestion.

It seems likely that the dropped data points are super-fast or super-long RT's that are from some data generating process other than the one you're studying, right? The participant sneezed or snoozed briefly and then went back to doing the task? You can wrap that understanding into the model! If your actual understanding of the data generating process suggests that RT's are usually log normal with a mean that depends on condition, but there are some uniformly distributed contaminants, you can write down a model that says exactly that and do inference with it. There's a nice rundown of the theory/motivation here: https://www.sciencedirect.com/science/article/pii/S0022249610000593 but it's gotten much easier to do since this article was written. There's a blow-by-blow walkthrough with code in this incredibly awesome blog post https://www.martinmodrak.cz/2021/04/01/using-brms-to-model-reaction-times-contaminated-with-errors/

This type of data (repeated measures on a group of subjects) lends itself perfectly for a linear mixed model (LMM) analysis. I'm into psychophysics myself and I have mostly abandoned simple ANOVAs and switched to LMMs almost completely, especially because it can handle missing data, and because LMMs allow for inclusion of trial and session number to correct for them in an elegant manner. In effect, you can potentially take learning effects and/or mental fatigue into account. Between-subject (baseline) differences and learning effects are two of the most pesky problems in psychophysics and repeated measure designs, respectively. This makes the use of LMMs so rewarding in this field.

An LMM has several huge advantages over standard ANOVA analysis and it is expected to replace standard ANOVA analysis in time.

Advantages of LMM include:

• Subjects (and other variables) can be included as random effect, such that the model corrects for within-subject differences (akin to repeated measures);
• Missing data are allowed and corrected for;
• Fixed and covariate factors can be included and tested for significance. In your case trial # can be included as a covariate. The LMM will calculate the effect of trial # on your outcome measure and will calculate whether that effect is significant or not (check 'model estimates' under statistics when you use SPSS);
• LMMs can be statistically powerful, despite low subject numbers.

If you wish to stick to standard RM-ANOVA, you can also opt to impute your data to correct for missing values. Especially multiple imputation is a robust way of estimating missing data (Van Netten et al., 2016), also available in SPSS.

<sub>Reference
- Van Netten et al. Ear Hear (2017); 38: 1–6</sub>

Mixed effects regression is somewhat related to repeated measures ANOVA and can handle missing data. However, they assume the missing data are missing at random.

Depending on what type of rejection criteria you are using, that assumption may be broken, in which case I think you're pretty much out of luck without a method to account for that. It may be that it is a sufficiently reasonable assumption, however, even if not strictly warranted.