Lesions detection in video & metrics

I’m develloping an object detection models to detect lesions in video record from operation.
A problem observed during evaluation is that, given that an operation is specifique to a patient, if a lesion is correctly detected, it has a good chance to be correctly detected during the whole video. The opposite is also true, a non-detected lesion will likely not be detected during the whole video.

The video are from real case, so they duration are not balanced, some video are longer than other.
Currently, I’m using the MaP frame-wise , but I believe it is not adequate with the issue I mention abose: some patients with longer video have more "weight" than the other.

An idea would be to calculate the MaP patient-wise (or video wise) then average it ?
Is there a standart approach ?