Using accuracy metric during training for unbalanced multiclass classification

I am training a convolutional neural network and the sensitivity and precision of the minority class is what is most important to me. I am using 10-Fold cross validation, and the test fold is evaluated on sensitivity and precision by the "best" model from the 40-epoch training loop. My problem is, I am currently determining the "best" model by its raw accuracy on the validation set, the model weights in the epoch with the highest validation accuracy is used for the test fold. I do this to avoid selecting over fit models.

I am worried that because I am choosing the model based on validation accuracy for the test fold, I am not actually choosing the best model for the sensitivity and precision of the minority class which is my focus, and thus the cross validation result is not the best it can be.

Can someone enlighten me on this topic? Am I just worrying for no reason or should I maybe try evaluating the models based on the precision and sensitivity of the minority class?