Dataset image size and inference speed

As you have phrased it the answer is no.

Models typically have a fixed input and output size (except RNNs, which are not really used for images). This means that you can not input smaller images to the network. If it appears that you are doing so it is either because the network is automatically resizing or padding your small image up to the model input size. In these cases, the time performance of the model will be fairly constant.

If you were to train the same model with a smaller image input size, you would be correct in saying that inference (and train) time would be faster than the larger image input size model. You could also load the pre-trained weights from the larger model into the smaller model, in most cases.

Note that what you are considering will change the input resolution and as such the features will change, decreasing the loss-performance of the model.