Model trained on CIFAR 10 performing badly when fed with internet downloaded images

Question

My model is based on Shallow Net.
When I am training my model, the results are:
loss: 1.1398 - accuracy: 0.6093 - val_loss: 1.2309 - val_accuracy: 0.5657

Then I downloaded 20 images (2 for each class) from the net to check the performance.

Labels corresponding to this dataset should be:0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9].
But my model's prediction is: [0,0,0,1,0,1,5,0,0,5,2,2,0,0,5,2,0,0,1,9].
The accuracy is: 0.2 which is quite low as compared to 0.5657.
My code to load these datasets:
for file in os.listdir("C:/Users/....."):
    img_arr=cv2.imread(os.path.join(os.getcwd(),"Dataset",file))
    img_arr=cv2.resize(img_arr,(32,32))/255
    img_arrs.append(img_arr)

img_arrs=np.array(img_arrs)
img_arrs=img_arrs.reshape(20,32,32,3)
model=load_model("weights.hdf5")
pred=model.predict(img_arrs).argmax(axis=1)

What could be the reason behind this? Can someone give me an insight?
Edit:(Added training code)
(x_train,y_train),(x_test,y_test)=cifar10.load_data()
x_train=x_train.astype(float)/255
x_test=x_test.astype(float)/255
lb=LabelBinarizer()
y_train=lb.fit_transform(y_train)
y_test=lb.transform(y_test)

labelNames = ["airplane", "automobile", "bird", "cat", "deer","dog", "frog", "horse", "ship", "truck"]

model=ShallowNet.ShallowNet.build(width=32, height=32, depth=3, classes=10)
sgd=SGD(0.001)
model.compile(optimizer=sgd,loss="categorical_crossentropy",metrics=["accuracy"])
H=model.fit(x_train,y_train,validation_data=(x_test,y_test),batch_size=32,epochs=50,verbose=1)

James · Answer

Here are some possibilities that come to mind:

The ordering of the dimensions. Depending on the network you are training with and the original training data, you may need to transpose some of the dimensions and/or reshape them differently. The original cifar 10 data is a little strange in that the color-dimension precedes the height/width dimensions. It may help to visualize both an image from the training data and the internet data side by side with the same function and ensure they are displaying the same way as a test of this.
Was any pre-processing performed on the training images, for example rescaling of values? If so, the same pre-processing should be performed on these images. Even if pre-processing was not explicitly performed image data can be stored in a variety of different ways. I would check the range of values on your training/validation images and compare against the images you are getting from the net.
Something funky is possibly happening when these images are resized. I would suggest visualizing the above examples after they have been resized and confirming that they are reasonable.
Another possibility is that something unexpected is happening when loading the model - it may seem redundant given the validation results but I would nonetheless take a set of 20 or so images from the original validation set and check the predictions after retrieving the model with the above load_model function.

10xAI · Answer

I think ~50% accuracy might not be so good to match it to new images on the basis of accuracy only.
 Similarly 2 images per class is also a bit small to check a model unless a model is built on millions of images which has seen almost every type of variances across pixels
What you may try - 
Check the Loss of Individual classes on training
Try to match the Loss of these images i.e. 2 per class
Do the same thing for Accuracy i.e. per class accuracy
Try to check the activation of 2nd last layer i.e. layer prior to Softmax
 Try to gather 80-90 images and repeat if your Model score ~80%
I meant you should try to look inside the Model instead of a Black-box approach

Model trained on CIFAR 10 performing badly when fed with internet downloaded images

Edit:(Added training code)

2 Answers

Add your own answers!

Ask a Question