Multi-class neural net always predicting 1 class after optimization

During training, the neural net settles into a place where it always predicts 1 of the 5 classes.

My train and test sets are distributed as such:

Train Set
Samples: 269,501. Features: 157
Data distribution
16.24% 'a'
39.93% 'b'
9.31%  'c'
20.86% 'd'
13.67% 'e'

Test Set
Samples: 33,967. Features: 157
Data distribution
10.83% 'a'
35.39% 'b'
19.86% 'c'
16.25% 'd'
17.66% 'e'

Note the percentages of class b!

I am training an mlp with dropout, and training and test (aka validation) accuracies both plateau, perfectly matching the train and test distributions of 1 of my 5 classes, i.e. it is learning to always predict 1 class out of 5 classes! I’ve verified the classifier is always predicting b.

I’ve tried batch_size of 0.25 and 1.0 and made double-y sure the data was shuffled the data. I tried both SGD and Adam optimizers with and without decay and different learning rates and still the same result. Tried dropout of 0.2 and 0.5. EarlyStopping of 300 epochs.

Every so often I’ll get a situation where during training it’ll pop out of where it has settled for training accuracy and validation accuracy but then validation always goes down and training goes up — or in other words, overfitting.

Output, cut off after 6 epochs. It doesn’t always converge this fast, just with this particular SGD optimizer:

Epoch 1/2000
Epoch 00000: val_acc improved from -inf to 0.35387, saving model to /home/user/src/thing/models/weights.hdf
269501/269501 [==============================] - 0s - loss: 1.6094 - acc: 0.1792 - val_loss: 1.6073 - val_acc: 0.3539
Epoch 2/2000
Epoch 00001: val_acc did not improve
269501/269501 [==============================] - 0s - loss: 1.6060 - acc: 0.3993 - val_loss: 1.6042 - val_acc: 0.3539
Epoch 3/2000
Epoch 00002: val_acc did not improve
269501/269501 [==============================] - 0s - loss: 1.6002 - acc: 0.3993 - val_loss: 1.6005 - val_acc: 0.3539
Epoch 4/2000
Epoch 00003: val_acc did not improve
269501/269501 [==============================] - 0s - loss: 1.5930 - acc: 0.3993 - val_loss: 1.5967 - val_acc: 0.3539
Epoch 5/2000
Epoch 00004: val_acc did not improve
269501/269501 [==============================] - 0s - loss: 1.5851 - acc: 0.3993 - val_loss: 1.5930 - val_acc: 0.3539
Epoch 6/2000

Code:
Model creation:

def create_mlp(input_dim, output_dim, dropout=0.5, arch=None):
    """Setup neural network model (keras.models.Sequential)"""
    # default mlp architecture
    arch = arch if arch else [64,32,32,16]

    # setup densely connected NN architecture (MLP)
    model = Sequential()
    model.add(Dropout(dropout, input_shape=(input_dim,)))
    for output in arch:
        model.add(Dense(output, activation='relu', W_constraint=maxnorm(3)))
        model.add(Dropout(dropout))
    model.add(Dense(output_dim, activation='sigmoid'))

    # compile model and save architecture to disk
    sgd = SGD(lr=0.01, momentum=0.9, decay=0.0001, nesterov=True)
    # adam = Adam(lr=0.001, decay=0.0001)
    model.compile(loss='categorical_crossentropy',
                  optimizer=sgd,
                  metrics=['accuracy'])
    return model

And inside main after some preprocessing:

    # labels must be one-hot encoded for loss='categorical_crossentropy'
    # meaning, of possible labels 0,1,2: 0->[1,0,0]; 1->[0,1,0]; 2->[0,0,1]
    y_train_onehot = to_categorical(y_train, n_classes)
    y_test_onehot = to_categorical(y_test, n_classes)

    # get neural network architecture and save to disk
    model = create_mlp(input_dim=train_dim, output_dim=n_classes)
    with open(clf_file(typ='arch'), 'w') as f:
        f.write(model.to_yaml())

    # output logs to tensorflow TensorBoard
    # NOTE: don't use param histogram_freqs until keras issue fixed
    #       https://github.com/fchollet/keras/pull/5175
    tensorboard = TensorBoard(log_dir=opts.tf_dir)

    # only save model weights for best performing model
    checkpoint = ModelCheckpoint(clf_file(typ='weights'),
                                 monitor='val_acc',
                                 verbose=1,
                                 save_best_only=True)

    # stop training early if validation accuracy doesn't improve for long enough
    early_stopping = EarlyStopping(monitor='val_acc', patience=300)

    # shuffle data for good measure before fitting
    x_train, y_train_onehot = shuffle(x_train, y_train_onehot)

    np.random.seed(seed)
    model.fit(x_train, y_train_onehot,
              nb_epoch=opts.epochs,
              batch_size=train_batch_size,
              shuffle=True,
              callbacks=[tensorboard, checkpoint, early_stopping],
              validation_data=(x_test,y_test_onehot))