Convolution neural network with 11 million parameters unable to overfit on 100 image samples

I have been trying to do some sort of image enhancement on grayscale images. I have used both pixel wise loss and perceptual loss (perceptual loss uses classifier between 2 classes trained on the same dataset).

So the input to my network is an bad image and the output is the enhanced image where edges in an image are clearer.

I have written the code in keras

inputs = Input(shape=image_shape)

x = Conv2D(filters=ngf, kernel_size=(7, 7), padding='same')(inputs)
x = BatchNormalization()(x)
x = LeakyReLU(0.2)(x)

n_downsampling = 2
for i in range(n_downsampling):
    mult = 2**i
    x = Conv2D(filters=ngf*mult*2, kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

mult = 2**n_downsampling
for i in range(n_blocks_gen):
    x = res_block(x, ngf*mult, use_dropout=False)

for i in range(n_downsampling):
    mult = 2**(n_downsampling - i)
    x = Conv2DTranspose(filters=int(ngf * mult / 2), kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

#x = ReflectionPadding2D((3, 3))(x)
x = Conv2D(filters=output_nc, kernel_size=(5, 5), padding='same')(x)
x = Conv2D(filters=output_nc, kernel_size=(3, 3))(x)
x = Activation('sigmoid')(x)

outputs = Add()([x, inputs])
#outputs = Lambda(lambda z: K.clip(z, -1, 1))(x)
outputs = Lambda(lambda z: z/2)(outputs)

model = Model(inputs=inputs, outputs=outputs, name='Generator')

Function for res_block

 def res_block(input, filters, kernel_size=(3, 3), strides=(1, 1), 
 use_dropout=False):
"""
Instanciate a Keras Resnet Block using sequential API.

:param input: Input tensor
:param filters: Number of filters to use
:param kernel_size: Shape of the kernel for the convolution
:param strides: Shape of the strides for the convolution
:param use_dropout: Boolean value to determine the use of dropout
:return: Keras Model
"""
x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides, 
           padding='same')(input)
x = BatchNormalization()(x)
x = Activation('relu')(x)

if use_dropout:
    x = Dropout(0.5)(x)

x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides, 
           padding='same')(x)
x = BatchNormalization()(x)

merged = Add()([input, x])
return merged

Any idea why this wouldn’t overfit ? I have been looking for other loss functions but I couldn’t find much. I personally tried to make pixel loss penalize the model on pixels where it should have completed a incomplete line in an image by using sobel operator on the enhanced image to get a mask which is multiplied by the difference between the input & target image hoping that the network would focus on completing incomplete edges or so but nothing changed.