How to Convolve a High-Res Image by a (fully convolutional) CNN kernel?

My CNN is an extremely simple neural network.

input_img = Input(shape=(80, 80, 1))

x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

decoder = Model(input_img, decoded)
decoder.compile(optimizer='adadelta', loss='binary_crossentropy')

As you can see, it is a fully convolutional neural network. Once it get trained, can I use this CNN as a kernel for convolution operated on a large, high-res image, meaning can I do something like this?

output = scipy.signal.convolve(target image (4k x 4k), CNN (trained on 80 x 80 pix), mode='same')

I need this to do image segmentation. I know there are U-nets to do segmenttion, but my problem doesn’t require such a big network, and I need something extremely light and fast.