Classifying videos with varying length using ConvLSTM2D in tensorflow

I found a solution that works for me.

Since I wanted to avoid using padding and masking, and didn't fully understand ragged tensors, I decided to continue with using varying input lengths.

My training data consists of a list of image stacks between 6-82 frames. When trying to use this directly with model.fit(x=x, y=y, batch_size=1) where x is a list of tensors, tensorflow will complain that the input dimensions have varying size. I thought this didn't matter since the batch size was 1, but apparently tensorflow tries to change the list of image stacks to a tensor.

A way around this, is to pass a training sample for every step using a generator:

class ArtificialDataset(tf.data.Dataset):
  def _generator(samples):
    for i in samples:
      yield (image_stack, output)

And then making sure that the steps per epoch is the size of the data set

training_data = ArtificialDataset(training_samples)
model.fit(training_data, epochs=epochs, steps_per_epoch=len(training_samples))

This way tensorflow never tries to create a tensor with varying inner dimensions. A drawback of this method is that no batching occurs, so training takes a while. In my case this doesn't matter much since the network size and input data already requires me to load very few samples at a time.

An optimization would be to batch same-length videos (or videos with same number of extracted frames). I might do this at a later time, since the implementation of varying batch sizes is too much work right now.