Train your own CycleGAN style translator

I have a set of images (a few hundred) that represent a certain style and I would like to train an unpaired image to image translator with CycleGAN.

I’m looking for a tutorial on how one would do this with NetTrain.

For example, in the Wolfram Neural Net repository there is a NetModel for Photo-to-Van Gogh Translation, but the construction notebook is missing any details about training:

Ok, so I’m trying to piece this back the missing training section. From the construction notebook we have these functions:

convIN[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001]}

convINRamp[args___] := {ConvolutionLayer[args], 
  InstanceNormalizationLayer["Epsilon" -> 0.00001], 
  ElementwiseLayer[Ramp]}

residualModule = NetGraph[
   {
    NetChain@
     Join[{PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, 
      convINRamp[128, 
       3], {PaddingLayer[{{0, 0}, {1, 1}, {1, 1}}, 
        "Padding" -> "Reflected"]}, convIN[128, 3]],
    TotalLayer[]
    },
   {NetPort["Input"] -> {1, 2}, 1 -> 2}
   ];

cycleGAN = NetChain[
  Join[
   {ElementwiseLayer[2 # - 1 &],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, 
     "Padding" -> "Reflected"]},
   convINRamp[32, 7],
   convINRamp[64, 3, "PaddingSize" -> 1, "Stride" -> 2],
   convINRamp[128, 3, "PaddingSize" -> 1, "Stride" -> 2],
   ConstantArray[residualModule, 9],
   {DeconvolutionLayer[64, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    DeconvolutionLayer[32, 3, "Stride" -> 2],
    PartLayer[{All, 2 ;;, 2 ;;}],
    InstanceNormalizationLayer["Epsilon" -> 0.00001],
    ElementwiseLayer[Ramp],
    PaddingLayer[{{0, 0}, {3, 3}, {3, 3}}, "Padding" -> "Reflected"],
    ConvolutionLayer[3, 7],
    ElementwiseLayer[0.5*Tanh[#] + 0.5 &]}
   ],
  "Input" -> NetEncoder[{"Image", 512}],
  "Output" -> NetDecoder[{"Image"}]
  ]

Now let’s grab some data in order to try training it:

styleFolder = "~/Downloads/style";
randomFolder = "~/Downloads/random";

CreateDirectory /@ {styleFolder, randomFolder};

URLDownload[#, styleFolder] & /@ 
  WebImageSearch["monet", "ImageHyperlinks", "MaxItems" -> 5];

URLDownload[#, randomFolder] & /@ 
  WebImageSearch["outdoors", "ImageHyperlinks", "MaxItems" -> 5];

styleImages = File /@ FileNames["*.jpg", styleFolder]
randomImages = File /@ FileNames["*.jpg", randomFolder]

This is where I’m stuck, and there are two main questions:

1. How to write cycle-consistency loss
2. How to format/feed the training data to the GAN

But I’m still a long way from filling in these last two lines:

trainingData = (* ? *)
NetTrain[cycleGAN, trainingData]

Links:

• The code and full results for the CycleGAN paper can be viewed at https://github.com/junyanz/