Summing mulitple GridCoverage Objects using GeoTools?

Question

I'm working in Java and GeoTools to process some ASCII rasters. At present, I'm trying to get the rasters summed onto one large grid. I've tried a couple of methods but I seem to be having some performance degradation.

(These are manually copied so apologies for typos)

I initially tried:

Operatons op = new Operations(null);
GridCoverage2D sum = initialise();
for (GridCoverage2D grid : grid){
    sum = op.add(sum, grid);
}

This rattled through seemly quickly but when I try to access sum is takes ages, so I assume these operations are conducted lazily. I then tried to add a writer in between to access the grid so it's calculated on the fly.

Operatons op = new Operations(null);
GridCoverage2D sum = initialise();
for (GridCoverage2D grid : grid){
    sum = op.add(sum, grid);
    GridCoverageWriter writer = new ArcGridWriter(File.createTempFile(...))
    writer.write(sum);
}

Which improved things a little. I then thought doing a recursive sum might help, but I'm getting continuously slower results still and hammering a single processor. Are there any ways of increasing the speed of these operations or am I approaching this in the wrong way?

The grids are approximately 4000x4000. At the moment summing 6 grids takes 5 minutes, 8 takes about 15, 10 takes approaching half and hour. The grids are generated using a fortran model that I don't have access to and there could be up to approaching 100 grids being added together. I have resampled the grids so they all fall in the same envelope. The only problem with the first approach is the runtime, but I was wondering if there would be a more efficient way of doing it?

Ian Turton · Answer

There are basically two ways to do this, one with an Op.add as you have now and the other with Jiffle.

GridCoverage2D[] grids = new GridCoverage2D[nRasters];
    for (int k = 0; k < nRasters; k++) {
      WritableRaster raster2 = RasterFactory.createBandedRaster(java.awt.image.DataBuffer.TYPE_INT, w, h, 1, null);
      for (int i = 0; i < w; i++) {// width...

        for (int j = 0; j < h; j++) {
          raster2.setSample(i, j, 0, k);
        }
      }
      grids[k] = gcf.create("name", raster2, referencedEnvelope);

    }

    Operations op = new Operations(null);
    float[][] zero = new float[w][h];
    for (int i = 0; i < w; i++) {
      for (int j = 0; j < h; j++) {
        zero[i][j] = 0.0f;
      }
    }

    long start = new Date().getTime();
    GridCoverage2D sum = gcf.create("name", zero, referencedEnvelope);
    for (GridCoverage2D grid : grids) {
      sum = (GridCoverage2D) op.add(sum, grid);
    }
    long mid = new Date().getTime();
    System.out.println("calc = " + (mid - start));

    String url = "sum.tif";
    File file = new File(url);
    GeoTiffWriter writer = new GeoTiffWriter(file);
    writer.write(sum, null);
    writer.dispose();
    long end = new Date().getTime();
    System.out.println("write = " + (end - mid));
    System.out.println("total = " + (end - start));
    org.geotools.process.Process jiffle = Processors.createProcess(new NameImpl("ras", "Jiffle"));
    Map<String, Object> inputs = new HashMap<>();

    String[] sourceNames = new String[nRasters];
    String letters = "abcdefghijklmnopqrstuvwxyz";
    String script = "dest = ";
    for(int i = 0;i<nRasters;i++) {
      sourceNames[i] = letters.substring(i, i + 1);
      script += sourceNames[i] + " + ";
    }
    script = script.substring(0, script.length() - 3) + ";";

    System.out.println(script);
    inputs.put(JiffleProcess.IN_SOURCE_NAME, sourceNames);
    inputs.put(JiffleProcess.IN_COVERAGE, grids);
    inputs.put(JiffleProcess.IN_SCRIPT, script);
    start = new Date().getTime();
    Map<String, Object> output = jiffle.execute(inputs, null);
    GridCoverage2D result = (GridCoverage2D) output.get(JiffleProcess.OUT_RESULT);
    mid = new Date().getTime();
    System.out.println("calc = " + (mid - start));

    url = "sum2.tif";
    file = new File(url);
    writer = new GeoTiffWriter(file);
    writer.write(result, null);
    writer.dispose();
    end = new Date().getTime();
    System.out.println("write = " + (end - mid));
    System.out.println("total = " + (end - start));

I get faster results with Op.add (roughly 2x faster with 10 4kx4k grids). But on my machine it's of the order 5-10 seconds.

calc = 122 ms
write = 5331 ms
total = 5453 ms

calc = 7762 ms
write = 2574 ms
total = 10336 ms

Update

Oddly, (as you report) if your grids are not square Op.add is much slower. Results as before but height = 4001 gives (while 4001x4001 is the same as before):

calc = 142 ms
write = 8,360,682 ms
total = 8,360,824 ms

calc = 7177 ms
write = 2802 ms
total = 9979 ms

So if you have "irregular" grids jiffle might be the way to go.

Summing mulitple GridCoverage Objects using GeoTools?

One Answer

Add your own answers!

Ask a Question