Error -Collection query aborted after accumulating over 5000 elements

Question

I have created a code that suppose to take sentinel images and calculate the NDVI for a speficif polygon I have. I need it to run it for 2 years, but I have realized that for some reason the dataset is way too big and I get wrong numbers, for example, if I ask for the size of the dataset between 01/01/2018 to 03/01/2018 I get that the size of the dataset is 5099 images which doesn't make any sense.

this is the code I have up to now:

/**
 * Function to mask clouds using the Sentinel-2 QA band
 * @param {ee.Image} image Sentinel-2 image
 * @return {ee.Image} cloud masked Sentinel-2 image
 */
function maskS2clouds(image) {
  var qa = image.select('QA60');

// Bits 10 and 11 are clouds and cirrus, respectively.
  var cloudBitMask = 1 << 10;
  var cirrusBitMask = 1 << 11;

// Both flags should be set to zero, indicating clear conditions.
  var mask = qa.bitwiseAnd(cloudBitMask).eq(0)
      .and(qa.bitwiseAnd(cirrusBitMask).eq(0));

return image.updateMask(mask).divide(10000);
}

// Map the function over one year of data and take the median.
// Load Sentinel-2 TOA reflectance data.
var dataset = ee.ImageCollection('COPERNICUS/S2')
                  .filterDate('2018-01-01', '2018-01-03')
                  // Pre-filter to get less cloudy granules.
                  .filter(ee.Filter.lt('CLOUDY_PIXEL_PERCENTAGE', 20))
                  .select('B2','B3','B4','B8','QA60')
                  .map(maskS2clouds);

var rgbVis = {
  min: 0.0,
  max: 0.3,
  bands: ['B4', 'B3', 'B2'],
};

var clippedCol=dataset.map(function(im){ 
   return im.clip(geometry);
});

//test if clipping the image collection worked
Map.centerObject(geometry);
Map.addLayer(clippedCol.median(), rgbVis, 'RGB');

// Get the number of images.
var count = dataset.size();
print('Count: ',count);
print(clippedCol);//here I get the error messege "collection query aborted after accumulation over 5000 elements
print(dataset,'dataset');//the same error here

//function to calculate NDVI
var addNDVI = function(image) {
  var ndvi = image.normalizedDifference(['B8', 'B4']).rename('NDVI');
  return image.addBands(ndvi);
};

//NDVI to the ckipped image collection
var withNDVI = clippedCol.map(addNDVI);

// Test the addNDVI function on a single image.
var ndvi1 = withNDVI.select('NDVI').mean();

var colorizedVis = {
  min: 0.0,
  max: 1.0,
  palette: [
    'FFFFFF', 'CE7E45', 'DF923D', 'F1B555', 'FCD163', '99B718', '74A901',
    '66A000', '529400', '3E8601', '207401', '056201', '004C00', '023B01',
    '012E01', '011D01', '011301'
  ],
};

Map.addLayer(ndvi1,colorizedVis,'test');
Map.centerObject(geometry);

my goal is in the end to see the NDVI mean value for every image.

Nishanta Khanal · Accepted Answer

First to address your question, the Error occurs when you are trying to print more than 5000 elements in your console which it says. Since you have 5099 images in specified date range, the error arises. Since you don't want to do a global analysis as you are clipping images, you can actually use

.filterBounds(geometry)

on your image collection along with your other filters so that you only get images within your region which should be much lower. As for viewing mean NDVI, i can see your layer without any issue. however if you are not seeing the NDVI layer at all then that is probably because the images in that date range in your area has cloud cover more than 20 percent which is filtered out by

.filter(ee.Filter.lt('CLOUDY_PIXEL_PERCENTAGE', 20))

in your code. You could try increasing the value 20 to something higher but you might still get empty pixels because of the cloud masking. So it might be a good idea to see the images first without removing clounds and then play with dates so that you get a good number of non-cloudy pixels.

Rishi · Answer

var dataset = ee.ImageCollection('COPERNICUS/S2')
              .filterDate('2018-01-01', '2018-01-03')
              // Pre-filter to get less cloudy granules.
              .filter(ee.Filter.lt('CLOUDY_PIXEL_PERCENTAGE', 20))
              .select('B2','B3','B4','B8','QA60')
              .map(maskS2clouds);

In this line of code add a filter to show only the path and row or the specific region that you need. It is the limitation by Image Collection. 
Just like this.

var collection = ee.ImageCollection('LANDSAT/LC08/C01/T1_TOA')
.filter(ee.Filter.eq('WRS_PATH', 44))
.filter(ee.Filter.eq('WRS_ROW', 34))
.filterDate('2014-0-01', '2014-08-01');

You can find the path row for sentinel here https://forum.step.esa.int/t/path-row-for-sentinel-2/8524

JonasV · Answer

If there are others who are just looking for a way to quickly check the general output of a really big Image Collection which you can't reduce below 5000 elements, the solution is to use .limit() while printing. With limit(10) for example the collection gets limited to 10 elements.
So let's assume we write a new property to a really big Image Collection and just want to check if it is actually in there how we want it. In this case .limit() can be used to quickly check that nothing went wrong.
// Let's write a new property to a really big collection
var s2 = ee.ImageCollection("COPERNICUS/S2_SR");
    .set("ANSWER_TO_EVERYTHING", 42)

// Now we try to check if the property is how we want it
print("This will fail:", s2)

print("This works:", s2.limit(1))

Error -Collection query aborted after accumulating over 5000 elements

3 Answers

Add your own answers!

Ask a Question