FME SherbandGeneralizer - estimate time for "Optimizing Memory usage"

Question

I have a workspace with a pointcloud of 240.000.000 points as input, goal is generalized countour lines, that do not intersect each other.
The workspace mainly consists of (cp. screenshot)

PointCloudSimplifier
SurfaceModeller (Output Countours)
LineCombiner / LengthCalculator / Tester to eliminate very small contour 'islands'
SherbandGeneralizer (input of remaining 82190 Features)

Testing with a small subset of the input data (~10.000.000 points) was very promising.
Except for the SherbandGenralizer the whole thing takes place in an acceptable time, but the latter produces an Optimizing memory usage. Please wait.... Right now, I'm waiting for two weeks.
The FME Process consumes somewhat around 16%CPU and very few working memory. Working memory load ist around ~11 of 16Gb.

I have an i7-6700 with 16Gb RAM, 64bit FME 2018.1.1.1 on Win10.
From the documentation we can only learn that the algorithm is time-intensive, and especially using constraints as in this case is highly-time-intensive, but this is not quantified. My thought was that the processing time would be in some way comparible with that of the PointCloudSimplifier (very processing-intensive according to the docs), but obviously it is not.
Question is: Is it somehow predictable how long I would have to wait for optimizing the memory usage? Taken memory load into account, I wonder why there is the need for optimization, should I try it over with more RAM (64Gb max possible), or should I give FME 2020 a try?

nielsgerrits · Accepted Answer

Two weeks processing time (and counting) is serious in my book. Probably someone from Safe Software has a better response, but here you got mine:
When I see the notification "Optimizing Memory" in the log, I stop the run and start to redesign the workbench. As a side note my system has 32GB RAM.
When optimizing memory, the process starts swapping between RAM and TEMP. In my experience the process will take ages to complete, caused by the massive amount of I/O. I don't think upgrading to 2020 fixes this, except if a memory hungry transformer got an upgrade.
As for more RAM, this might fix it. I would test with different sized smaller sets and register max memory usage in the log after each run and try to predict what you need when you run the full set. If the trend is exponential more RAM is probably not going to help.
In my experience, processing spatially large datasets often asks for sub-processing: Tiling the point cloud in manageable chunks, run these one by one and glue the results together. This will probably give you new challenges (like different results on the edges on different tiles) but solvable ones.
Maybe tile, buffer to a larger tile, process, clip back to the original tile then merge. Start with two tiles, if it works out well run the entire bunch.

FME SherbandGeneralizer - estimate time for "Optimizing Memory usage"

One Answer

Add your own answers!

Ask a Question