Which C++ Multi-objective Optimization libraries allows the addition of custom problems and custom algorithms?

Would the COCO framework be useful? See the competitions at GECCO 2016 (and earlier conferences). http://gecco-2016.sigevo.org/index.html/HomePage#&panel1-1

I'm not sure if it exactly has SPEA2 or NSGA, but check out DAKOTA. It has all manner of genetic algorithms that can tackle discrete and continuous optimization problems. You can either link to it directly, or use it as a driver which calls your programs as a black box. It's all open source so I'm sure you could add your own custom algorithms, though I can't speak to how easy that is.

Overall it's a fantastic and very capable program for optimization (and many other related problems).

Look at the jMetal library 1 it is in Java, which is very similar to C++. It is very extensive and has several, and rather recent, multi objective methods. It is probably the best MO library I have found out there in any language.

Also I remember some professor had a C++ implementation of NSGA III. He could not reproduce the paper that proposed the algorithm , but it may perfectly fit your needs.

EDIT: I added a thread that discusses NSGA III implementations. Also, I would like to comment that I have used the variation of NSGA II implemented in Matlab with many objectives ( >10) and binary variables, and it worked pretty well for my application. So it is always good to try, even if people say that it is limited to a certain number of objectives. You also have to use you parameters intelligently (population size, mutation rate, etc.)

References

1. J.J. Durillo, A.J. Nebro jMetal: a Java Framework for Multi-Objective Optimization. Advances in Engineering Software 42 (2011) 760-771.
2. A.J. Nebro, J.J. Durillo, M. Vergne Redesigning the jMetal Multi-Objective Optimization Framework. GECCO (Companion) 2015, pp: 1093-1100. July 2015.

Despite late response, I believe openGA is the library that fits your requirements.

It is supporting multi-objective problems with NSGA-III. It is based on C++ and supports Linux, Windows and OSX. The templated structure of openGA requires you to define the solution by yourself and manage it however, this means you have maximum customization on it and you can even implement genetic programming with it. Even though it was not originally planning for GP.

You can apply any constraint you want on the solutions and these constraints can act at the beginning, during it or at the end of the cost function.