FEM and High Performance Computing

You will have more control over things if you use PETSc. The most difficult part of writing a performant FE code is parallel assembly and solve and PETSc takes care of both. PETSc even has routines for managing unstructured meshes (DMPLEX).

With other codes your choice of programming language, type of meshes/elements, etc. are somewhat limited. PETSc also has the backing of DOE and mostly likely will be around even 20-30 years from now. Same may not be true for other projects. Right now there are too many of them and I doubt all will be actively maintained after 10-15 years.

Why would you want to do things on your own? The libraries you mention have all been run on 10,000+ cores and under the hood use PETSc, Trilinos, hypre, ... for the solution of linear systems or use matrix-free approaches. You would have to invest tens of man-years of work to implement the functionality and optimizations that has gone into these libraries -- you won't be able to compete, and nor should you: all you will achieve is a software that does what others have already done for years, and will do so slower.

(Disclaimer: I'm one of the principal authors of deal.II.)

To say you should just use open source is quite naive. I think it depends on what you are interested in.

If you are interested in code development which should be published later I highly recommend implementing your own stuff.

Here are some arguments:

1. You know what you are doing!
2. It often appears that something is not exactly implemented as you thought it would be.

If you are interested in the applications, where you only want to make changes to physical modelling or using different equation systems then simply use open source libraries.

I would highly suggest you go with an available FEM open-source library (say deal.II, FENICS, MFEM, etc.) instead of writing your own FEM code and then using PETSC as the underlying parallel algebra library. First, the majority of open source HPC FEM code already use either PETSC or Trilinos under the hood (deal.II supports both, FENICS uses PETSC, etc.). Additionnaly, I think you underestimate the complexity of writing your own parallel FEM code. It might doable in a few months for a simple Poisson equation if you already know what you are doing, but if you want to solve non-linear vector-equations and support more than just P1 elements, it will take you considerable time to write and test a piece of software that can reach the same degree of performance as the majority of open source libraries. Additionnaly, this re-inventing of the wheel leads to results that are very difficult to publish because they bring nothing new. You have to think about the amount of elements you will need to re-program : parameter parsing, mesh parsing, result output in parallel (which is not a trivial task), etc.

I would really advise agaisnt writing your own FEM platform from scratch. It is better to contribute to an existing platform to add the functionnalities that you are after. Although you need to invest the time to learn a library, you generally recuperate this initial investment as soon as you do something more complex.

The correct answer to your question, IMHO, is "depends on your target and your problem at hand".

1.) If your target is to simulate a large-scale problem on HPC and if you know of an existing code which can model the physics of your problem readily, then use the existing code.

2.) If an existing code does not yet support the physics of your problem but offers a lot of utilities, then build the required features on top of that library.

3.) If you are a researcher working on Computational Physics or Computational Engineering, then having your own code might be a good idea. All the current opensource libraries have some or other limitations: they may not support some element types or material laws, or their way of application of BCs is not robust.

Still, you should make use of the libraries such as PETSc, Eigen, Boost and VTK, and develop the code for solving the physics of the problem using a numerical scheme of your choice.

Note that even with all the third-party libraries, it takes a lot of effort to develop the code on your own. But it is necessary if you are working on new numerical schemes.