Need some advice, control system animation. Appreciate some help!

You can use FlightGear as your image generation (animation) software. What you have to do is:

1. model your system dynamics (forces, moments)
2. write code to solve your system dynamics in time (use Equations of Motion libraries, or write your own)
3. use your code to start and run the simulation and feed the states to FlightGear.

You can replace FlightGear with any other tool that you might find. It is mentioned for availability, being free and open source and it has lots of documentation and tutorials.

Your code can be in any language that you are comfortable with: Python, Matlab, C, C++, Java, Assembly, etc.

If you are a bit familiar with python programming language, this can be a streamline solution:

1) Install free simple 3D engine, e.g. Panda3D or V-REP.

2) Find appropriate STL models of helicopter, landing zone, etc.

3) Import them to 3D scene, scale and set zero positions and orientation.

4) Write doing-all-the-math function in python (there is also interface to MATLAB). Derivatives from equations now are transformed to increments, e.g. v += a*dt, x += v*dt

5) Call this function from 3D engine cycle loop to manipulate (and read from) scene objects. Engine loop calculates dt for every frame that leads to smooth real-looking sumulation.

This is more an answer to your problem, than the actual question.

When in the world of professional and commercial CAE (where Simulink is), the natural way to implement the controller for the helicopter would be software like Simulink, or Activate, or similar. This is a strong indication if the controller is complex, most often if you develop a commercial grade controller. The natural way to simulate the helicopter is an MBD (multi-body-dynamics) software. It usually comes with prostprocessing software, that allows you to plot any values or render a 3d animation. This is a strong indication if you develop a helicopter, that you turned into a prototype and that should actually fly at some point. MotionView/MotionSolve, ADAMS, RecurDyn, Simpack, ...

FEM can back the MBD model with elastic bodies. If the helicopter is not an RC type, then you probably want the rotor blades modeled like this.

If one of the two is a simplyfied model, you can do it in the other software. This is like trying to do the helicopter in Simulink. But, the same can be done the other way round. The helicopter model in MBD could contain some differential equations for the controller.

Industries use both at the same time. MBD and controls can be connected via: a) Cosimulation, where at finite time steps, values are passed between the models. When designing controllers, this is not so wrong, because digital controllers also have to work at finite time steps. b) FMU's can be used to intergrate one model in the other. The FMU implements only the equations, the software of the other part integrates/solves these. For example: You use MBD to simulate the helicopter and choose this as host. You export the FMU of the controller from the controls software and import it in the mbd software. The mbd software will solve the equations of the controller along with the equations of the mbd-model. This will prevent the solvers from interchanging values at time steps or use values from last time step.

Using professional software might take away some of the thrill of developing it on your own, but if you're after that, you should probably choose the python approach, that Alexander proposed. I am not sure how much simulink would be adding to that, if you're in a educational situation.