Big vibration with Tektro Auriga brakes

Your question is not quite clear, but there is no braking power difference between rotor brands. Differences are solely in weight, design, durability, heat dissipation on long descents etc.

So if your RT56 rotor is not causing vibration, but it is less powerful, then it means that the other rotor is a different size. Otherwise the braking performance would be the same. Rotors of a different size will require an adapter also to fit to the same frame.

It is quite possible that a bigger rotor causes vibration where a smaller one does not, since the braking force is directly proportional to the diameter of the rotor.

I'd expect 160mm to be sufficient rotor size (you don't mention) for most purposes. Remember that most force comes from the front, so for XC MTB some people use 180mm front, 160mm rear. You could experience with using your RT56 rotors on the rear and the Tektro (?) on the front, or vice versa (depending on size).

Also your fork will definitely have a rated rotor size, which relates to the braking force the fork is designed for. If you are putting too much in, then it might well vibrate. You mention 26" wheels, which suggests an older bike - older forks were often 25.4mm, whereas today 32mm is standard. I'd review the rotor size specs for your fork and the rotor size originally fitted to the bike.

What QR Skewer do you use?

I had the same problem on my QR disc brake bike. After upgrading to BB7 with a good, brand new rotor and good bedding in, the issue got better but did not go away completely. The scariest part was that the QR would just untighten with brake use, so I had to control regularly.

After that I upgraded the stock QR for a old Shimano internal cam one and the problem was completely gone. The issue was not only a weaker cam mechanism, but also that the stock QR had a plastic part that deformed under braking. The QR I now use is all metal.

Take a look at this article for more details: https://handsonbike.blogspot.com/2013/08/difference-between-good-and-bad-qr.html

That kind of vibration can be difficult to eliminate without replacing the rotor.

The window pattern on a given rotor can contribute to it, as can the uneven transfer layer that can develop from not bedding in the rotor properly to start with.

Some improvement can be had by sanding aggressively, then cleaning and re-bedding the rotor. Sanding it off the hub is better because otherwise you'll bias the work on the easy side.

Usually I replace rotors outright to solve this problem because it's more of a guaranteed fix.