Zomas' answer is only partially correct. There's currently three "types" of damage in TFT: physical, magical, and true damage. There's also two "sources": spell/ability damage, and basic attack damage (...and I guess item damage such as Static Shiv, but that doesn't scale with anything so we'll leave that out of the discussion for now).

Vayne's special ability is that every third basic attack on an enemy, she deals ability damage in the form of true damage based on the enemy's max health. This ability does, in fact, get stronger with additional ability power. Giving her AP will increase the scaling of max health damage she deals.

In fact, there's a "joke" build for her that works surprisingly well: Put a 3* Vayne in a team of 6 sorcerers (for the +120 AP) and give her three Rabadon's Deathcaps (increases AP by 75% each). This will make her max health true damage so high that it will literally deal over 100% of the target's total HP on the third attack.

This is considered a joke build because although it sounds great on paper, Vayne's attack speed is too slow to be able to reliably kill more than maybe two or three units before she dies herself. Still quite funny vs a full health Chogath, though. You can see this build in action in this video.