If I own a lot of one company's stock via vested RSUs should I diversify?

Here's one way to look at it:

If you had that amount in cash, would you buy your company's stock?

I always look at stock income as cash the moment it's vested. It's fabulous that the stock has done so well thus far, and so long as you can afford the risk there's nothing wrong with keeping it in company stock. But there's no guarantee that it will continue to grow at all, let alone at it's historic pace. Most advisors will tell you that more than 10% of your portfolio in any one instrument is a lot of risk, which is why they encourage people not to have too much in their own company's stock.

If there's other things that you could use the money for (debt repayment, emergency fund, etc.), or if you just want to diversify to reduce risk, then it may be a good time to cash in on some of that growth.

So look at your overall financial picture and decide for yourself what the best use of those funds would be.

If I do sell some of it and diversify into an S&P500 index fund, would I have to pay capital gains when I sell the AMZN stock and reinvest in an an index fund?

Any growth of the stock from the point that it was vested would be considered capital gains. That growth (as well as short- or long-term gain designation) is calculated per lot of stock that you sell. You should have already had tax withheld when each lot of stock was granted, so you only need to capital gains tax on the growth.