Why does paying points cost different amounts depending on how much I'm lowering my rate?

As you reduce the interest more, the difference shouldn't be linear but it shouldn't be as stark as the difference between your first and second eighth of a point. Let's look at the difference mathematically.

Think of the concept of paying points as a loan. You give the bank capital upfront (they "borrow" from you), and they in turn pay you a monthly "payment" equivalent to the difference in monthly payment between the "base" loan and a loan with a lower interest rate. If the "rate" of that loan (meaning the rate needed to get the desired payment amount) is more than the rate you're paying for the mortgage, then the points are a good deal (providing you have the cash up front to pay them). Otherwise, you'd be better off borrowing less and saving interest on the mortgage in the long run.

An easier figure to look at is the pay-back period, meaning how long will it be before you save enough in interest to make up for the points you pay? For a rough estimate, take the difference in interest amounts of the first payment between the two loans and divide the cost of the points by that amount. That's the number of months that it will take to make up for the points (principal doesn't matter because you're essentially paying that to yourself in the form of a lower loan balance). If you plan to be in the house longer than that, the points are a good deal.

Obviously paying $110 to save 1/8 % is a no-brainer since you'll make that up in a matter of a few months. Which is why I question the validity of that amount...