How to pay tax on IRA to Roth rollover

A little less than half the contributions were pre-tax, and I understand I have to now pay tax on that amount ....

When you convert a traditional IRA (or employer plan, see below) to Roth, or when you distribute it, you must pay tax on everything except post-tax/undeducted contributions, in other words on pre-tax contributions and earnings/gains. Depending on how old you are now and at what age you started the account, and thus how long you have had the account, earnings are often the bulk of the balance.

I'd like to avoid any penalties for not making quarterly payments ....

The general rule is that you must pay during the year within 10% of your net tax by withholding plus estimated payments together, or within $1000 by withholding alone. But there is a safe harbor that frequently helps with single-year variations: if you pay 100% of the prior year tax (110% if your prior year AGI was over $150k) you are exempt from the penalty -- and as a limiting case if you owed no tax for the prior year you never have to make estimated payments. Usually for people whose wages stay the same or gradually increase from year to year and have no or little other taxable income, so that wage withholding is accurate, and don't have status changes such as marriage or childbirth, withholding this year will automatically be 100% or a little more of last year's tax. But this year with COVID, and with a change of job, that might not be the case -- look at your actual numbers to check. Also this may not work for people whose pay is mostly bonus based on business performance and thus varies a lot, like some salesmen and investment managers. (Also relevant here: if your yearly combined wages from multiple employers exceeds the Social Security 'base', currently about $138k, they will collectively withhold too much Social Security tax, and the excess is counted as a (refundable) payment on your income tax when you file, see schedule 3 line 11.)

If you need to pre-pay and you have a job, you can either make the estimated payments directly or you can request on W-4 your (new) employer do 'extra' withholding, or any combination that adds up to the required amount. See https://www.irs.gov/businesses/small-businesses-self-employed/estimated-taxes (which covers all types of non-wage income even though it is navigated under 'self-employed'). Withholding is the easiest, because it is treated as if paid evenly through the year, and thus always timely, even if it wasn't really.

If you choose to use estimated payments:

1. yes, you use 1040-ES if paying by check, or by cash in-person (not recommended, especially now). But it's very easy to fill out: it contains only your name, SSN, and address, and the amount. You don't need to show your details and calculations at the time you pay, only at year-end on your return. If you pay electronically you only enter your name and SSN and select that it is a 1040-ES payment; the amount is captured automatically.

2. since you are starting mid-year, on your return you will need to use the 'annualized' method on form 2210 schedule AI to show that not only was your total payment enough, but it is okay that it wasn't until the third 'quarter' because your income varied then. This is a lot of extra work, and requires you to keep more extensive records of everything, which is why extra withholding as above is easier.

Should I expect a W2 from the company that managed the old IRA?

No, you should expect a 1099-R. W-2 is used to report wages, and sickpay that replaces wages, and (oddly) gambling winnings. Various 1099-series forms are used for other reportable income including interest, dividends, (taxable) stock sales, credit card or network payments, contract or other non-employee work, unemployment and state/local tax refunds, and as here retirement payments (including pension and Social Security). You should also get a 5498 from the custodian of the target IRA showing the rollover coming in.

Was the old plan really an IRA? Some employer plans are, like SEP and SIMPLE, but most employer plans are not IRAs. Both employer plans and IRAs get almost all the same tax treatment, especially of conversions and distributions, and including as relevant here 1099-R reporting, but that does not make them the same thing.

Is this lump sum even treated like regular income because it's from selling stocks?

Yes. For investments you hold directly, capital gains from stock held more than a year, and 'qualified' dividends, are treated specially and taxed at lower rates. And they are taxed in they year they occur. For tax-deferred accounts like an IRA or an employer DC (defined contribution) plan, transactions you do in the account are not taxed at all -- you can sell one stock for another, or switch between stocks and bonds and 'cash', as much as you like with no tax, even though these would be taxed if held directly.* But all distributions, and conversions, from a traditional/non-Roth deferred account are taxed at ordinary rates. Plus a separate 10% for early distributions (but not conversions) unless certain exceptions apply.

*: whether you should actively trade in your retirement account(s), and try to time the market etc, is a different Q I don't address. But you can do so without tax.