how to reproduce a matching network of a vendor

1. Compensation structure in datasheet RO4350 0.254mm Estimating in QUCS / tools->line calculation (http://qucs.sourceforge.net/):
   

w=0.3 l=0.55 -> Z0 = 68 Ohm; Angle = 25 Deg ≈ 0.7λ
w=1.1 l=1.65 -> Z0 = 31 Ohm; Angle = 83 Deg ≈ 0.23λ

2. Tuning smith chart software (http://fritz.dellsperger.net/smith.html): Effective lengths are longer. Wide section is probably quarterwave transformer, so i used 0.25λ:
   Thin section is shortened. In this particular case it is better to use EM simulation software, I think Sonnet Lite is capable of doing that.

3. If you are going to use FR4 material at 24 GHz, 1.6mm is too thick. It will result in extremely wide lines, and they may act not as expected. I recommend to use 0.6mm, maximum 1.0mm.

4. Here is my compensation structure for BGT24MTR12:
   
   It differs from original compensation structure. Thin section is longer and act as half-wavelength impedance repeater. It will have worse bandwidth than original RO4350 compensation structure.

Check new bgt24ltr11, although it may be more difficult to solder (UPDATE 2020: bgt24ltr is not 50 Ohm matched, TX/RX port impedance is around 100 Ohm and may have some little complex part. Datasheet/appnote provides matching structure to 50Ohm).

Your use of the smith chart is correct, but your numbers are a little off due to the wavelength you used (6.5mm).

In stripline, the fields are completely contained inside the substrate. The fields are partially in air around a microstrip line, resulting in an effective permittivity which is lower than the relative permittivity of the substrate. Wavelength can be calculated using the effective permittivity or the delay (Tpd). Both are available from your impedance calculator. Er effective is 3.0635. Tpd is 5.8e-9 s/m.

$ Wavelength = frac{c}{sqrt{E_{r}} cdot f} = frac{1}{T_{pd} cdot f} $
$ Wavelength(w/ E_{r}) = frac{c}{sqrt{E_{r}} cdot f} = 6.5 mm $
$ Wavelength(w/ E_{eff}) = frac{c}{sqrt{E_{eff}} cdot f} = 7.1 mm $
$ Wavelength(w/ T_{pd}) = frac{1}{T_{pd} cdot f} = 7.1 mm $

The difference between 6.5mm and 7.1mm may seem small, but check out the difference in return loss.

$ Z_{out}(Lambda:6.5mm) = 33-j24 Omega $ >>>> $ ORL=9dB $
$ Z_{out}(Lambda:7.1mm) = 46-j14 Omega $ >>>> $ ORL=16dB $

For a sanity check, match from the load to the conjugate of the generator. In other words, start at a load of $50Omega$ and go towards the chip. You'll end up around $20.8+j20.2Omega$. This is typical for most output stages, since they're usually low resistance and capacitive.

Your method of matching will provide a decent ballpark, but I'd like to point out a few important sources of error.

1. Tx & Txx are intended to be 100ohm differential signals. The small spacing between them produces an additional shunt capacitance. You need to include the extra capacitance, or use a coupled line model.
2. The datasheet doesn't say how they measured the Tx/Txx load impedances. Did they measure at the PCB output port and try to de-embed everything up to the chip? I'm skeptical of the accuracy without knowing where they came from. Additionally, I think these are the impedances looking into the Tx/Txx ports. I would call the load impedance the conjugate of the printed values, or the impedance presented to the Tx/Txx ports. It's all unclear.
3. The package pad (footprint) will have a different impact on a different board stackup.
4. I don't see anything to cancel mismatch from the launchers. The launchers will perform differently on a different board stackup.

The matching network they used is simple and probably low loss. I guess you could call it a stepped-impedance match. I don't know. It's a plus to avoid shunt components or stubs, and the high Q resonances they might produce.

Since you have a two-sided FR4 you won't be able to reproduce their matching network. What you can do is to use a microwave design software.

I suggest awr microwave office or agilent ads. There you should create two ports for TX and TXX and assign their respective impedances. You should also create a load that you want to use.

Then use microstrip libraries to create a matching network (there's plenty of examples in both programs).

Create S21 graph and tune microstrip' width and length manually or use a built-in optimizer (set constraints and goals first).