class variable not updating after method call

A JDialig is just like a JFrame. That is you add components to it like you do any frame.

The difference is that you can make a JDialog modal. This means that when you use:

dialog.setVisible(true);
System.out.println("here");

The code after the setVisible(...) statement will not be executed until the dialog is closed. It also means you can't click on the parent JFrame until the dialog is closed.

An easy way to create a modal JDialog is to use a JOptionPane. It has some predefined methods that make prompting for user input easy.

For example in your case you could do something like:

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;

public class SSCCE extends JPanel
{
    private int scoringOption = -1;

    public SSCCE()
    {
        JButton button = new JButton("Change Points Option");
        add(button);

        button.addActionListener((e) -> displayOptionDialog());
    }

    private void displayOptionDialog()
    {
        Window window = SwingUtilities.windowForComponent( this );

        // Custom button text

        Object[] options = {"Option 1: Same Points Per Hit", "Option 2: Unique Points Per Hit"};

        scoringOption = JOptionPane.showOptionDialog(
            window,
            "Select your scoring option:",
            "Scoring Option",
            JOptionPane.YES_NO_CANCEL_OPTION,
            JOptionPane.QUESTION_MESSAGE,
            null,
            options,
            null);

        System.out.println( scoringOption );
    }

    private static void createAndShowGUI()
    {
        JFrame frame = new JFrame("SSCCE");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.add(new SSCCE());
        frame.pack();
        frame.setLocationByPlatform( true );
        frame.setVisible( true );
    }

    public static void main(String[] args) throws Exception
    {
        java.awt.EventQueue.invokeLater( () -> createAndShowGUI() );
    }
}

The above is also an example of an "MRE". The code is simple and contained in a single class that you can copy/paste/compile and test.

Read the section from the Swing tutorial on How to Use Dialogs for more examples of using a JOptionPane.

If you really want to use radio buttons, then you can create a panel with the radio buttons and display them on the option pane using the showConfirmDialog(...) method. When the dialog closes you would then need to get the action command from the ButtonModel of the ButtonGroup.

See: how to set & manage the layout of JOptionPane for a basic example of this approach to get you started.

JFrames are not modal -- you create one and display it, it does not block the code flow, and so you are extracting the value of scoreOption right as the JFrame is being displayed and before the user has had any chance to change it. You need to use a modal dialog such as a JDialog that is created as a modal dialog or use a JOptionPane (which is actually just a modal JDialog under the hood). This will block the flow of code so that you extract the data only after it has been changed by the user.

An example that proves the point:

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;

public class FooGui01 extends JPanel {
    private String frameTest = "";
    private String dialogTest = "";
    private JFrame mainFrame = new JFrame("Main GUI");
    
    private JFrame subFrame;
    private JDialog dialog;

    
    public FooGui01() {
        JButton showFrameBtn = new JButton("Show JFrame");
        showFrameBtn.addActionListener(e -> {
            changeTest1WithJFrame();
            System.out.println("frameTest: " + frameTest);
        });
        
        JButton showDialogBtn = new JButton("Show JDialog");
        showDialogBtn.addActionListener(e -> {
            changeTest2WithModalDialog();
            System.out.println("dialogTest: " + dialogTest);
        });
        
        JPanel panel = new JPanel();
        panel.add(showDialogBtn);
        panel.add(showFrameBtn);
        
        mainFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        mainFrame.add(panel);
        mainFrame.pack();
        mainFrame.setLocationByPlatform(true);
        mainFrame.setVisible(true);
        
    }
    
    public void changeTest1WithJFrame() {

        if (subFrame == null) {
            subFrame = new JFrame("Frame");
            JButton button = new JButton("Press me");
            button.addActionListener(e -> {
                frameTest = "Hello World and frameTest";
                subFrame.setVisible(false);
            });

            JPanel panel = new JPanel();
            panel.add(button);
            
            subFrame.setDefaultCloseOperation(JFrame.HIDE_ON_CLOSE);
            subFrame.add(panel);
            subFrame.pack();
            subFrame.setLocationByPlatform(true);
        }
        subFrame.setVisible(true);
    }
    
    public void changeTest2WithModalDialog() {
        
        if (dialog == null) {       
            dialog = new JDialog(mainFrame, "Dialog", Dialog.ModalityType.APPLICATION_MODAL);
            JButton button = new JButton("Press me");
            button.addActionListener(e -> {
                dialogTest = "Hello World and dialogTest";
                dialog.setVisible(false);
            });

            JPanel panel = new JPanel();
            panel.add(button);
            
            dialog.add(panel);
            dialog.pack();
            dialog.setLocationByPlatform(true);
        }
        dialog.setVisible(true);
    }
        
    public static void main(String[] args) {
        SwingUtilities.invokeLater(() -> new FooGui01());
    }
}

If you run the code, when you show the sub JFrame, the test text is displayed immediately in the console before the dialog has been dealt with. If you press the button to show the dialog, the test text display is delayed until after the button has been pushed, changing the text.

Pressing the frame button twice will finally show the correct text since the text was set by the first time it was displayed.