Debugging an Expo project with Attach to Debugger option throws t.RawDebugSession.handleErrorResponse error after a while or saving changes

This is becoming really annoying & time-consuming behaviour of VS Code that affect my coding experience very badly.

I usually debug my Expo project by using the Attach to Debugger option in VS Code.
Everything seems good for a while but after some time passes, as soon as I hit the Reload button in my Expo Developer menu (accessing Cmd + D in simulator), my lovely bottom orange bar suddenly goes back to blue and a Chrome page pops up addressing http://localhost:19001/debugger-ui/.

Also in the Output pane, below errors are thrown:

[2020-09-04 21:45:54.816] [renderer3] [error] timeout after 1000 ms: Error: timeout after 1000 ms
    at t.RawDebugSession.handleErrorResponse (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:3209:226)
    at file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:3208:532
    at async t.RawDebugSession.shutdown (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:3206:536)

At that point, I have to manually stop the debugger, stop Remote Debugging in Expo Developer menu, start the debugger and start Remote Debugging again over and over.

My launch.json looks like this:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Attach to packager",
            "cwd": "${workspaceFolder}",
            "type": "reactnative",
            "request": "attach",
            "port": "19001",
            "sourceMaps": true,
        }
    ]
}

I came across a similar Github issue relating debugger restart but I do my reload in Expo developer menu, so it doesn’t really address my case.

So, how can i get rid of these debugger issues and keep my debugging experience uninterrupted?