How to get Blender 2.80 to render through an ssh connection? [Minimal working example provided]

virtual x screen tool:xvfb work for me

sudo apt-get install -y xvfb

xvfb-run --auto-servernum --server-args="-screen 0 1600x1024x16" blender -b -o /tmp/ -E BLENDER_EEVEE -f 1

I figured out how to answer my question. The main source that I relied on was this wonderful answer (and then this follow-up) in the Blender devtalk thread that I referenced to in my original question. (Those answers were posted after I made this question here.) Here's my answer which somewhat cleans up their explanations, but credit goes to those two who answered on devtalk.

Here are the steps that I followed. Here, I assume that the remote machine has an NVIDIA GPU, is running Ubuntu 18.04, and has Blender 2.82a installed, with the blender command runnable.

(1) I ssh into my remote machine. I do NOT use X forwarding, so I do not use ssh -X.

(2) Running glxinfo returns this error:

$ glxinfo
Error: unable to open display

If you don't have glxinfo, run sudo apt install mesa-utils.

(3) To fix the "unable to open display" error, first run the following to see available GPU(s):

$ nvidia-xconfig --query-gpu-info
Number of GPUs: 1

GPU #0:
  Name      : TITAN X (Pascal)
  UUID      : GPU-9adbb93a-e6c6-b54c-651c-66d1414a6b82
  PCI BusID : PCI:1:0:0

You may get other outputs for additional GPUs (for multi-GPU machines) and for monitors if there are attached monitors. For now we don't worry about those. We just pick the GPU we want, and look at the PCI BusID value.

(4) Next, we run this, where I insert the PCI BusID into the --busid argument:

$ sudo nvidia-xconfig --busid=PCI:1:0:0 --use-display-device=none --virtual=1280x1024
[sudo] password for user:

Using X configuration file: "/etc/X11/xorg.conf".
Backed up file '/etc/X11/xorg.conf' as '/etc/X11/xorg.conf.backup'
New X configuration file written to '/etc/X11/xorg.conf'

According to devtalk, this will "create a virtual x screen" and "create[s] the xorg.conf file. Xserver will read this file and setup the xorg server accordingly".

(5) Now, run sudo Xorg :1. If you get an error like I originally had:

$ sudo Xorg :1
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
(EE)
Fatal server error:
(EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE)
(EE)
Please consult the The X.Org Foundation support
     at http://wiki.x.org
 for help.
(EE) Please also check the log file at "/var/log/Xorg.1.log" for additional information.
(EE)
(EE) Server terminated with error (1). Closing log file.

Then run ps -C Xorg to see what Xorg programs are already running. Then use sudo kill -9 [pid].

(6) A working command produces this output for me:

$ sudo Xorg :1

X.Org X Server 1.20.4
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-148-generic x86_64 Ubuntu
Current Operating System: Linux takeshi 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.3.0-40-generic root=UUID=99a59366-4621-4034-b183-bfaf2a8af7eb ro quiet splash vt.handoff=1
Build Date: 02 May 2019  08:06:54AM
xorg-server-hwe-18.04 2:1.20.4-1ubuntu3~18.04.1 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.34.0
    Before reporting problems, check http://wiki.x.org
    to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
    (++) from command line, (!!) notice, (II) informational,
    (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.1.log", Time: Mon Apr 27 12:06:11 2020
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"

Now, this will be running in one shell. Using CTRL+C will not work to kill it. For your subsequent work, you can use a separate shell.

(7) For this, I decided to do another ssh command to enter my remote machine. This step is critical: export DISPLAY=:1. This will set the display variable to be what the Xorg command did in the other shell.

(8) As a sanity check, make sure that glxinfo returns expected output:

$ glxinfo | grep -e render -e NVIDIA
direct rendering: Yes
server glx vendor string: NVIDIA Corporation
client glx vendor string: NVIDIA Corporation
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: TITAN X (Pascal)/PCIe/SSE2
OpenGL core profile version string: 4.6.0 NVIDIA 430.50
OpenGL core profile shading language version string: 4.60 NVIDIA
    GL_ARB_conditional_render_inverted, GL_ARB_conservative_depth,
    GL_NVX_conditional_render, GL_NVX_gpu_memory_info, GL_NVX_nvenc_interop,
    GL_NV_command_list, GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_path_rendering_shared_edge,
    GL_NV_shader_thread_shuffle, GL_NV_stereo_view_rendering,
OpenGL version string: 4.6.0 NVIDIA 430.50
OpenGL shading language version string: 4.60 NVIDIA
    GL_ARB_compute_variable_group_size, GL_ARB_conditional_render_inverted,
    GL_NVX_conditional_render, GL_NVX_gpu_memory_info, GL_NVX_nvenc_interop,
    GL_NV_command_list, GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_path_rendering_shared_edge,
    GL_NV_shader_thread_shuffle, GL_NV_stereo_view_rendering,
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 430.50
    GL_EXT_multisampled_render_to_texture,
    GL_EXT_multisampled_render_to_texture2, GL_EXT_occlusion_query_boolean,
    GL_EXT_render_snorm, GL_EXT_robustness, GL_EXT_sRGB,
    GL_NV_clip_space_w_scaling, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_path_rendering_shared_edge,
    GL_NV_shadow_samplers_cube, GL_NV_stereo_view_rendering,
    GL_OES_element_index_uint, GL_OES_fbo_render_mipmap,
    GL_OVR_multiview_multisampled_render_to_texture

(9) Finally, I run my ultimate test: blender -b -E BLENDER_EEVEE -f 1 to see if the rendering works. If all goes well, you'll see this output:

$ blender -b -E BLENDER_EEVEE -f 1
Blender 2.82 (sub 7) (hash 375c7dc4caf4 built 2020-03-12 05:30:40)
found bundled python: /home/user/blender-2.82a-linux64/2.82/python
Fra:1 Mem:88.40M (0.00M, Peak 88.68M) | Time:00:00.32 | Syncing Cube
Fra:1 Mem:88.72M (0.00M, Peak 89.01M) | Time:00:00.32 | Syncing Light
Fra:1 Mem:88.72M (0.00M, Peak 89.01M) | Time:00:00.32 | Syncing Camera
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.33 | Rendering 1 / 64 samples
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.37 | Rendering 26 / 64 samples
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.39 | Rendering 51 / 64 samples
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.40 | Rendering 64 / 64 samples
Fra:1 Mem:48.47M (0.00M, Peak 89.01M) | Time:00:00.43 | Sce: Scene Ve:0 Fa:0 La:0
Saved: '/tmp/0001.png'
 Time: 00:00.75 (Saving: 00:00.31)


Blender quit

I am able to secure-copy (using scp) that image onto my local machine, and I can confirm that it looks correct.

(10) Once I'm done, the last thing I do is run nvidia-smi to see what processes are there:

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     22948      G   /usr/lib/xorg/Xorg                            14MiB |
|    0     22976      G   /usr/bin/gnome-shell                           6MiB |
|    0     23332      G   /usr/lib/xorg/Xorg                            10MiB |
|    0     30462      C   python                                       847MiB |
+-----------------------------------------------------------------------------+

Then I can run the usual sudo kill -9 [pid] where the process ID comes from above. I think one of the two Xorg commands will constantly restart itself so it may not be easy to kill, but I will need to do more investigation.

Final comments: thanks to @sambler in the comments, I now know that to test the rendering, it is easier to use these one-line commands:

blender -b -E BLENDER_WORKBENCH -f 1
blender -b -E BLENDER_EEVEE -f 1
blender -b -E CYCLES -f 1 

rather than the python script that I was using in my original question. These use all three of Blender's available rendering engines. The last command, using CYCLES, will work remotely without the bells-and-whistles here, while the first two will not. But the downside is that CYCLES is much slower than the other rendering engines.

I did another test where I enter GNU screen (via screen command) and ran the same one-line blender testing command above. It seems to work successfully, suggesting that one can run in screen and then log out of the machine, and have images repeatedly be generated.

You can also run the Xorg command in a screen shell so that you can exit it, and the command will just run permanently until you turn it off. I was able to do this and run the Blender tests successfully. This is probably my preferred setup since it doesn't require me to explicitly leave a "tab" open (e.g., a tab in iTerm2, which I use to ssh and to program).

Finally, for faster iterations, you can combine the display and the blender commands:

DISPLAY=:1  blender -b -E BLENDER_EEVEE -f 1

and this should also work.

Update 1: sometimes, step (6) does not work. I've noticed that if you log into a remote machine that does not already have an X-server running (which I think we can tell by running nvidia-smi and checking that there exists an Xorg process running) then you might get this error:

$ sudo Xorg :1

X.Org X Server 1.20.4
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-148-generic x86_64 Ubuntu
Current Operating System: Linux mason 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:12:28 UTC 2019 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-23-generic root=UUID=877a0d47-b6cc-4687-8778-3a947fc7cb04 ro quiet splash vt.handoff=1
Build Date: 02 May 2019  08:06:54AM
xorg-server-hwe-18.04 2:1.20.4-1ubuntu3~18.04.1 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.34.0
    Before reporting problems, check http://wiki.x.org
    to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
    (++) from command line, (!!) notice, (II) informational,
    (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.1.log", Time: Wed Apr 29 10:11:48 2020
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
(EE)
Fatal server error:
(EE) no screens found(EE)
(EE)
Please consult the The X.Org Foundation support
     at http://wiki.x.org
 for help.
(EE) Please also check the log file at "/var/log/Xorg.1.log" for additional information.
(EE)
(EE) Server terminated with error (1). Closing log file.

In this case, starting up a normal X server should work. However, I have yet to get this working. The above error message with a machine that has GPUs, but does not have an attached monitor, whereas I got mine working from another machine that did have a monitor. I think the fix is to consult these references (source 1, source 2, source 3).

Update 2 (July 21, 2020): I tried running:

sudo nvidia-xconfig --busid=PCI:1:0:0 --virtual=1280x1024  --use-display-device="DFP-0"

and I have been able to fix the above error with that. I would suggest starting with the above if that works.

To answer a few of your questions: ssh -x is indeed forwarding X-11 (X-Window) traffic from your remote machine to your local one. It assumes you have a local display. If you were to ssh to machine A and then from there ssh -x to machine B, you would not have a display. It might be best to start on your machine with a display and ssh to machine B directly.

Blender is primarily a GUI application and I'm not too familiar with running it without displaying graphics. However, it seems --background should be doing that. But, clearly it is not keeping it entirely from running without communicating to X-11. So, there may be something to add to the python program to get it to exit blender when completed, instead of maybe starting up the GUI. Like:

import sys
sys.exit(0)

Another option: Do you need to run blender remotely or can you do the processing on one machine and copy the results (scp) to other machines?