How to enable org mode and evil mode for untitled buffer when creating a new frame

All you have to do is replace the set-buffer-major-mode call which sets the buffer to a default mode (fundamental mode in this case) with a couple of calls to set the modes you want.

org-mode is a major mode, so you set it by calling the function that implements it, with no arguments, in the buffer whose mode you want to set:

(set-buffer <some buffer>)
(org-mode)

evil-mode is a global minor mode, which can be enabled or disabled at will, like all minor modes. To enable it, you have to call its function with a positive argument:

(evil-mode 1)

To disable it, you call the function with a negative argument.

All these things and more can be glimpsed by checking the doc string of the functions: C-h f org-mode RET and C-h f evil-mode RET. If a function does not behave the way you expect, you probably have a wrong expectation and you can check that by reading its doc string, so C-h f <function> RET is something you should do often.

Putting it all together, the result is:

(defun new-empty-frame ()
  "Create a new frame with a new empty buffer."
  (interactive)
  (let ((buffer (generate-new-buffer "untitled")))
    (set-buffer buffer)
    (org-mode)
    (evil-mode 1)
    (display-buffer buffer '(display-buffer-pop-up-frame . nil))))

Only partially tested, since I don't have evil-mode installed.