As a simple proof of concept, I tried:

. ~/.bashrc
set -m
vim $arg

and:

arg=file-to-edit xterm -e bash --init-file vimbashrc

which seems to behave more or less as you specified, and allows you to pass arguments to your interactive command.

So, using a tempfile would probably result in something like

#!/bin/bash

if [ "$1" = "" ] ; then
        . ~/.bashrc
        set -m
        (IFS=$'n'; vi $(cat "$tmpfile"))
else
    tmpfile=$(mktemp /tmp/vimstart.XXXXXX)
    for arg in "$@" ; do
        echo "$arg" >> $tmpfile
    done
    tmpfile=$tmpfile xterm -e bash --init-file ./vimstart
    rm "$tmpfile"
fi

Because the filename may contain n, you can also use the NULL character as separator:

#!/bin/bash

if [ "$1" = "" ] ; then
    . ~/.bashrc
    set -m
    (IFS=$'0'; vi $(cat "$tmpfile"))
else
    tmpfile=$(mktemp /tmp/vimstart.XXXXXX)
    for arg in "$@" ; do
        echo -n "$arg" >> $tmpfile
                echo -n $'x00' >> $tmpfile
    done
    tmpfile=$tmpfile xterm -e bash --init-file ./vimstart
    rm "$tmpfile"
fi

I could think of a couple of methods, and I think the first one is less hackish on Bash mainly because it seems (to me) to have quirks that can be more easily handled. However, as that may also be a matter of taste, I'm covering both.

Method one

The "pre-quoting" way

It consists in making your script expand its $@ array, thus doing it on behalf of the inner interactive bash, and you might use the /dev/fd/X file-descriptor-on-filesystem facility (if available on your system) as parameter to the --init-file argument. Such file-descriptor might refer to a Here String where you would handle the filenames, like this:

xterm -e bash --init-file /dev/fd/3 3<<<". ~/.bashrc; set -m; vim $@"

One bonus point of this file-descriptor-on-filesystem trick is that you have a self-contained solution, as it does not depend on external helper files such as your .vimbashrc. This is especially handy here because the contents of the --init-file is dynamic due to the $@ expansion.

On the other hand it has a possible caveat wrt the file-descriptor's actual persistence from the script all the way through to the inner shell. This trick works fine as long as no intermediate process closes the file-descriptors it received from its parent. This is common behavior among many standard tools but should e.g. a sudo be in the middle with its default behavior of closing all received file-descriptors then this trick would not work and we would need to resort to a temporary file or to your original .vimbashrc file.

Anyway, using simply $@ as above would not work in case of filenames containing spaces or newlines because at the time the inner bash consumes the sequence of commands those filenames are not quoted and hence the spaces in filenames are interpreted as word separators as per standard behavior.

To address that we can inject one level of quoting, and on Bash's versions 4.4 and above it is only a matter of using the @Q Parameter Transformation syntax onto the $@ array, like this:

xterm -e bash --init-file /dev/fd/3 3<<<". ~/.bashrc; set -m; vim ${@@Q}"

On Bash's versions below 4.4 we can obtain the same by using its printf %q, like this (as Here Document for better readability, but would do the same as a Here String like above):

printf -v files ' %q' "$@"
xterm -e bash --init-file /dev/fd/3 3<<EOF
. ~/.bashrc
set -m
vim $files
EOF

On a side-note, depending on your setup you might consider sourcing /etc/bash.bashrc too, prior to the user's .bashrc, as that is Bash's standard behavior for interactive shells.

Note also that I left the set -m command for familiarity to your original script and because it's harmless, but it is actually superfluous here because --init-file implies an interactive bash which implies -m. It would instead be needed if, by taking your question's title literally, you'd wish a job-controlling shell yet not a fully interactive one. There are differences in behavior.

Method two

The -s option

The Bash's (and POSIX) -s option allows you to specify arguments to an interactive shell just like you do to a non-interactive one¹. Thus, by using this -s option, and still as a self-contained solution, it would be like:

xterm -e bash --init-file /dev/fd/3 -s "$@" 3<<'EOF'
. ~/.bashrc
set -m  # superfluous as bash is run with `--init-file`; you would instead need it for a job-controlling yet "non-interactive" bash (ie no `--init-file` nor `-i` option)
exec <<<'exec < /dev/tty; vim "$@"'
EOF

Quirky things to note:

1. the Here Document's delimiter specification must be within single quotes or the $@ piece inside the Here Document would be expanded by your script (without correct quoting etc.) instead of by the inner bash where it belongs. This is as opposed to the "pre-quoting" method where the Here Document's delimiter is deliberately non-quoted
2. the Here String (the exec <<<... stdin redirection piece) must be the single-quoted type as well², or the "$@" piece inside it would be expanded by the inner bash at a time when its $@ array is not yet populated
3. specifically, we need such stdin redirection (the one made via the exec <<< Here String) as a helper just to make the inner bash "defer" the execution of the commands needing a fully populated $@ array
4. inside such helper stdin redirection (the Here String piece) we need to make the inner bash redirect its own stdin again, this time back to its controlling terminal (hence the exec < /dev/tty line) to make it recover its interactive functionality
5. we need all commands meant to be executed after the exec < /dev/tty (ie the vim "$@" here) to be specified on the same line³ as the exec < /dev/tty because after such redirection the Here String will no longer be read⁴. In fact this specific piece looks better as a Here String, if it can be short enough like in this example

This method may be better used with an external helper file like your .vimbashrc (though dropping the self-contained convenience) because the contents of such file, with regard to the filename-arguments problem, can be entirely static. This way, your script invoked by the file-manager would become as simple as:

xterm -e bash --init-file .vimbashrc -s "$@"

and its companion .vimbashrc would be like:

. ~/.bashrc
#  (let's do away with the superfluous `set -m`)
exec <<<'exec < /dev/tty && vim "$@"'  # let's also run vim only if the redirection to the controlling tty succeeded

The companion file still has the quirks but, besides any "cleanliness" consideration, one possible bonus point of this latter version (non self-contained) is that its whole xterm -e ... command, away of the "$@" part, could be used directly by your file-manager in place of your "script", if it were so kind to allow you to specify a command which it dutifully splits on spaces to produce the canonical "argv" array along with the filenames arguments.

Note also that this whole -s method, in all its versions, by default updates the user's .bash_history with the commands specified inside the helper stdin redirection, which is why I've been keeping that specific piece as essential as possible. Naturally you can prevent such updating by using your preferred way, e.g. by adding unset HISTFILE in the --init-file.

--

Just as a comparison, using this method with dash would be far more convenient because dash does populate the $@ array for the script specified by the ENV environment variable, and thus a self-contained solution would be as simple as:

xterm -e env ENV=/dev/fd/3 dash -s "$@" 3<<'EOF'  # `dash` run through `env` just to be positive that `ENV` is present when dash starts
vim "$@"
EOF

HTH

---

¹_{with the exception that $0 can't be specified, as opposed to when invoking a shell with -c option}

²_{if you used an additional Here Document it would need to have its delimiter quoted as well}

³_{actually on the same "complete_command" as defined by POSIX, meaning that you can still span multiple lines as long as they are part of the same "complete_command", e.g. when such lines have the line-continuation backslash or are within a compound block}

⁴_{this should be standard behavior, presumably deriving from the first, brief, paragraph of this sub-section}