What was the first console or handheld to have a user interface outside of games?

It's pretty hazy by what you define as a user interface. Judging by the criteria you laid out in the comments then the Sega Saturn (1994) is the first console to have a home dashboard with various options including playing games, music, and console settings.

User Timmy Jim was correct, stating:

Not sure if it's the first, but certainly a very early contender - the Sega Saturn from around the mid 90s had an interface that would load up if you didn't have a disc in I believe

User Fana also pointed out:

After some research, I've found this video that shows a Phillips CDi interface (from 1992).

It has a couple of options (like checking the memory cards, changing language / options, etc ...). I don't know if you would accept this as an "user interface".

Before this, the earliest main menu I could find was the Sega CD (1991), but that shows only a Press Start which does not meet your criteria.

Earlier gaming devices were not consoles but actual computers such as the Apple II (1977) and the Commodore 64 (1982) but did accept inputs to start games. Neither of these are 'video game consoles' and thus do not fit your criteria.

A notable console is the Atari 2600 (1977) for having the first screensaver. While this could hardly be considered an interactive dashboard, it would turn on after a period of inactivity, change colors to prevent screen burn-in, and would turn off after user input. If we're playing loose with definitions, this is the first menu of any kind on a dedicated gaming console.