Earliest I've been to work

Since you're saying, "[This is the] earliest I have been to work," I would say yes, this is correct. You're specifically referring to yourself and the past.

Whether you're on time or not is irrelevant to the actual statement since the subject of the sentence is you getting to work, not the time your work begins.

There is a logical and a linguistic answer to your question.

If you wanted to make your statement completely precise, you might say something like this:-

The time at which I arrived today was the earliest of all the times at which I have ever arrived.

But that precision is bought at the expense of making it stilted and laboured.

Your statement is, strictly, a sentence fragment with no main verb. So you would not be well-advised to such a casually unrepentant justification to your enraged boss.

Nevertheless, it is clear. It is a sort of ellipsis, which takes for granted something like

[My arrival today was the] earliest [time at which] I’ve ever been to work.

This is not to say that the works in angle brackets passed through the speaker’s (or listener’s) mind unspoken!

This is the earliest I've been to work.

That is strictly correct. Despite the fact that you were late, it is the earliest actual time you've arrived.

But if you want to make a specific point about still being late, you could instead say:

The is the least late I've been to work.

At least in US English, "been" wouldn't be narrow enough for what you're trying to convey. So here are some slight variants of what you proposed, that would be clearer:

This is the earliest I've ever gotten to work or ever started work or ever arrived at work

These seem fine. However, I would say: This is the closest to being on time I've ever been!