I studied or was studying?

I would say 'I studied' because this is a completed past event. You consider it as a fact. But if you want to emphasize the duration of your studies, you can use the other sentence /with the Past Continuous Tense/, but it is optional.

Your first sentence actually can mean what you want, except that it wouldn't normally be interpreted that way. On reading it, the natural assumption is as you indicate. So, practically speaking, it will be misinterpreted.

But the actual way of explicitly referring to the previous 5 years—assuming you wanted to—would be:

I have been studying at Oxford for (the past) 5 years.

This is the meaning people will get out your first sentence without being forced to think of it in the way you want.

Another note is that you do need the preposition. Without for the sentence doesn't make sense.

---

Your second sentence is fine.

It can also be emphasized:

I once studied at Oxford for 5 years.
In the past, I studied at Oxford for 5 years.
In my youth, I studied at Oxford for 5 years.

---

You could also rephrase the sentence to avoid the verb study altogether.

I had a 5-year period of study at Oxford.

---

Last, you can't use your final sentence as it stands.

I was studying at Oxford for 5 years.

This is an incomplete thought. Because of how this verb tense works, you need to provide a concluding thought. On hearing this, people would ask, "Yes, and then what?"

Something else has to follow:

I was studying at Oxford for 5 years until the school mysteriously burned down.

---

In response to a comment, I have to mention an oddity of English.

This question and answer exchange is fine:

Q: "I see there's a gap in your travels from 2000 to 2005. Were you (studying) at school?"
A: "(Yes,) I was studying at Oxford."

As is this one:

Q: "What were you doing at Oxford from 2000 to 2005?"
A: "I was studying (at Oxford)."

If asked as a question or in response to a question, the past continuous is fine. So, context plays an important role.

Unfortunately, it means that you can't tell the appropriateness or inappropriateness of some sentences outside of context.

You went to Oxford and you are asking me? :)

But since you asked, i would vote for #2. it is a completed event, way in the past. And you aren't relating a story. You aren't saying "when i was studying at Oxford..." it would be by the way awkward to throw in 5 years at that exact moment..