How does autoregression work in R?

You may use ar.ols to estimate this non-stationary series.

From the docs,

ar.ols fits the general AR model to a possibly non-stationary and/or multivariate system of series x.

Example with your data:

ar.ols(x,demean=FALSE)

Call:
ar.ols(x = x, demean = FALSE)

Coefficients:
1  2  
1  1 

As I said in my comment, you are going toward a wrong direction.

ar is assuming x as a stationary process AR(p). The default estimation method "yule-walker" is a moment estimator. Please see Yule-Walker equations of autoregressive process for more.

ar selects order p by minimizing AIC. For you example Fibonacci sequence x, it has selected p = 1. The resulting coefficient, by Yule-Walker equations, matches the sample ACF at lag 1:

z <- acf(x, lag.max = 1)
#    0     1
#1.000 0.553

Since model assumption is wrong, you definitely can not get c(1, 1) as the answer. A crude way to get you to the right estimation is using least squares linear regression:

N <- length(x)
y <- x[3:N]
x1 <- x[2:(N-1)]    # lag-1 
x2 <- x[1:(N-2)]    # lag-2

lm(y ~ x1 + x2 - 1)  ## drop intercept (as you know it for sure)
#Coefficients:
#x1  x2  
# 1   1