How to solve Nonlinear coupled ODEs using DSolve

It seems that the equations has no explicit solution, we should use NDSolve or NDSolveValue

Clear["`*"];
    Clear[Derivative];
    L = 10;
    sol = NDSolve[{y''[t] == 
        0, -1 + x[t] + x[t]*(y'[t]^2 + y''[t]^2) - 2 x''[t] == 0, 
       x'[0] == 0, x'[L] == 0, y[0] == 0, x[L]^2*y'[L] == 100}, {y, 
       x}, {t, -5, 5}]
    ParametricPlot[{x[t], y[t]} /. sol, {t, -5, 5}, 
     AspectRatio -> Automatic]

The problem with the initial/B.C. conditions. It does not look like there is a real solution that satisfies them.

This below solves the ODE's and give 3 equations in 3 constants of integrations.

If it is possible to solve these 3 equations, then you can obtain the general solution. But Mathematica says there is no real solution. So may be you should examine how you obtained these ODE's with such BC.

Solve the first ode on its own, with one IC only.

Take this solution and plug it into the second ODE. Then solve the resulting second ODE with no IC's.

So now the solution for the second ODE contains 3 constants of integrations. One from the first solution (since we only used one IC there) and two from the second ODE since we did not use any IC.

Now setup 3 equations using those not used IC's. And try to solve them.

ClearAll[y, x, t];
L = 10;
ode1 = y''[t] == 0;
ic11 = y[0] == 0;
ic12 = x[L]^2*y'[L] == 100;
soly = DSolve[{ode1, ic11}, y, t][[1, 1]]

ode2 = -1 + x[t] + x[t]*(y'[t]^2 + y''[t]^2) - 2 x''[t] == 0;
ode2 = ode2 /. soly

ic12 = ic12 /. soly

ic21 = x'[0] == 0
ic22 = x'[L] == 0
solx = DSolve[ode2, x, t][[1, 1]]

Now setup 3 equations from the 3 remaining IC's

 eq1 = ic12 /. solx

 eq2 = ic21 /. solx

 eq3 = ic22 /. solx

 Solve[{eq1, eq2, eq3}, {C[2], C[3], C[4]}]
 (* waited too long *)

 Solve[{eq1, eq2, eq3}, {C[2], C[3], C[4]}, Reals]
 (* {} *)

So the problem is now is changed to solving 3 equations in 3 unknowns. If you can solve these equations, then you have your solution. FindInstance can find one solution

solIC = FindInstance[{eq1, eq2, eq3}, {C[2], C[3], C[4]}] 
N[solIC]

(* {{C[2] -> -0.0353443 - 1.03537 I, C[3] -> 0., C[4] -> 0.}} *)

So that is one solution. Hence the solutions are

soly /. solIC

solx /. solIC

Which is the same as

Verify the solutions:

ode1 /. soly
(*True*)
ode2 /. solx /. solIC
(*True*)

Also, all IC's are verified true.

So bottom line is, the solutions are

  y(t) = t (-0.0353443 - 1.03537 I)
  x(t) = -(1/(-1 - (-0.0353443 - 1.03537 I)^2))

I think DSolve could not solve it, since it could not find solution using Solve for the constants of integrations.

Noticed also NDSolve have hard time with your BC/IC

ClearAll[y, x, t];
L = 10;
ode1 = y''[t] == 0;
ic11 = y[0] == 0;
ic12 = x[L]^2*y'[L] == 100;
ode2 = -1 + x[t] + x[t]*(y'[t]^2 + y''[t]^2) - 2 x''[t] == 0;
ic21 = x'[0] == 0;
ic22 = x'[L] == 0;
NDSolve[{ode1, ode2, ic11, ic12, ic21, ic22}, {x, y}, {t, 0, 1}]