How can the angular velocity vector be obtained from angular displacement which is not a vector?

This answer is probably too advanced for the OP, but might be interesting to those who have studied more advanced math or physics. (I mean this entirely respectfully.)

The "best" way to mathematically represent a (proper) rotation in 3D space is not by a vector, but instead by an element of the Lie group $mathrm{SO}(3)$. $mathrm{SO}(3)$ is a three-dimensional Lie group just like the vector space $mathbb{R}^3$ (with vector addition as the group operation), although it has a different global topology. It is a "better" representation than $mathbb{R}^3$ because rotations compose correctly, which is not the case for the vector representation (as you illustrate in the OP). Unlike $mathbb{R}^3$, $mathrm{SO}(3)$ is nonabelian - the order of operations (in this case, rotations) makes a difference.

Infinitesimal rotations are a qualitatively different beast, and are represented by the elements of the Lie algebra $mathfrak{so}(3)$ (note the different font and capitalization). $mathfrak{so}(3)$ is indeed a vector space and is isomorphic to $mathbb{R}^3$. (However, it has some additional algebraic structure in the form of the Lie bracket, which geometrically corresponds to the cross product and contains the information that reflects the failure of finite rotations to commute). So infinitesimal rotations really are vectors, and can be equivalently added together in any order.

There's two possible views here.

• One of these is that you can, indeed, consider angular displacement or position as a vector in that it can be encoded with one: if you have $$mathbf{Theta} := left<theta_x, theta_y, theta_zright>$$ then you can declare that this encodes an angular displacement with axis $mathbf{hat{Theta}}$ and with rotational angle $$theta = sqrt{theta_x^2 + theta_y^2 + theta_z^2}$$ There is nothing wrong with this, mathematically. And, indeed, if the angular displacement vector grows in mangnitude with time, then the derivative is the angular velocity $mathbf{Omega}$. It's just that, if you choose to do so, then the vectorial addition $mathbf{Theta}_1 + mathbf{Theta}_2$ of arbitrary angular displacements will not be the same as composition of the rotational transformations they represent, but some other operation for which what it represents I am not sure right now.

• The second view is to understand the difference between displacement and position, and that position in general is not a vector to begin with. Position is a point, or better, a numerical (i.e. coordinates) way of denoting points, and while you can "get away with" treating it as a vector in some cases, you cannot in others, e.g. on a curved manifold. Displacements are vectors derived from positions.

  Positions and vectors, are, if you like (and there is actually a foundational mathematical theory, but unfortunately it doesn't seem to be too well used, called "type theory" for this) belong to different "data types", with different semantic denotations and different operations you can and cannot perform them. In paritcular, while both positions and vectors can be represented as tuples of real numbers drawn out of the sets $mathbb{R}^n$, you cannot add or scale positions, but you can vectors.

  Indeed, typically you can't "do" anything to positions except to compare them in some fashion - whether that be in terms of linear ordering, as in one dimension, i.e. the real line - or in terms of measuring the distance between them (which is what a metric space or metrical manifold [Riemannian etc.] is about). But you can "do" things to vectors - add, scale, etc. .

  And this, I believe, is the better way to make sense of what is going on here. In the particular case of positions in Euclidean space, which is what is called an "affine space", you do also have the option of subtracting, to get vectorial displacements $mathbf{d} = Q - P$ for points $P$ and $Q$. However, this does not work on curved manifolds in any natural way, because of their nonlinear geometry - unless you are talking about a displacement who is so small that the curvature can be neglected and the manifold considered flat. That is, on curved manifolds, you can only differentiate a curvilinear displacement, which is in most proper sense a curve, or function, between the points, i.e. $gamma: [0, 1] rightarrow M$, such that $gamma(0) = P$ and $gamma(1) = Q$.

  And angular position, turns out, most honestly "lives" in a curved manifold, and does not live in Euclidean space - instead, it "lives" on a manifold that is a form of the "real projective space" $mathbf{RP}^3$, and can be constructed more explicitly using a convention such as the Euler angles or Tait-Bryan angles. That is, a point in angular position manifold is given by $$(psi, theta, phi)$$ where these are the three Euler angles (personally, I find the Tait-Bryan angles ['roll/pitch/yaw'] more intuitive).

  That said, there is, then, also a corresponding rigorous notion of angular displacement - but this is because the way these angular positions act to rotate the object is through the action of the corresponding rotation matrix, and those rotation matrices can be composed. The resulting structure is thus an algebraic group, not a vector space; it is the Lie group $mathrm{SO}(3)$. And you can write its displacement elements in the above form, too, with angles, but the composition in terms of the angles is not simple addition thereof ala a vector space, hence they are not "vectors", but their own thing which is specific to angle space. Nonetheless, they can still be differentiated to vectors, as you have observed - this is because any curve $gamma$ on a differentaible manifold will differentiate to a vector in the tangent spaces $T_xM$.

The key is in the parenthetical statement in your first block quote: focus on the "unless they are very small" part. This can be seen by doing the simple "experiment" below. While this answer is not mathematically rigorous (for the rigor, see @The_Sympathizer's answer), I think it gets at the heart of the idea behind how we can get angular velocity as a vector from angular displacements. This experiment is from a "discussion question" exercise from Sears & Zemansky's "University Physics with Modern Physics" 13th edition:

Although angular velocity and angular acceleration can be treated as vectors, the angular displacement $theta$, despite having a magnitude and direction, cannot. This is is because $theta$ does not follow the commutative law of vector addition.

Prove this to yourself in the following way: Lay your physics textbook flat on the desk in front of you with the cover side up so that you can read the writing on it. Rotate it through $90^circ$ about a horizontal axis so that farthest edge comes toward you. Call this angular displacement $theta_1$. Then rotate by $90^circ$ about a vertical axis so that the left edge comes toward you. Call this angular displacement $theta_2$. The spine of the book should now face you, with the writing oriented so that you can read it.

Now start over again but carry out the two rotations in the reverse order. Do you get a different result? That is, does $theta_1+theta_2$ equal $theta_2+theta_1$?

Now repeat this experiment but this time with an angle of $1^circ$ rather than $90^circ$. Do you think that the infinitesimal displacement $text dboldsymboltheta$ obeys the commutative law of addition and hence qualifies as a vector? If so, how is the direction of $text dboldsymboltheta$ related to the direction of $boldsymbol omega$?

The idea is that the angular velocity vector $boldsymbolomega=text dboldsymboltheta/text dt$ only ever deals with infinitesimal angular displacements, so the example in the pictures you have posted are not valid for comparing angular displacement to angular velocity. The infinitesimal angular displacement can be treated as a vector just fine, even if "larger" displacements cannot be.

Vectors are mathematical expressions which should transform in the right way. Vector transformation means the way common vectors (like displacement) transforms under the translational or rotation of coordinate system.

Can a vector arise from a non-vector quantity?

Yes and they always do. Gradient of any scalar field is a vector field. There is also possibility of converting a vector into a scalar, divergence of any vector is a scalar.

Angular displacement has to specify a plane. Then it is a vector as long as you only add it up with corresponding vectors( of the same plane).

Angular position can be considered a vector $vec theta$. Just not a so-called geometric vector (or Euclidean vector).

The author of your text book implicitly refers to geometric vectors, it seems. Many mathematical operations are defined for geometric vectors only, since we here see high degrees of symmetry and thus see simple relations such as the commutative law (that the order of added parts doesn't matter).

Whether it is mathematically consistent that geometric and non-geometric vectors are each others' derivatives, might be a better question for the Math SE site. Else, possibly you can find some explanations by searching for geometric calculus.

But even those derivates still do not fully behave like geometric vectors - for instance, the superposition principle does not apply to angular velocity (not in 3 dimensions, at least). If you make an object spin about all three axes, you will not see them "merge" into one resultant angular velocity, as you would see with linear velocities (actually you will, but that axis is no fixed). Instead, all three motion overlap in a bulgy and "tumbled" spin.