Conventional unit cell and point group symmetries?

The unit cell is a 3D figure which possess certain symetry (e.g. cube, tetragon etc.). The unit cell is selected after you have found out what is the symmetry of the crystal, and it is selected in a way that it has the symmetry of the crystal (can be rather complicated, like here). You cannot build unit cell if you don't know what your crystal look like (just from the number of atoms, etc.). With a given unit cell you can reproduce your crystal.

A definition of a conventional unit cell of a lattice is one that contains the same point group symmetries as the overall lattice and is the smallest such cell.

I don't think this is the definition of a "conventional unit cell".

The "smallest cell" that fully describes any structure is the primitive cell, which is the smallest cell that contains only one lattice point.

https://en.wikipedia.org/wiki/Primitive_cell

The important symmetries for the primitive cell are the translational symmetries, which are part of the space group symmetry of the lattice, not the point group symmetry. Because the definition of the primitive cell does not specify the position of the cell origin with respect to the contained lattice point, the point group symmetry of a primitive cell is not uniquely defined, and depends on the choice of cell origin. You may, or may not, be able to find point group symmetry operations within a given cell that you would expect from looking at the full crystal.

Part of your difficulty is that you are not choosing a point about which to define your symmetry operations (after all, they are called point symmetries). In the case of your specific square, the symmetry operations are defined w/r to the center of the square.

This makes it clear that the reflections - indicated in blue or red - are about planes going through the symmetry point. In particular, a reflection about the horizontal axis interchange $(12)leftrightarrow (43)$.

If you choose an atom in the square as symmetry point, then you will need to use (discrete) translations to bring the transformed square back to its original position. These translations are also included in the symmetry group of the lattice so no real harm is done since any two cells are equivalent.

There are several good sources on this but one I like is

A.W. Joshi, Elements of group theory for physicists.