How is CPU different from GPU?

The basic architecture of CPU and GPU are different. Modern CPU's consist of a set of cores. Each core has its own a set of registers, a ALU and a control unit with some private cache. In a CPU the number of ALU's (the actual processor) is in 1-100 range. In a GPU number of ALU's are in thousands. The register files are larger and is shared among multiple ALU. A large register file with a set of ALU's(typically in range 32--128) and some additional hardware are known as streaming multiprocessors in a NVIDIA GPU. A GPU has a large number of streaming multiprocessors (can be in 100's). Having a large number of ALU's give the GPU's its extreme compute capability. Only limitation is the computation should have a certain kind of parallelism.

GPGPU don't have some hardware component like texture units (which are required for graphics processing but would not be relevant for general computing).

GPUs are highly specialized: they are very good at a very small number of things, but they are extremely bad at everything else.

CPUs are general: they are mediocre at everything.

GPUs are good at applying the same operation out of a small number of potential operations, and in particular, the same very simple operation to a lot of identical things. E.g. when you want to brighten up a picture, you need to apply the same extremely simple multiplication to all 3 colors of all 10million pixels of a 4K screen.

A CPU is almost exactly the opposite: the number of potential different operations is not small, it is almost endless. (Word processing, spreadsheet crunching, compiling, web browsing, emailing, controlling a robot, driving a car, …) The operations aren't simple, they are extremely complex. And you don't apply the same operation to a huge number of identical things, you apply a different operation to a number of different things.

GPGPU (General-purpose GPU) is a misnomer, really. They are not "general purpose" in the same sense that a general purpose CPU is. What is meant by that, really, is that they can be used for purposes other than creating pictures that still have the same characteristics as graphics. For example, modern graphics cards can do particle simulations for generating realistic rain or explosions. What "GPGPU" means is that you can use those same particle simulation features without generating a picture from it, and instead, say, generate velocity predictions for the fragments of an exploding asteroid.

But, reading your mail is a very different thing than simulating a billion particles.