which one is right answer?

I would recommend that you copy a complete question from a workbook. Otherwise, it is not clear how to answer your question. If 3 is indicated as the only correct answer, then this is the case of studying not usual grammar construction such as verbal adjective.

In this case, the meaning of this sentence is as follows: They had laid plans to use some time for looking at pictures in the museum. They spent a lot of this time.

1, 2, and 3 are all correct but the meaning slightly changes.

1. They spent a lot of time looking at the pictures in the museum.

This suggests the act of looking at the pictures is what they spent a lot of time on.

2. They spent a lot of time for looking at the pictures in the museum.

This emphasises the spending of time, suggesting that you didn't expect looking at the pictures in the museum to take so long.

3. They spent a lot of time to look at the pictures in the museum.

This could be read as meaning they spent a lot of time, perhaps before coming to the museum, in order to look at the pictures in the museum.