The potential at infinity for a infinite large system is zero/constant, while the potential at infinity for a finite system is not zero/constant

I'm guessing you got it backwards: for example, consider an infinite plate of charge in the $xy-$plane with a constant charge density $sigma$. The electric field in the upper half plane (i.e. for $z>0$) is given by $$mathbf{E} = frac{sigma}{2epsilon_0} mathbf{hat{z}},$$ which means its potential in this region is $$V = -frac{sigma z}{2epsilon_0},$$ which clearly blows up as $ztoinfty$. You could do a similar analysis for an infinite line of charge too, and it blows up as well (though it does so logarithmically).

Alternatively, consider a point charge, whose potential is $$V = -frac{1}{4piepsilon_0} frac{1}{r}.$$ This function goes to zero as $rtoinfty$.