Ant validation/deployment errors using sessionId

I have a weird problem with Ant setup on CI. I have a script that fetches access token using JWT. The token is retrieved successfully, and when I want to use it with Ant deployment (switching in ant properties to sf.sessionId instead of username/password) I get an error ‘Invalid username, password, security token; or user locked out.’. It happens only in one sandbox (tested it on 3 different sandboxes, the other ones are working fine). I’m not using username/password/security token for deployment, and the user is not locked out. Deploying to that sandbox with username/password in ant properties works, but I don’t want to do that.
Anyone had similar problems?
EDIT:
It looks like the only thing that doesn’t work is Ant deployment, the access_token works fine when I try to access some REST resources like /services/data/v49.0/sobjects/ etc
Maybe it’s the problem with ant-salesforce.jar? Is there a way to debug it?