Is it possible to change password database file(/etc/passwd) in linux?

You're right: /etc/passwd and /etc/shadow are consulted by pam_unix.so, which are part of PAM. At least on modern Linuxes. You could change this by patching pam_unix.so. If the manpage is to be believed, you can't change the location of the system databases.

And you really don't want to. /etc/passwd isn't just used for authentication, it's also used for (reverse) name resolution and to look up things like the user's full name, shell, et cetera. The name and location are so standardised that moving them would almost definitely break things outside of just PAM. You'd have to patch a lot more than you bargained for.

Update: If you're trying to hide the /etc/{passwd,shadow,group} files for security, don't worry about it. Security by obscurity rarely helps as a policy. Leave them where they are and tighten the rest of your policies.

Update: a possible solution

If you have some custom software you need to access a different set of user/group databases, you could make a copy of the relevant PAM and NSS modules and patch them to use your custom databases. The original unix databases stay where they are so software isn't confused, but you can set PAM and NSS to use your custom modules wherever you need to and using whatever policy makes sense to you.

Keep the unix databases essentially pristine and you have what you asked for. This is pretty much what the radius and ldap PAM/NSS modules do: they provide an additional (not replacement) source of credentials and user/group information.

Going one step further: you can go back to PAM and NSS and disable the unix database lookups altogether. Leave the files there for old software (naturally, their views of the user/group databases will be inaccurate, but at least they won't break).