Can I change my mortgage loan servicer?

The only way to change mortgage loan servicers is by refinancing. But you should not refinance solely to get a new servicer. Here’s what to know.