The clearance on both sides should be equal. The pot joints fits to the box much the same as the drive shaft fits to the CV joints, splines and an internal circlip. There is a service tool to release them, but they can usually be levered or wedged off with a breaker bar or chisel between the joint and a bolt head on the diff side cover.
The driveshaft needs to slide in & out of the pot joint as the steering & suspension moves, otherwise it should be free of play. Put it in gear and jack just one wheel clear of the floor; with one wheel and the final drive held the free movement in the airborne wheel will be down to play in the diff or drive shaft joints, and you can see the if the pot joints aren't moving because of sloppy driveshafts.