Having a hard time picturing what you're saying. But I'm assuming you installed the correct pulley. Most trucks had A/C, which use a smooth pulley.
If you're saying that the tensioner itself has play in it, replace the tensioner.
Again, not picturing exactly what you're describing.