You are correct. I just crawled under mine and the bumper ties into the hitch in that the bumper is bolted to the hitch, then the hitch is bolted to the frame. If you want to remove the hitch then you need to get the bumper that comes with out the hitch.
You best bet is to find someone who would like a hitch and is willing to swap bumpers with you. Since your giving up a hitch, that will be easy to find. The tough part is to find someone with the same color bumper.
SSG57 is looking for a hitch for his truck.
Factory Hitch Brand