Second article I had on this. The first one was installing a Windows machine, now in Red Hat 5 enterprise 64 bit Intel.
The answer was simple in this case it was the autoinst.iso --- this could also be a floppy.flp, but not in my case. All I had to do is shutdown the machine (did not try with it on), and remove the device. After I did this, I powered it on, and the VMWare tools then were able to install. See Screen shot: