Hmmm... seems like your table isn't level, or your drilling head isn't perfectly vertical. Is it badly off?
I know you've got a few problems with the tool itself, from what you've stated. The most obvious thing that comes to mind, is that your mounting rig (where you put the block) has to be perfectly clean. Is it?
You can use a locking washer, to help prevent those screws from coming loose from vibrations.
Good luck with the hold down clamps: it looks more promising.