well as fast as you can, I used 2500rpm on mine.
You definately could get past 5k with a pully change, I currently have my mill at 6700rpm but I have almost no torque.
I also had to do a bearing change to get that kind of speed, otherwize my bearings would have melted
Im sure your drill press will have that problem too.
Jon