Ahhhh, fellow hand CNC'er
I did the same thing, just with the mill bits (2 mm to be precise)... With block in the vice, I errr...'milled' it by hand, but I moved the vice on the table with one hand (vice was loosely attached with two bolts to the table) and went up and down with mill bit. The result is not so 'nice', but serves the purpose well...
pic is crappy, but...