I thought I would post back on this thread regarding the advice I received and how making this part proceeded.
Summary: I was successful in making the adapter. It ended up being a nice tight press fit with near zero runout once attached to the servo.
Making it was very easy once I followed the advice I got. Some key points:
1. Used a moderately sharp HSS tool: not carbide.
2. Kept speeds high->very high (> 1600 RPM)
All my tools earlier were carbide and I had to get HSS and learn to sharpen them. My initial piece was 3/8" that I needed to turn down to 6mm. I kept the length very short: around 5/8" projecting from collet end to avoid deflection. Interestingly, even with this precaution, if I took cuts deeper than a few thousands at a time, the deflection was measurable: the diameter of the part varied a bit over 0.001" over its length.
As I got closer to completion, I reduced the cut depth to practically nothing. That seemed to have reduced the deflection to nothing and the part ended up well within 0.0005" along its length.