Afternoon Michael,
MMMmmmm!

Bit of a poser this one!
Had a rake through the Honda parts lists and it would appear that the domed head, cap screw, part number:
90128MM9000 has been used on the Honda Dominator NX650, 1988 to 1999 and the same cap screw was used on the Honda Vigor FX650, 1999 to 2000. There are also a further 450 plus Honda models, worldwide, on which this cap screw is used.
Are you using a 'pattern' sprocket? The thickness of material may be different from 'OE' sprocket.
Are you aware that in all the cases that I have viewed the parts list that there is a washer between the lock nut and the sprocket carrier on the inside of the carrier? This washer has the part number:
90465MM9000 and is fitted to over 1,000 Honda models, worldwide.
The specific nut for this application is part number:
90304GA6003 and is fitted to over 1,800 Honda models, worldwide.
Whilst the nut is a locknut and is also flanged, the washer is also required and if you do not have these, this maybe the cause of your inability to torque the assembly correctly.
I would add that stainless steel items would not be my choice for attaching a sprocket to the carrier; high tensile steel, zinc plated, items would be my choice, if Honda 'original' parts aren't being used and it isn't going to be an 'outline' restoration job.
Hopefully, this helps you resolve the problem.

My regards, Bill