Aye Simons...XBR500,
From your description, I would tend to agree with Ian's diagnosis, "Give it some revs!". Over-square Japanese singles don't like being 'lugged' in high gears and will make horrible rattling noises that can lead to broken gearboxes and a knocking big-end!

Long stroke singles e.g. Norton, BSA, AJS/Matchless and Arial will "thump" quite happily in a high gear, with a touch of retarded ignition and a slight throttle; they also often have a ruddy great big spring shock absorber on the drive side end of the crankshaft to absorb some of this torque reaction and smooth out the thumps!

As well as a rubber shock absorber in the rear hub.
If as your 'handle' suggests that you are already familiar with Japanese singles, then maybe the alternator rotor is starting to chatter on the crankshaft taper due to the 'lugging' and or a fretted woodruff key and an insufficiently torqued rotor nut.
Usually, these motors require 2,000rpm to get them off the mark without stalling and to be doing at least 55 - 60mph afore selecting 5th gear, unlees you have altered the sprockets to provide alternative gearing.
I hope this helps and you can get to enjoy the bike, rather than worry about it.
My regards, Bill