If there is any force applied to the key you have some issues. The square key is a locational item only. If your prop shaft is properly done there should be a milled flat where the set screws contact the shaft so they can be TIGHTENED to the shaft without leaving scars on the shaft that prevent it from coming apart. If you can tighten 4 set screws and 6 pinch bolts on the shaft AND are driving off the keyway you have problems that need to be investigated. 42 years as a tool and die maker /machinist and the round peg where a square slot should be is wrong on so many levels. The runout from a keyway cutter is of no consequence in the over all scheme of things and was the only way they were done in the good old days. Cutting with an end mill was what guys did when they only had one key to do or didn't have access to a key cutter. You have a square keyway on the output of the v-drive and on the prop shaft and prop, why would I worry about the one end on the prop and coupler. The discussion about have many variables exist in your round key would take another paragraph.