No need to remove the factory inserts. Just toss the bolts and inserts (spreaders) that came with the bar-ends.
Remove the weights from the end of your bars. and connect it all together like this:
The bolt on the right is identical to the stock bar-end bolt, but is 5mm longer. The washers on the left came with my new mirrors, and provide a bit of space between the new mirror mount and the grips.
Assemble like this, and screw back into the factory handlebar insert.
I found longer replacement bolts for the install at my local Fastenal. The ones I purchased are 5mm longer than the stock bolts, which was plenty to get both sides mounted securely.
Here's what I purchased if anyone wants to save the trouble of searching or experimenting:
M6-1.0 X 55
Socket Head Cap Screw
Class 12.9
Black Oxide Finish
Part #2139574