I have seen when cast iron flywheels let go. I have not seen a steel one "let go". Usually the iron ones crack.

The reason I said use it was because to enlarge the holes enough to fit a sleeve around around the bolt would only weaken it more. Which is exactly what you are worried about.
There are 8 dowels, Thats a huge amount driving that flywheel. So much better than using bolts to drive a flywheel.