I would fall into your both category Andrew
Most guys I know don't or won't wear originals at all and instead collect them for their history and/or artwork. I used to be one of them as well, but have found some pleasure in wearing a original every now and then. (Sparingly)
That being said, I think besides the many wearers (including myself) on this very small forum, we are the minority vs strictly collectors. I know of thousands of collectors that just buy for the historical aspect.
As to the question, is it a crime...Not technically
. Would it hurt the value of the jacket.......Absolutely.