Seat belt buckle scratches

Need some recommendations.

Am finding that the seat belt buckle, when retracted, rests against a plastic cover on the B pillar and scratches the plastic. The passenger buckle, if not being used, similarly scratches and kinda bangs on the plastic cover too.

As a possible solution, am considering applying a piece of gray felt or carpet tile about 6" down from the plastic seam. That way the buckle doesn't bang or scratch the plastic. Will try to match colors or get a complimentary color as best I can.

Would appreciate any other ideas.