ISO 13485, clause 4.2.3 Medical Device File

Clause 4.2.3 is not possible to exclude or mark as not applicable. It means that you need to have the following documents in your Medical device file: CE certificate of the medical device and any special instruction or label about storage and transport conditions. Maybe there are more documents considering the nature of the medical device, but generally, those two documents are enough.

I understand if I am the device holder but if I am contractor making a medical device component and not the finished device why is this still necessary

In that case, you need to have a medical device file for the component that you produce. This means the specification for this component, if applicable some instructions on how your component is integrated into the final medical device, label, and any other kind of instructions (instructions of use, instructions for cleaning).