I’m assuming that by “contingent workers” you mean outsourced or non-permanent personnel who are hired on a per-project basis (e.g., freelancers, independent contractors, consultants, etc.).
Considering that, and ISO 27000, which defines the vocabulary for information security management systems compliant to ISO 27001, you can use the concept of “outsourced organization”.
For ISO 27000:
outsource refers to a situation where an external organization performs part of the organization's duties
organization refers to a person or group of persons with defined responsibilities, authorities, and relations
Please also note that you can also use other terms like contractors, external parties, because they are present in ISO management standards, although there is no formal definition for these terms in ISO glossaries.