Personal data

In case of wholesale VoIP carrier activity, the only information about a caller is just a phone number without any connection and links to any personal data (like name, home address, gender, etc). In this case, it is difficult to say that having just a number is a personal data, because no person may be identified by the carrier by the number. From my point of view, just a number (without any other information that helps to identify the personality) may be considered by an intermediate wholesale VoIP traffic carrier mostly as a metadata than a personal data, and such data does not fall under GDPR rules. Am I right in my suggestions?