Number holder

A Number holder is one who came into contact with it and wielded a "Number" card in a Duel.

Possessed Duelists
"Numbers have the ability to possess people, and do so when they are initially released. Numbers can have varying control over people, either by increasing their already present emotions (As in the case with Flip) to actually controlling and manipulating their mind (As with Nelson Andrews). Some "Number"-possessed wish to gain more "Numbers" (Number 20's holder, Kaio, and Rikuo).

Numbers Hunter
A Number Hunter is a person who collects "Number" cards for one reason or another. Kaito Tenjo claimed himself to be a Number Hunter, being the only one, but later he identified Yuma Tsukumo as a Number Hunter. Yuma most likely doesn't see himself as a Numbers Hunter, but he technically is one. Number Hunters have ways of detecting when a Number holder appears, and a method to take away that Number when they defeat it. Possessed duelists who want more "Numbers" shouldn't be considered Number Hunters.

Others
These people are immune to the corrupting effects of "Numbers" and can fight against them despite not having any "Numbers" of their own. They either borrowed another person's "Number" or previously had one and now uses a different method to defeat them, instead. They seem to gain some immunity by having a large amount of will (Bronk wishing to save Yuma and Shark trying to protect Yuma's Key) or had no interest on finding "Numbers" for themselves.