Proctorio's facial recognition is racist.
Also, it kind of sucks in general.
QED.
Wait. That's it?
Failures in the real world of their facial recognition algorithm are widely known, even if Mike is in denial (EDIT: see page 21, paragraph 2-3 of the letter to the senate, “fewer than five” issues due to race). See @Procteario [1] [2] for more; or for generic failures, even this image a friend sent me.
Now I have the data to prove it.
I talked about how I got my hands on their facial recognition data in my previous post – with my hands on their 'prized algorithm', I decided to take it on a test drive.
I set out to look for a representative dataset, and I found FairFace – a facial recognition dataset balanced for race, gender and age. I found their GitHub, which included a link to the dataset. I downloaded their [Padding=1.25] set, and ran the facial recognition algorithm against the 10954 faces stored in val
.
Proctorio includes all four OpenCV models – and runs all four of them at once too, despite that being... quite inefficient. But, I gave them the benefit of the doubt: I count an image as a “pass” in the above graph if any one of the models thought it found anything like a face at all; though often times it just doesn't work that way.
Several “failures” were also embarassingly obvious:
Note that all images featuring faces were taken from the FairFace dataset by Kimmo Kärkkäinen and Jungseock Joo. Original images licensed under the CC-BY 4.0 license.
EDIT: Added raw data.
Race | Pass | Fail | Pass % | Relative % from avg |
---|---|---|---|---|
Black | 669 | 887 | 42.99 | -28.72 |
Middle Eastern | 718 | 491 | 59.39 | -1.53 |
White | 1258 | 827 | 60.34 | 0.05 |
East Asian | 971 | 579 | 62.65 | 3.88 |
Southeast Asian | 917 | 498 | 64.81 | 7.46 |
Indian | 984 | 532 | 64.91 | 7.63 |
Latino / Hispanic | 1089 | 534 | 67.1 | 11.26 |
Total | 6606 | 4348 | 60.31 |
Audits, audits...
I earlier had the experience of looking through Proctorio's “zero knowledge encryption” claims, and their audit from a 'White Oak Security', which I found... less than promising.
Now, I have the privilege of looking through their facial recognition suite, the centerpiece of their “unbiased, unblinking” service (quote from home page), supposedly undergoing an audit by a 'BABL AI', which has been supposedly happening for months now.
As an outsider to their company, I managed to pull their facial recognition outside of their application and run it against an open-source dataset in a weekend.
With this track record of audits that take too long and lead to nowhere, I'm starting to question not just Proctorio, but the questionable people they contract with to rubber-stamp their broken products.