Why do biometric devices that work in one setting do not work well in other settings? The usual answers to this question are:
- There is a variance in the conditions.
- Operator errors which will go away once everyone gets trained on using the device (including users).
- There are people who want to see the system fail (Malefic intent)
We have seen almost all these answers to explain significant biometric authentication failures on the field, but so far no one has considered the question
Has the UIDAI made fantastical assumptions about the success of biometric authentication by ignoring basic probability theory taught in 10th Standard Mathematics books?
During this post, we will explore this question slowly.
How do you measure failures?
We will only concern ourselves with False Rejection Rate (FRR) which causes exclusion during biometric authentication. Every device used in the field is certified by STQC (Standardization Testing and Quality Certification) to have a FRR of < 2%. So why do field reports always report > 10% on average? To understand this, we must understand a bit of probability.
It is intuitive to understand probability using a coin. We don’t really need to conduct an experiment to understand that P(Heads) = ½ and P(Tails) = ½. However, for biometric authentication, we need to conduct an experiment to determine the Probability (FRR) because it is not obvious like the coin toss example.
The FRR is determined thus (through counting):
- Find “N” people (N = 3000 say). Enroll their fingerprints and authenticate all of them.
- Observe how many people failed to get themselves authenticated (Failed) FRR = (Failed / 3000)
- It can be expressed a percentage, but it is not very relevant for the discussion.
The next question to ask, once we get this number (< 2%) is:
Is this a constant number across the entire population?
STQC itself knows that this is not so, since the biometric testing depends on National Institutes of Standards and Technology (NIST) standards. The variance of across finger print quality and gender is given below:
So even given the < 2% FRR, we can already conclude the following without looking at any authentication data:
1) There is a direct correlation with increasing age and increasing authentication failures.
2) If < 2% is the average failure rate, given the rapid fall of finger print quality as one ages, the failure rates will be much higher for aged people.
3) There could be a potential 25% increase in failures for women (2.0 vs 2.5 Image quality, assuming linear relationship between failures and image quality) and for older women it could get worse.
There is still the question of how do we convert these FRR probability numbers to understand exclusion? For that we must play a little game.
Probability and Outcome are different
There are two players (A, B) in the game and a single coin. If the coin turns up as Head, A wins ₹1 from B and if the coin turns up as Tail, B wins ₹1 from A. Now the key question:
How much would A and B have won, if they start from zero and play the game long enough?
The answer is obvious. It Is Zero and can be written down as (½ * 1) — (½ * 1) for both A and B.
Now let us change the rules and ask a different question:
- If B loses thrice in a row, he/she loses the game and a different B comes in.
- If B wins even once, he/she wins the game and a different B comes in.
- Irrespective of which B plays the game, there is a small chance that they will never get tails.
How many B’s will return empty handed, if this game is played long enough?
You don’t have to answer this question through computation, but intuitively we can understand it that a lot of people will return empty handed.
In other words, the conditions of the game are more important to understand the people who would return empty handed than the probability of getting a tail. It is impossible to understand the impact (people returning empty handed) by looking at the raw probability of getting a tail.
Mathematically this is taught as (X) and F(X) are not one and the same thing, where (X) is a random variable (Fooled by Randomness is a good start).
Biometric Game of Authentication
We can now understand why exclusion caused by biometric authentication is wide spread on every program that it touches.
- Every B has a different probability of success. Elderly women have the worst chance followed by the elderly (>50). A few may have no chance of success.
- Depending on the program where it is deployed, the number of repeat trials may be very less. For instance, if biometric authentication is mandated for an exam, a student may not get more than 2 trials, if the queue is long and the exam begins in 10 minutes.
- Hardware failures, Mobile connectivity etc. adds additional failure chances irrespective of the B trying to authenticate.
All the above changes, worsens the expectations function and thus causes exclusion. Mathematically the expectations function is concave, while the probability function is linear and one cannot surmise the former from the later. Yet UIDAI and STQC test and report only the probability function and refuse to publish data about the expectation function.
A class 10 student can understand the distinction between probability and expectation functions. UIDAI seems to behave like a student, who wrote the wrong answer to a standard probability vs. expectation question and when called out, alleges malefic intent on the examiner.
It is the mathematical understanding of the student that is the problem here, not the malefic intent of the examiner. May be RD Sharma will help?Tags: Aadhaar Authentication Uidai