The key issue here is that ‘level of certainty’ doesn’t really mean what you would like it to.
You get back a number yes, but it can change according to what’s visible in the background, the angle that the plants at, how close is it to the camera, and how nice the camera is you’re using (professional photographers use expensive cameras and take shots of different things to everyone else).
Interpreting this score as “how safe is it to eat the plant” is a really bad idea. You will still eat the wrong plant. These scores can lead to very confident random guessing when you show it a plant it’s never seen before.
And no, softmax is a trick for making the scores all sum to one, so you get back a confidence for every possible thing the image could be of.
I feel like you’re viewing this from the wrong angle, or at very least we’re viewing it from different angles. You seem to be doing a binary classification (Is this plant edible) rather than a group classification (what plant is this?) where edibility is an attribute of the plant to be returned to the user (yes; no; when green; only the roots; etc.) - the latter is the approach most of these apps take, classify the image into a species (or list of potential species) then give the user details such as identifying features, common growing areas, edibility, and lookalikes.
You’re right about softmax, it’s been a couple of years since I’ve done the programming side of this so my terminology is a bit rusty.
I’m not describing binary classification, I’m describing multiclass. “Group classification” isn’t really a thing. Yes, your ml system probably guesses what kind of plant it is and then looks up the ediblity of components.
The problem with this is how they will handle rare plants that aren’t in the dataset, or that are in the dataset but with insufficient data to be recognised.
Because multiclass assumes that it’s seen representative data on all possible outputs (e.g. plant types) it will tend to be dangerously confident on plant types it hasn’t seen before.
This is because it can rule out other classes. E.g. if you’re trying to classify as rose, tulip, or daisy and you get a bramble, your classifier is likely to be very certain it’s a rose because tulips and daisies don’t have thorns. So your softmax score is likely to show heavy confidence in rose even though it’s actually none of them.
This is exactly what can go wrong when you try to use the softmax/standard multiclass approach and come across an interesting rare mushroom or wild carrot. You don’t want it to guess which type of plant in the database it’s most like, even if this guess comes with scores, you want it to say that it genuinely doesn’t know and you shouldn’t eat it.
The key issue here is that ‘level of certainty’ doesn’t really mean what you would like it to.
You get back a number yes, but it can change according to what’s visible in the background, the angle that the plants at, how close is it to the camera, and how nice the camera is you’re using (professional photographers use expensive cameras and take shots of different things to everyone else).
Interpreting this score as “how safe is it to eat the plant” is a really bad idea. You will still eat the wrong plant. These scores can lead to very confident random guessing when you show it a plant it’s never seen before.
And no, softmax is a trick for making the scores all sum to one, so you get back a confidence for every possible thing the image could be of.
I feel like you’re viewing this from the wrong angle, or at very least we’re viewing it from different angles. You seem to be doing a binary classification (Is this plant edible) rather than a group classification (what plant is this?) where edibility is an attribute of the plant to be returned to the user (yes; no; when green; only the roots; etc.) - the latter is the approach most of these apps take, classify the image into a species (or list of potential species) then give the user details such as identifying features, common growing areas, edibility, and lookalikes. You’re right about softmax, it’s been a couple of years since I’ve done the programming side of this so my terminology is a bit rusty.
I’m not describing binary classification, I’m describing multiclass. “Group classification” isn’t really a thing. Yes, your ml system probably guesses what kind of plant it is and then looks up the ediblity of components.
The problem with this is how they will handle rare plants that aren’t in the dataset, or that are in the dataset but with insufficient data to be recognised.
Because multiclass assumes that it’s seen representative data on all possible outputs (e.g. plant types) it will tend to be dangerously confident on plant types it hasn’t seen before.
This is because it can rule out other classes. E.g. if you’re trying to classify as rose, tulip, or daisy and you get a bramble, your classifier is likely to be very certain it’s a rose because tulips and daisies don’t have thorns. So your softmax score is likely to show heavy confidence in rose even though it’s actually none of them.
This is exactly what can go wrong when you try to use the softmax/standard multiclass approach and come across an interesting rare mushroom or wild carrot. You don’t want it to guess which type of plant in the database it’s most like, even if this guess comes with scores, you want it to say that it genuinely doesn’t know and you shouldn’t eat it.