Softmax: Difference between revisions
Jump to navigation
Jump to search
mNo edit summary |
mNo edit summary |
||
| (7 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
{{#addbodyclass:tag_math}} | |||
{{stub}} | {{stub}} | ||
'''softmax''' (sometimes called softargmax {{comment|(except it's not really like [[argmax]])}}, normalized exponential function, and other things) | |||
softmax ( | * takes a vector of numbers | ||
:: (any scale) | |||
* takes a vector of numbers | |||
* | * returns a same-length vector of probabilities | ||
:: all in 0 .. 1 | :: all in 0 .. 1 | ||
:: that sum to 1.0 | :: that sum to 1.0 | ||
<!-- | |||
Note that it is ''not'' just normalization, nor is it just a way to bring out the strongest answer. | |||
Note that it is ''not'' just normalization. | |||
The exponent in its internals, plus the "will sum to 1.0 part" will mean things shift around in a non-linear way, | |||
so even relative probabilities already in in 0..1 and summing to 1.0 will change, e.g. | so even relative probabilities already in in 0..1 and summing to 1.0 will change, e.g. | ||
: softmax([1.0,0.5,0.1]) ~= 0.5, 0.3, 0.2, | : softmax([1.0,0.5,0.1]) ~= 0.5, 0.3, 0.2, | ||
| Line 32: | Line 23: | ||
The name might suggest to you it is a numerically smoothed maximum. It is not. | |||
It is much closer to [[argmax]] | |||
A smooth approximation to the arg max function: the function whose value is the index of a vector's largest element. In fact, the term "softmax" is also used for the closely related LogSumExp function, which is a smooth maximum. For this reason, some prefer the more accurate term "softargmax", but the term "softmax" is conventional in machine learning.[3][4] This section uses the term "softargmax" to emphasize this interpretation. | |||
<!-- | |||
The output has the probabilities of a probability distribution. | |||
If you squint, it is ''something'' like [[sigmoid]] function (because this is a generalization of the [[logistic function]]) | |||
but it is not directly comparable to transfer functions, | |||
and you can't get an easy plot of it, | |||
exactly ''because'' it takes multiple inputs. | |||
It is a more generic mathematical tool, | |||
historically seen a bunch in machine learning, | |||
and these days many references are its use in neural nets. | |||
In that context they will | |||
take activation on any sort, | |||
and put them into 0..1 scale sensibly, | |||
mostly as a normalization step that is often used at least in the final layer, | |||
and sometimes at the end of smaller building blocks as well. | |||
When using nets as multiclass classifiers, you would need ''something'' like softmax to be able to respond on all the labels, | |||
and in a way that looks like probabilities. | |||
In part it's just a choice of what you want to show (you could output classification margin scores instead), | |||
in part it's a choice that | |||
| Line 48: | Line 70: | ||
--> | --> | ||
https://en.wikipedia.org/wiki/Softmax_function | |||
[[Category:Math on data]] | |||
Latest revision as of 23:15, 21 April 2024
✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.
softmax (sometimes called softargmax (except it's not really like argmax), normalized exponential function, and other things)
- takes a vector of numbers
- (any scale)
- returns a same-length vector of probabilities
- all in 0 .. 1
- that sum to 1.0