Machine learning goals, problems, and glossary

This is more for overview of my own than for teaching or exercise. Overview of the math's areas Arithmetic · 'elementary mathematics' and similar concepts Set theory, Category theory Geometry and its relatives · Topology Elementary algebra - Linear algebra - Abstract algebra Calculus and analysis Logic Semi-sorted : Information theory · Number theory · Decision theory, game theory · Recreational mathematics · Dynamical systems · Unsorted or hard to sort Math on data: Statistics as a field some introduction · areas of statistics types of data · on random variables, distributions Virtues and shortcomings of... on sampling · probability glossary · references, unsorted Footnotes on various analyses Other data analysis, data summarization, learning Machine learning goals, problems, and glossary Data modeling, restructuring, and massaging Statistical modeling · Classification, clustering, decisions, and fuzzy coding · dimensionality reduction · Optimization theory, control theory · State observers, state estimation Connectionism, neural nets · Evolutionary computing More applied: Formal grammars - regular expressions, CFGs, formal language Signal analysis, modeling, processing Image processing notes Varied text processing

Overall concepts, Glossary

Some broad problem types (mostly around ML)

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Descriptors, architectures, combinations, and other meta

Structured prediction

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

https://www.youtube.com/watch?v=e5sPNlgbZAE

https://pystruct.github.io/intro.html

https://en.wikipedia.org/wiki/Structured_prediction

"Deep learning"

Deep learning

Properties and behaviours of learners

Underfitting and overfitting

Underfitting is when a model is too simple to encode all the interesting things in the data.

Underfitted models and learners may still generalize very well, but they may not be covering/describing all the patterns in the data.

Sometimes that can be done intentionally, e.g. to describe only the major patterns.

But usually people want it to go as far as it could, so underfitting is a term used to point out maybe you want to allow more complexity into your model.

It may be hard to quantify how crude is too crude, though.

Overfitting goes the other way, where a model is too awkwardly specific

The exact implication of overfitting depends on the sort of learner it is (and the data you have), but often it describes cases where the model generalizes pretty well but also starts encoding so much specific/local data, insignificant variance, noise, or specific mistakes in the training set that that starts drowning out the more general parts.

Overfitting can happen

when your training data isn't large, or diverse, or balanced enough to actually suggest much generalization,

when there are entire groups of real-world examples that aren't represented, and/or

when the model is allowed to contain so much complexity that it can essentially just model all training data's cases, meaning it is no longer forced to generalize at all, so probably perform worse on unseen data

It might still continue to score better on your data - only,

From another angle, overfitting it is the (mistaken) assumption that convergence in the training data means convergence in all data.

A little localized overfitting is not disruptive, in part because when there's still real patterns to be learned (there's not often a real cuttoff between "we have learned all useful things and have now moved on to nonsense" - if there was this would not be an issue), that signal drowns out the noise.

But more than a little overfitting tends to start drown out the generalization with messines.

There are a few useful tests to evaluate overfitting and underfitting.

More concepts, some shared tools

On the theory / math side

Log-Linear and Maximum Entropy

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Bayesian learning

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Bayesian (Belief) Network

See below

Kernel method

https://en.wikipedia.org/wiki/Kernel_method