Discriminative model
From Wikipedia, the free encyclopedia
Discriminative
models are a class of models used
in machine learning for
modeling the dependence of an unobserved variable y on an
observed variable x. Within a statistical framework, this is done
by modeling the conditional
probability distribution P(y | x),
which can be used for predicting y from x.
Discriminative
models differ from generative models in
that they do not allow one to generate samples from the joint distribution of x and y.
However, for tasks such as classification andregression that
do not require the joint distribution, discriminative models generally yield
superior performance. On the other hand, generative models are typically more
flexible than discriminative models in expressing dependencies in complex
learning tasks. In addition, most discriminative models are inherently supervised and
cannot easily be extended tounsupervised learning.
Examples
of discriminative models used in machine learning include:
- Logistic regression,
a type of generalized
linear regression used for predicting binary or categorical outputs
(also known as maximum
entropy classifiers)
- Linear
discriminant analysis
- Support vector
machines
- Boosting
- Conditional
random fields
- Linear regression
- Neural networks
Generative
model
From Wikipedia, the free encyclopedia
In probability and statistics, a generative model is
a model for randomly generating observable data, typically given some hidden
parameters. It specifies a joint probability
distributionover observation and label sequences. Generative models
are used in machine learning for
either modeling data directly (i.e., modeling observed draws from a probability
density function), or as an intermediate step to forming a conditional
probability density function. A conditional distribution can be
formed from a generative model through the use of Bayes' rule.
Shannon (1948) gives an example in which a
table of frequencies of English word pairs is used to generate a sentence
beginning with "representing and speedily is an good"; which is not
proper English but which will increasingly approximate it as the table is moved
from word pairs to word triplets etc.
Generative
models contrast with discriminative models,
in that a generative model is a full probabilistic model of all variables,
whereas a discriminative model provides a model only for the target variable(s)
conditional on the observed variables. Thus a generative model can be used, for
example, to simulate (i.e. generate) values of any variable in the
model, whereas a discriminative model allows only sampling of the target
variables conditional on the observed quantities. On the other hand, because
discriminative models do not need to model the distribution of the observed
variables, they can generally express more complex relationships between the
observed and target variables, and as a result often perform better than
generative models at classification and regression tasks.
Examples
of generative models include:
- Gaussian mixture
model and other types of mixture model
- Hidden Markov model
- Naive Bayes
- AODE
- Latent
Dirichlet Allocation
- Restricted
Boltzmann Machine
- Probabilistic Linear discriminant analysis (A PLDA model is essentially a Gaussian distribution with a structured covariance model comprising of a speaker and a channel component.)
If
the observed data are truly sampled from the generative model, then fitting the
parameters of the generative model to maximize the
data likelihood is a common method. However, since most
statistical models are only approximations to the true distribution,
if the model's application is to infer about a subset of variables conditional
on known values of others, then it can be argued that the approximation makes
more assumptions than are necessary to solve the problem at hand. In such
cases, it is often more accurate to model the conditional density functions
directly, using a discriminative model (see
above).
