API and Modules

Dimensionality Reduction sklearn Compatible Estimators

TorchDR provides a set of classes that are compatible with the sklearn API. For example, running TSNE can be done in the exact same way as running sklearn.manifold.TSNE with the same parameters. Note that the TorchDR classes work seamlessly with both Numpy and PyTorch tensors.

For all methods, TorchDR provides the ability to use GPU acceleration using device='cuda' as well as LazyTensor objects that allows to fit large scale models directly on the GPU memory without overflows using keops=True.

TorchDR supports a variety of dimensionality reduction methods. They are presented in the following sections.

Spectral Embedding

Those classes are used to perform classical spectral embedding from a torchdr.Affinity object defined on the input data. They give the same output as using torchdr.AffinityMatcher with this same torchdr.Affinity in input space and a torchdr.ScalarProductAffinity in the embedding space. However, torchdr.AffinityMatcher relies on a gradient-based solver while the spectral embedding classes rely on the eigendecomposition of the affinity matrix.

PCA([n_components, device, verbose, ...])	Principal Component Analysis module.
KernelPCA(affinity, n_components, device, ...)	Kernel Principal Component Analysis module.

Neighbor Embedding

TorchDR supports the following neighbor embedding methods.

SNE([perplexity, n_components, lr, ...])	Implementation of Stochastic Neighbor Embedding (SNE) introduced in [1].
TSNE([perplexity, n_components, lr, ...])	Implementation of t-Stochastic Neighbor Embedding (t-SNE) introduced in [2].
TSNEkhorn([perplexity, n_components, lr, ...])	Implementation of the TSNEkhorn algorithm introduced in [3].
InfoTSNE([perplexity, n_components, lr, ...])	Implementation of the InfoTSNE algorithm introduced in [15].
LargeVis([perplexity, n_components, lr, ...])	Implementation of the LargeVis algorithm introduced in [13].
UMAP([n_neighbors, n_components, min_dist, ...])	Implementation of UMAP introduced in [8] and further studied in [12].

Advanced Dimensionality Reduction with TorchDR

TorchDR provides a set of generic classes that can be used to implement new dimensionality reduction methods. These classes provide a modular and extensible framework that allows you to focus on the core components of your method.

Base Classes

The torchdr.DRModule class is the base class for a dimensionality reduction estimator. It is the base class for all the DR classes in TorchDR.

torchdr.AffinityMatcher is the base class for all the DR methods that use gradient-based optimization to minimize a loss function constructed from two affinities in input and embedding spaces.

DRModule([n_components, device, keops, ...])	Base class for DR methods.
AffinityMatcher(affinity_in, affinity_out[, ...])	Perform dimensionality reduction by matching two affinity matrices.

Base Neighbor Embedding Modules

Neighbor embedding base modules inherit from the torchdr.AffinityMatcher class and implement specific strategies that are common to all neighbor embedding methods such as early exaggeration.

In particular, torchdr.SparseNeighborEmbedding relies on the sparsity of the input affinity to compute the attractive term in linear time. torchdr.SampledNeighborEmbedding inherits from this class and adds the possibility to approximate the repulsive term of the loss via negative samples.

NeighborEmbedding(affinity_in, affinity_out)	Solves the neighbor embedding problem.
SparseNeighborEmbedding(affinity_in, ...[, ...])	Solves the neighbor embedding problem with a sparse input affinity matrix.
SampledNeighborEmbedding(affinity_in, ...[, ...])	Solves the neighbor embedding problem with both sparsity and sampling.

Affinity Classes

The following classes are used to compute the affinities between the data points. Broadly speaking, they define a notion of similarity between samples.

Simple Affinities

GaussianAffinity([sigma, metric, zero_diag, ...])	Compute the Gaussian affinity matrix.
StudentAffinity([degrees_of_freedom, ...])	Compute the Student affinity matrix based on the Student-t distribution.
ScalarProductAffinity([device, keops, verbose])	Compute the scalar product affinity matrix.
NormalizedGaussianAffinity([sigma, metric, ...])	Compute the Gaussian affinity matrix which can be normalized along a dimension.
NormalizedStudentAffinity([...])	Compute the Student affinity matrix which can be normalized along a dimension.

Affinities Normalized by kNN Distances

SelfTuningAffinity([K, normalization_dim, ...])	Compute the self-tuning affinity introduced in [22].
MAGICAffinity([K, metric, zero_diag, ...])	Compute the MAGIC affinity introduced in [23].

Entropic Affinities

SinkhornAffinity([eps, tol, max_iter, ...])	Compute the symmetric doubly stochastic affinity matrix.
EntropicAffinity([perplexity, tol, ...])	Solve the directed entropic affinity problem introduced in [1].
SymmetricEntropicAffinity([perplexity, lr, ...])	Compute the symmetric entropic affinity (SEA) introduced in [3].

Quadratic Affinities

DoublyStochasticQuadraticAffinity([eps, ...])

Compute the symmetric doubly stochastic affinity.

UMAP Affinities

UMAPAffinityIn([n_neighbors, tol, max_iter, ...])	Compute the input affinity used in UMAP [8].
UMAPAffinityOut([min_dist, spread, a, b, ...])	Compute the affinity used in embedding space in UMAP [8].

Utils

The following classes are used to perform various operations such as computing the pairwise distances between the data points as well as solving root search problems.

pairwise_distances(X[, Y, metric, keops])	Compute pairwise distances matrix between points in two datasets.
binary_search(f, n[, begin, end, max_iter, ...])	Implement the binary search root finding method.
false_position(f, n[, begin, end, max_iter, ...])	Implement the false position root finding method.

References

[1] (1,2)

Geoffrey Hinton, Sam Roweis (2002). Stochastic Neighbor Embedding. Advances in Neural Information Processing Systems 15 (NeurIPS).

[2]

Laurens van der Maaten, Geoffrey Hinton (2008). Visualizing Data using t-SNE. The Journal of Machine Learning Research 9.11 (JMLR).

[3] (1,2)

Hugues Van Assel, Titouan Vayer, Rémi Flamary, Nicolas Courty (2023). SNEkhorn: Dimension Reduction with Symmetric Entropic Affinities. Advances in Neural Information Processing Systems 36 (NeurIPS).

[8] (1,2,3)

Leland McInnes, John Healy, James Melville (2018). UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.

[12]

Sebastian Damrich, Fred Hamprecht (2021). On UMAP’s True Loss Function. Advances in Neural Information Processing Systems 34 (NeurIPS).

[13]

Tang, J., Liu, J., Zhang, M., & Mei, Q. (2016). Visualizing Large-Scale and High-Dimensional Data. In Proceedings of the 25th international conference on world wide web.

[15]

Sebastian Damrich, Jan Niklas Böhm, Fred Hamprecht, Dmitry Kobak (2023). From t-SNE to UMAP with contrastive learning. International Conference on Learning Representations (ICLR).

[22]

Max Zelnik-Manor, L., & Perona, P. (2004). Self-Tuning Spectral Clustering. Advances in Neural Information Processing Systems 17 (NeurIPS).

[23]

Van Dijk, D., Sharma, R., Nainys, J., Yim, K., Kathail, P., Carr, A. J., … & Pe’er, D. (2018). Recovering Gene Interactions from Single-Cell Data Using Data Diffusion. Cell, 174(3).