A collaborative project

Increasing awareness in distance construction

A line of work for distance-based learning with mixed data

Scale/type-aware
Additivity, commensurability, and bias in mixed data

Association-aware
Redundancy, correlations, and categorical associations

Interaction-aware
Continuous–categorical relationships and local neighbourhood structure

Response-aware
Supervised neighbourhoods and prediction-oriented distances

1. Learning from distances

Learning from mixed-type data

Many statistical learning methods rely, explicitly or implicitly, on comparing observations.

Terminology alert!

I will often use distance as shorthand for pairwise dissimilarity.

Some measures discussed here are not metrics in the strict mathematical sense,
but they all quantify how different two observations are.

  • If the data are mixed-type, then comparison is not straightforward:

Many learning methods can operate directly on a dissimilarity matrix.

Dimension reduction

Multidimensional scaling (MDS)1

Clustering methods

Hierarchical clustering (HC), PAM2, and spectral clustering3

Nearest-neighbour prediction

Nearest-neighbour classification and nearest-neighbour averaging for regression

building pairwise dissimilarities: intuition

2 continuous variables: add up by-variable (absolute value or squared) differences

building pairwise dissimilarities: intuition

2 continuous variables: add up by-variable (absolute value or squared) differences

building pairwise dissimilarities: intuition

2 continuous variables: add up by-variable (absolute value or squared) differences

building pairwise dissimilarities: intuition

2 continuous variables: add up by-variable (absolute value or squared) differences

building pairwise dissimilarities: intuition

2 continuous variables: add up by-variable (absolute value or squared) differences

building pairwise dissimilarities: intuition

2 continuous and 1 categorical variables

building pairwise dissimilarities: intuition

one might consider purple and blue closer than e.g. purple and yellow

2. Desirable properties and general setup

Desirable properties1

Multivariate Additivity

Let \(\mathbf{x}_i=\left(x_{i1}, \dots, x_{iQ}\right)\) denote a \(Q-\)dimensional vector. A distance function \(d\left(\mathbf{x}_i,\mathbf{x}_\ell\right)\) between observations \(i\) and \(\ell\) is multivariate additive if

\[ d\left(\mathbf{x}_i,\mathbf{x}_\ell\right)=\sum_{j=1}^{Q} d_j\left(\mathbf{x}_i,\mathbf{x}_\ell\right), \]

where \(d_j\left(\mathbf{x}_i,\mathbf{x}_\ell\right)\) denotes the \(j-\)th variable specific distance.

  • Manhattan distance satisfies the additivity property; the Euclidean distance does not

Desirable properties1

If additivity holds, by-variable distances are added together: they should be on equivalent scales

Commensurability

Let \({\boldsymbol X}_i =\left(X_{i1}, \dots, X_{iQ}\right)\) denote a \(Q-\)dimensional random variable corresponding to an observation \(i\). Furthermore, let \(d_{j}\) denote the distance function corresponding to the \(j-\)th variable.

We have commensurability if, for all \(j\), and \(i \neq \ell\),

\[ E[d_{j}({ X}_{ij}, {X}_{\ell j})] = c, \]

where \(c\) is some constant.

Mixed-data setup

a mixed data set

  • \(n\) observations described by \(Q\) variables, \(Q_{n}\) numerical and \(Q_{c}\) categorical

  • the \(n\times Q\) data matrix \({\bf X}=\left[{\bf X}_{n},{\bf X}_{c}\right]\) is column-wise partitioned

A formulation for mixed distance between observations \(i\) and \(\ell\):

\[\begin{eqnarray}\label{genmixeddist_formula} d\left(\mathbf{x}_i,\mathbf{x}_\ell\right)&=& \sum_{j_n=1}^{Q_n} d_{j_n}\left(\mathbf{x}_i^{(n)},\mathbf{x}_\ell^{(n)}\right)+ \sum_{j_c=1}^{Q_c} d_{j_c}\left(\mathbf{x}_i^{(c)},\mathbf{x}_\ell^{(c)}\right)=\\ &=& \sum_{j_n=1}^{Q_n} w_{j_n} \delta^{(n)}_{j_n}\left(\mathbf{x}_i^{(n)},\mathbf{x}_\ell^{(n)}\right)+ \sum_{j_c=1}^{Q_c} w_{j_c}\delta^{(c)}_{j_c}\left(\mathbf{x}_i^{(c)},\mathbf{x}_\ell^{(c)}\right) \end{eqnarray}\]

numeric case

  • \(\delta^n_{j_n}\) is a function quantifying the dissimilarity between observations on the \(j_n-\)th numerical variable

  • \(w_{j_n}\) is a weight for the \(j_n-\)th variable.

categorical case

dissimilarity between the categories chosen by subjects \(i\) and \(\ell\) for categorical variable \(j_c\)

  • \(w_{j_c}\) is a weight for the \(j_c-\)th variable

Scale/type-induced biases

Distributions, scaling and bias: the numeric case

as long as variables have the same underlying distribution and scaling, commensurability holds

  • skewed variables may be under- or over-contributing to the distance, depending on the scaling (range and robust, respectively)

    • the contribution of a variable to the overall distance may be biased

distributions, scaling and bias: the categorical case

skewed frequency distribution

  • \(q\in \{2,3,5,10\}\)
  • \(p_1 \in \{0.05,0.1,0.2,0.33, 0.5,0.66, 0.8,0.9,0.95\}\)
  • \(p_j = (1-p_1)/(q-1)\), with \(j=2,\dots,q\),

The expected distance increases with the heterogeneity of the distribution and with the number of categories

Independence-based distances

Independence-based pairwise distance

No inter-variable relations are considered.

  • in the continuous case: Euclidean or Manhattan distances

  • in the categorical case: Hamming / matching distance, among many others

  • in the mixed-data case: Gower dissimilarity index

variable contributions may be balanced, but still treated as separate sources of information

Beyond commensurability

commensurability makes variable contributions comparable across scales and data types.

  • If variables are correlated or associated, the same information may contribute repeatedly to the distance: redundancy

the next step is to account for the structure among variables

3. Association-aware distances

by variable differences: independence-based

  • When variables are correlated or associated, shared information is effectively counted multiple times

  • inflated dissimilarities may cause potential distortions in downstream unsupervised learning tasks.

by variable differences: independence-based

  • When variables are correlated or associated, shared information is effectively counted multiple times

  • inflated dissimilarities may cause potential distortions in downstream unsupervised learning tasks.

by variable differences: independence-based

The Euclidean distance \(\longrightarrow\) shared information is over-counted

accounting for inter-variable relations: association-based

The Mahalanobis distance \(\longrightarrow\) shared information is not over-counted

this is an association-based distance for continuous data

association-based distance

Association-based for continuous: Mahalanobis distance

Let \({\bf X}_{con}\) be \(n\times Q_{d}\) a data matrix of \(n\) observations described by \(Q_{d}\) continuous variables, and let \(\bf S\) the sample covariance matrix, the Mahalanobis distance matrix is

\[ {\bf D}_{mah} = \left[\operatorname{diag}({\bf G})\,{\bf 1}_{n}^{\sf T} + {\bf 1}_{n}\,\operatorname{diag}({\bf G})^{\sf T} - 2{\bf G}\right]^{\odot 1/2} \] where

  • \([\cdot]^{\odot 1/2}\) denotes the element-wise square root

  • \({\bf G}=({\bf C}{\bf X}_{con}){\bf S}^{-1}({\bf C}{\bf X}_{con})^{\sf T}\) is the Mahalanobis Gram matrix

  • \({\bf C}={\bf I}_{n}-\tfrac{1}{n}{\bf 1}_{n}{\bf 1}_{n}^{\sf T}\) is the centering operator

association-based distance

Association-based for categorical: total variation distance (TVD)(Le & Ho, 2005)

The distance matrix \({\bf D}_{tvd}\) can be defined via the delta framework1 upon properly defining the block-diagonal matrix \({\bf \Delta}\)

Let \({\bf X}_{cat}\) be \(n\times Q_{c}\) a data matrix of \(n\) observations described by \(Q_{c}\) categorical variables.

\[ {\bf D} = {\bf Z}{\Delta}{\bf Z}^{\sf T} = \left[\begin{array}{ccc} {\bf Z}_{1} & \dots & {\bf Z}_{Q_{c}} \end{array} \right]\left[\begin{array}{ccc} {\bf\Delta}_1 & & \\ & \ddots &\\ & & {\bf\Delta}_{Q_{c}} \end{array} \right] \left[ \begin{array}{c} {\bf Z}_{1}^{\sf T}\\ \vdots \\ {\bf Z}_{Q_{c}}^{\sf T} \end{array} \right] \]

  • in the framework, setting \({\Delta}_j\) determines the categorical distance measure of choice (independent- or association-based)

association-based distance

Association-based for categorical: total variation distance (TVD) (Le & Ho, 2005) (2)

Consider the empirical joint probability distributions stored in the off-diagonal blocks of \({\bf P}\):

\[ {\bf P} = \frac{1}{n} \begin{bmatrix} {\bf Z}_1^{\sf T}{\bf Z}_1 & {\bf Z}_1^{\sf T}{\bf Z}_2 & \cdots & {\bf Z}_1^{\sf T}{\bf Z}_{Q_c} \\ \vdots & \ddots & \vdots & \vdots \\ {\bf Z}_{Q_c}^{\sf T}{\bf Z}_1 & {\bf Z}_{Q_c}^{\sf T}{\bf Z}_2 & \cdots & {\bf Z}_{Q_c}^{\sf T}{\bf Z}_{Q_c} \end{bmatrix}. \]

The block matrix \(\bf R\) refer to the conditional probability distributions for each variable \(j\) given each variable \(i\) (\(i,j=1,\ldots,Q_c\), \(i\neq j\)), stored in the block matrix

\[ {\bf R} = {\bf P}_z^{-1}({\bf P} - {\bf P}_z). \]

where \({\bf P}_z = {\bf P} \odot {\bf I}_{Q^*}\), and \({\bf I}_{Q^*}\) is the \(Q^*\times Q^*\) identity matrix.

association-based distance

Association-based for categorical: total variation distance (TVD)(Le & Ho, 2005) (3)

Let \({\bf r}^{ji}_a\) and \({\bf r}^{ji}_b\) be the rows of \({\bf R}_{ji}\), the \((j,i)\)th off-diagonal block of \({\bf R}\).

The category dissimilarity between \(a\) and \(b\) for variable \(j\) based on the total variation distance (TVD) is defined as

\[ \delta^{j}_{tvd}(a,b) = \sum_{i\neq j}^{Q_c} w_{ji} \Phi^{ji}({\bf r}^{ji}_{a},{\bf r}^{ji}_{b}) = \sum_{i\neq j}^{Q_c} w_{ji} \left[\frac{1}{2}\sum_{\ell=1}^{q_i} |{\bf r}^{ji}_{a\ell}-{\bf r}^{ji}_{b\ell}|\right], \label{ab_delta} \]

where \(w_{ji}=1/(Q_c-1)\) for equal weighting (can be user-defined).

TVD-based dissimilarity matrix is, therefore,

\[ {\bf D}_{tvd}= {\bf Z}{\Delta}^{(tvd)}{\bf Z}^{\sf T}. \]

association-based distance: a small example

Data

Consider two categorical variables:

  • \(X_1\) with categories \(A,B,C\)
  • \(X_2\) with categories \(u,v\)
# A tibble: 10 × 3
      id X1    X2   
   <int> <fct> <fct>
 1     1 A     u    
 2     2 A     u    
 3     3 A     v    
 4     4 B     u    
 5     5 B     u    
 6     6 B     v    
 7     7 C     u    
 8     8 C     v    
 9     9 C     v    
10    10 C     v

Indicator matrices

\[ {\bf Z}_1 = \begin{pmatrix} 1&0&0\\ 1&0&0\\ 1&0&0\\ 0&1&0\\ 0&1&0\\ 0&1&0\\ 0&0&1\\ 0&0&1\\ 0&0&1\\ 0&0&1 \end{pmatrix}, \qquad {\bf Z}_2 = \begin{pmatrix} 1&0\\ 1&0\\ 0&1\\ 1&0\\ 1&0\\ 0&1\\ 1&0\\ 0&1\\ 0&1\\ 0&1 \end{pmatrix}. \]

association-based distance: from \({\bf Z}\) to \({\bf P}\)

Let

\[ {\bf Z} = [{\bf Z}_1,{\bf Z}_2]. \]

The empirical co-occurrence matrix is

\[ {\bf P} = \frac{1}{10}{\bf Z}^{\sf T}{\bf Z}. \]

For this example,

\[ {\bf P} = \begin{pmatrix} \color{#2A9D8F}{0.30} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#E76F51}{0.20} & \color{#E76F51}{0.10}\\ \color{#2A9D8F}{0} & \color{#2A9D8F}{0.30} & \color{#2A9D8F}{0} & \color{#E76F51}{0.20} & \color{#E76F51}{0.10}\\ \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0.40} & \color{#E76F51}{0.10} & \color{#E76F51}{0.30}\\ \color{#E76F51}{0.20} & \color{#E76F51}{0.20} & \color{#E76F51}{0.10} & \color{#2A9D8F}{0.50} & \color{#2A9D8F}{0}\\ \color{#E76F51}{0.10} & \color{#E76F51}{0.10} & \color{#E76F51}{0.30} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0.50} \end{pmatrix}. \]

diagonal blocks contain marginal information; off-diagonal blocks contain joint proportions

association-based distance: from \({\bf P}\) to \({\bf R}\)

The diagonal part of \({\bf P}\) is

\[ {\bf P}_z = {\bf P} \odot {\bf I}_{Q^*} = \operatorname{diag}(0.30,0.30,0.40,0.50,0.50). \]

The block matrix of conditional profiles is

\[ {\bf R} = {\bf P}_z^{-1}({\bf P}-{\bf P}_z). \]

For this example,

\[ {\bf R} = \begin{pmatrix} \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#E76F51}{0.67} & \color{#E76F51}{0.33}\\ \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#E76F51}{0.67} & \color{#E76F51}{0.33}\\ \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0} & \color{#E76F51}{0.25} & \color{#E76F51}{0.75}\\ \color{#E76F51}{0.40} & \color{#E76F51}{0.40} & \color{#E76F51}{0.20} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0}\\ \color{#E76F51}{0.20} & \color{#E76F51}{0.20} & \color{#E76F51}{0.60} & \color{#2A9D8F}{0} & \color{#2A9D8F}{0} \end{pmatrix}. \]

each off-diagonal block contains conditional profiles across variables

association-based distance: reading \({\bf R}_{12}\)

For the categories of \(X_1\), the relevant block is

\[ {\bf R}_{12} = \begin{pmatrix} 0.67 & 0.33\\ 0.67 & 0.33\\ 0.25 & 0.75 \end{pmatrix}. \]

Interpretation

Rows of \({\bf R}_{12}\) describe the distribution of \(X_2\) within each category of \(X_1\):

  • \(P(X_2=u \mid X_1=A)=\color{#E76F51}{\mathbf{0.67}}\), \(P(X_2=v \mid X_1=A)=\color{#E76F51}{\mathbf{0.33}}\)
  • \(P(X_2=u \mid X_1=B)=\color{#E76F51}{\mathbf{0.67}}\), \(P(X_2=v \mid X_1=B)=\color{#E76F51}{\mathbf{0.33}}\)
  • \(P(X_2=u \mid X_1=C)=\color{#E76F51}{\mathbf{0.25}}\), \(P(X_2=v \mid X_1=C)=\color{#E76F51}{\mathbf{0.75}}\)

categories are compared through their association profiles

association-based distance: from \({\bf R}\) to \(\Delta_1^{(tvd)}\)

Compare the rows of \({\bf R}_{12}\) using TVD.

\[ \delta^{1}_{tvd}(A,B) = \frac{1}{2} \left( |0.67-0.67| + |0.33-0.33| \right) = 0. \]

\[ \delta^{1}_{tvd}(A,C) = \frac{1}{2} \left( |0.67-0.25| + |0.33-0.75| \right) = 0.42. \]

\[ \delta^{tvd}_{1}(B,C) = 0.42. \]

Therefore,

\[ \Delta^{(tvd)}_1 = \begin{pmatrix} 0 & 0 & 0.42\\ 0 & 0 & 0.42\\ 0.42 & 0.42 & 0 \end{pmatrix}. \]

\(A\) and \(B\) are close because they have the same profile with respect to \(X_2\)

association-based distance: reading \({\bf R}_{21}\)

For the categories of \(X_2\), the relevant block is

\[ {\bf R}_{21} = \begin{pmatrix} 0.40 & 0.40 & 0.20\\ 0.20 & 0.20 & 0.60 \end{pmatrix}. \]

Interpretation

Rows of \({\bf R}_{21}\) describe the distribution of \(X_1\) within each category of \(X_2\):

  • \(P(X_1=A \mid X_2=u)=\color{#E76F51}{\mathbf{0.40}}\), \(P(X_1=B \mid X_2=u)=\color{#E76F51}{\mathbf{0.40}}\), \(P(X_1=C \mid X_2=u)=\color{#E76F51}{\mathbf{0.20}}\)
  • \(P(X_1=A \mid X_2=v)=\color{#E76F51}{\mathbf{0.20}}\), \(P(X_1=B \mid X_2=v)=\color{#E76F51}{\mathbf{0.20}}\), \(P(X_1=C \mid X_2=v)=\color{#E76F51}{\mathbf{0.60}}\)

categories of \(X_2\) are compared through their association profiles with respect to \(X_1\)

association-based distance: from \({\bf R}_{21}\) to \(\Delta_2^{(tvd)}\)

Compare the rows of \({\bf R}_{21}\) using TVD.

\[ \delta^{tvd}_{1}(u,v) = \frac{1}{2} \left( |0.40-0.20| + |0.40-0.20| + |0.20-0.60| \right) = 0.40. \]

Therefore,

\[ \Delta^{(tvd)}_2 = \begin{pmatrix} 0 & 0.40\\ 0.40 & 0 \end{pmatrix}. \]

\(u\) and \(v\) are different because they imply different profiles over \(X_1\)

association-based distance: from \(\Delta\) to \({\bf D}\)

We collect the category dissimilarity matrices in a block-diagonal matrix:

\[ \Delta^{(tvd)} = \begin{pmatrix} \color{#2A9D8F}{\Delta^{(tvd)}_1} & \color{#E76F51}{0}\\ \color{#E76F51}{0} & \color{#2A9D8F}{\Delta^{(tvd)}_2} \end{pmatrix}. \]

The observation-level categorical distance matrix is then

\[ {\bf D}_{tvd} = {\bf Z}\Delta^{(tvd)}{\bf Z}^{\sf T} = \begin{bmatrix} {\bf Z}_1 & {\bf Z}_2 \end{bmatrix} \begin{pmatrix} \Delta^{(tvd)}_1 & 0\\ 0 & \Delta^{(tvd)}_2 \end{pmatrix} \begin{bmatrix} {\bf Z}_1^{\sf T}\\ {\bf Z}_2^{\sf T} \end{bmatrix}. \]

Equivalently,

\[ {\bf D}_{tvd} = {\bf Z}_1\Delta^{(tvd)}_1{\bf Z}_1^{\sf T} + {\bf Z}_2\Delta^{(tvd)}_2{\bf Z}_2^{\sf T}. \]

category-level dissimilarities are translated into observation-level distances

No one-size-fits-all distance

The need for diagnostics

From distances to data representation

Different distance definitions induce different distance-based representations of the same data.

Same data, different representation

Changing the distance changes the global dissimilarity structure on which downstream learning methods rely.

Leave-one-variable-out diagnostics

How can we measure the contribution of each variable to this structure?

  • compare the dissimilarity matrix computed with and without the variable in question

LOVO-based benchmark: evaluated distances

The benchmark compares distance definitions that differ in how they treat scale, type, additivity, and association.

Additive distances

  • gower: classical Gower dissimilarity

  • mod_gower: modified Gower coefficients (Liu et al., 2024)

  • hl_add: additive version of Hennig–Liao scaling (C. Hennig & Liao, 2013)

  • u_ind: unbiased independence-based distance

  • u_dep: unbiased association-based distance

  • u_mix: unbiased Manhattan and TVD

Non-additive distances

  • naive: Euclidean distance on scaled numerical variables and one-hot-encoded

  • hl: Hennig–Liao scaling with Euclidean distance

  • gudmm: generalized multi-aspect distance metric for mixed-type data (Mousavi & Sehhati, 2023)

  • dkps: distance using kernel product similarity (Ghashti & Thompson, 2025)

LOVO-based diagnostics: what is evaluated?

For each distance and each variable \(X_j\), we compare the full-data representation \({\bf D}\) with the representation obtained after removing \(X_j\), that is \({\bf D}_{-j}\).

1. Distance level

Numeric comparision between \({\bf D}\) and \({\bf D}_{-j}\).

 

  • mean absolute difference between distance matrices.

2. MDS level

Compute MDS from \({\bf D}\) and from \({\bf D}_{-j}\), then compare the resulting configurations.

 

  • alienation coefficient between MDS representations.

LOVO diagnostics assess how each variable contributes to the dissimilarity structure

LOVO diagnostics: distance-level effect

LOVO diagnostics: MDS-level effect

  • commensurability balances expected distance contributions, not necessarily the role of variables in every downstream representation

From diagnostics to downstream learning

LOVO diagnostics show how variables affect the distance matrix and the MDS representation.

 

But we also want to know whether distance biases affect a downstream learning task.

Clustering experiment

Use each distance matrix as input to PAM and evaluate how well the resulting partition recovers the known cluster structure.

Clustering experiment

Data generation

  • (n = 200) observations from (4) equal-sized clusters
  • data generated with genRandomClust
  • each dataset contains (8) numerical and (8) categorical variables
  • categorical variables are obtained by discretizing numerical variables into (9) categories
  • scenarios vary the number of signal and noise variables within each type
  • (100) datasets are generated for each scenario

Evaluation

For each mixed-data distance, PAM is applied to the dissimilarity matrix with (K = 4).
Recovery of the true cluster labels is measured using the adjusted Rand index.

PAM-based clustering results

  • hl performs well when categorical variables are noise, but poorly when numerical variables are noise
  • gower tends to show the opposite pattern
  • u_mix and u_dep are comparatively stable in the mixed signal/noise scenarios

4. Interaction-aware extensions

Cross-type interactions

Association-aware distances account for relations within variable blocks:

  • continuous–continuous relations;
  • categorical–categorical relations.

Cross-type structure

In mixed data, categorical differences may be meaningful because they are reflected in the continuous variables.

the next step is to make distances interaction-aware

The numerical geometry suggests two groups

Two interlocking half-moons shown without categorical labels.

2

connected shapes

A numerical distance can recover the upper (U) and lower (L) moon.

Geometry alone

There is no preferred subdivision within either moon.

One factor is structured; the other is deliberately neutral

Half-moons with signal band mapped to colour and noise category mapped to shape.

Signal band

  • colour: \(A,B,C\)
  • unequal arc lengths: \(10\%,45\%,45\%\)
  • repeated on both moons

Noise factor

  • shape: \(N_1,N_2,N_3\)
  • balanced within every band
  • spatially random

only the signal factor has a systematic relationship with the numerical geometry

The target is the moon AND the signal category

\[ \text{cluster} = \text{moon}\times\text{band} \]

The desired partition contains six groups:

\[ \begin{aligned} &U\!A,\ U\!B,\ U\!C,\\ &L\!A,\ L\!B,\ L\!C. \end{aligned} \]

The six-way intersection

Moon alone gives two groups. Band alone gives three.

The six target groups formed by crossing upper and lower moon with bands A, B, and C.

Without interaction, the graph follows geometry

Six-cluster spectral partition obtained without the continuous-categorical interaction.

ARI = 0.490

Spectral clustering is asked for \(K=6\), but the graph does not know where \(A\), \(B\), and \(C\) change.

The wrong boundaries

The moons are subdivided according to numerical geometry—not the signal factor.

With interaction, the graph recovers the six intersections

The exact six-group spectral partition obtained with the continuous-categorical interaction.

ARI = 1.000

The numerical distance preserves the two moons.

\(\Delta_{\mathrm{int}}\) adds barriers at the \(A/B/C\) boundaries.

Exact recovery

The graph now encodes the required moon AND category structure.

How to measure interactions1

Define \(\Delta^{int}\) to account for continuous–categorical interactions and use it to augment \(\Delta^{tvd}\).

The mixed dissimilarity becomes

\[ {\bf D}_{mix}^{(int)} = {\bf D}_{mah} + {\bf D}_{cat}^{(int)}. \]

where

\[ {\bf D}_{cat}^{(int)}={\bf Z}\tilde{\Delta}{\bf Z}^\top \]

and

\[ \tilde{\Delta} = (1-\alpha)\Delta^{tvd} + \alpha \Delta^{int}, \qquad \alpha=\frac{1}{Q_c}. \]

What is \(\Delta^{int}\)?

The entry \(\delta_{int}^{j}(a,b)\) measures how much the continuous variables help discriminate between observations choosing category \(a\) and those choosing category \(b\) for categorical variable \(j\).

Category-pair classification problem

For each pair \((a,b)\):

  • use the continuous variables as predictors;
  • classify observations belonging to categories \(a\) and \(b\);
  • use a nearest-neighbour rule in the continuous space.

Computing \(\Delta^{int}_{j}\)

For each categorical variable \(j\) and each category pair \((a,b)\):

  1. use \({\bf D}_{mah}\) to identify neighbours among observations belonging to \(a\) or \(b\);
  2. consider a proportion of neighbours, say \(\pi_{nn}=0.1\);
  3. classify observations using a prior-corrected decision rule;
  4. compute balanced accuracy.

\[ \operatorname{BAcc}^{j}(a,b) = \frac{1}{2} \left[ \operatorname{TPR}^{j}_{a}(a,b) + \operatorname{TPR}^{j}_{b}(a,b) \right]. \]

Map predictive performance onto a non-negative separability scale:

\[ \delta^{j}_{int}(a,b) = \max\left\{ 0,\, 2\operatorname{BAcc}^{j}(a,b)-1 \right\} \in[0,1]. \]

\(0\) means chance-level or worse; \(1\) means perfect separation

Computing \(\Delta^{int}_{j}\)

For categorical variable \(j\) with \(q_j\) categories, compute
\(\frac{q_j(q_j -1)}{2}\) category-pair quantities.

\[ \Delta_{int} = \begin{pmatrix} 0 & \cdot & \cdot & \cdot \\ \cdot & 0 & \cdot & \cdot \\ \cdot & \cdot & 0 & \cdot\\ \cdot & \cdot & \cdot & 0 \end{pmatrix} \]

Computing \(\Delta^{int}_{j}\)

\[ \Delta_{int} = \begin{pmatrix} 0 & \color{#E76F51}{0.94} & \cdot & \cdot \\ \color{#E76F51}{0.94} & 0 & \cdot & \cdot \\ \cdot & \cdot & 0 & \cdot\\ \cdot & \cdot & \cdot & 0 \end{pmatrix} \]

Computing \(\Delta^{int}_{j}\)

\[ \Delta_{int} = \begin{pmatrix} 0 & 0.94 & \color{#E76F51}{0.40} & \cdot \\ 0.94 & 0 & \cdot & \cdot \\ \color{#E76F51}{0.40} & \cdot & 0 & \cdot\\ \cdot & \cdot & \cdot & 0 \end{pmatrix} \]

Computing \(\Delta^{int}_{j}\)

\[ \Delta_{int} = \begin{pmatrix} 0 & 0.94 & 0.40 & \color{#E76F51}{0.39} \\ 0.94 & 0 & \cdot & \cdot \\ 0.40 & \cdot & 0 & \cdot\\ \color{#E76F51}{0.39} & \cdot & \cdot & 0 \end{pmatrix} \]

Computing \(\Delta^{int}_{j}\)

\[ \Delta_{int} = \begin{pmatrix} 0 & 0.94 & 0.40 & 0.39 \\ 0.94 & 0 & \color{#E76F51}{0.54} & \cdot \\ 0.40 & \color{#E76F51}{0.54} & 0 & \cdot \\ 0.39 & \cdot & \cdot & 0 \end{pmatrix} \]

Computing \(\Delta^{int}_{j}\)

\[ \Delta_{int} = \begin{pmatrix} 0 & 0.94 & 0.40 & 0.39 \\ 0.94 & 0 & 0.54 & \color{#E76F51}{0.55} \\ 0.40 & 0.54 & 0 & \cdot \\ 0.39 & \color{#E76F51}{0.55} & \cdot & 0 \end{pmatrix} \]

Computing \(\Delta^{int}_{j}\)

\[ \Delta_{int} = \begin{pmatrix} 0 & 0.94 & 0.40 & 0.39 \\ 0.94 & 0 & 0.54 & 0.55 \\ 0.40 & 0.54 & 0 & \color{#E76F51}{0} \\ 0.39 & 0.55 & \color{#E76F51}{0} & 0 \end{pmatrix} \]

summarize category-pair separability in the continuous space

Just one-way interaction?

Factorization motivates the direction

For \({\bf x}_i=({\bf x}_{i_n},{\bf x}_{i_c})\),

\[ f({\bf x}_{i_n},{\bf x}_{i_c}) = f({\bf x}_{i_n}) f({\bf x}_{i_c}\mid {\bf x}_{i_n}). \]

We therefore use the numerical block as the reference representation for categorical distinctions.

Dissimilarity and a reference density

For a non-negative dissimilarity to a prototype, when the exponential kernel is normalisable,

\[ f({\bf x};{\bf c},{\bf S}) = g({\bf c},{\bf S},\eta) \exp\{-\eta d_{\bf S}({\bf x},{\bf c})\}. \]

Since \(M=f({\bf c};{\bf c},{\bf S})=g({\bf c},{\bf S},\eta)\),

\[ \begin{aligned} \overline K({\bf x},{\bf c}) &=\frac{f({\bf x};{\bf c},{\bf S})}{M} =\exp\{-\eta d_{\bf S}({\bf x},{\bf c})\},\\ d_{\bf S}({\bf x},{\bf c}) &=-\frac{1}{\eta}\log\overline K({\bf x},{\bf c}). \end{aligned} \]

Pairwise consequence

Take another observation as the reference prototype, \({\bf c}\leftarrow{\bf x}_{i'}\), and set \(\eta=1\):

\[ \begin{aligned} \overline K_{\mathrm{mix}}(i,i') &= \overline K_{\mathrm n}(i,i')\,\overline K_{\mathrm c\mid n}(i,i') \\ -\log\overline K_{\mathrm{mix}}(i,i') &= d_{\mathrm n}(i,i')+d_{\mathrm c\mid n}(i,i'). \end{aligned} \]

Distance-to-prototype construction: (Ben-Israel & Iyigun, 2008; Christian Hennig, Viroli, & Anderlucci, 2019). Full mixed-data derivation: (Iodice D’Enza et al., 2026).

downstream analysis

Spectral clustering: a graph partitioning problem

Graph representation

A graph representation of the data matrix \({\bf X}\): the aim is to cut it into \(K\) groups, or clusters.

The affinity matrix \({\bf A}\)

The elements \({\bf w}_{ij}\) of \({\bf A}\) are high when observations \(i\) and \(j\) are likely to belong to the same group, and low otherwise.

Spectral clustering: making the graph easy to cut

An approximate solution to the graph partitioning problem (A. Ng, Jordan, & Weiss, 2001):

From distances to affinities

Start from the pairwise distance matrix \({\bf D}\) and build the affinity matrix

\[ {\bf A} = \exp\left(-\frac{{\bf D}^{2}}{2\sigma^{2}}\right), \qquad a_{ii}=0. \]

The parameter \(\sigma\) controls the neighbourhood scale.

Normalized graph Laplacian

\[ {\bf L} = {\bf D}_{r}^{-1/2} {\bf A} {\bf D}_{r}^{-1/2} = {\bf Q}{\Lambda}{\bf Q}^{\sf T}, \]

where \({\bf D}_{r}=\operatorname{diag}({\bf r})\), \({\bf r}={\bf A}{\bf 1}\), \({\bf 1}\) is an \(I\)-dimensional vector of ones.

Spectral embedding

apply \(K\)-means to the rows of \({\bf \tilde Q}\), containing the first \(K\) eigenvectors of \({\bf L}\).

Why spectral clustering here?

Interaction-aware distances can encode local connectivity and non-convex structure.

Synthetic experiments

Simulation 1: associated mixed variables

General design

  • two sample sizes: \(I = 1000\) and \(I = 2000\)
  • number of clusters fixed to \(K = 3\)
  • 50 datasets generated for each configuration
  • spectral clustering applied to each dissimilarity matrix
  • performance measured by adjusted Rand index

Variable composition

Four scenarios were considered:

  1. \(Q_n = 15\), \(Q_c = 15\)
  2. \(Q_n = 10\), \(Q_c = 20\)
  3. \(Q_n = 20\), \(Q_c = 10\)
  4. \(Q_n = 10\), \(Q_c = 10\) plus noise

Aim

The simulation creates mixed data with:

  • correlated numerical variables;
  • associated categorical variables;
  • categorical variables depending on numerical variables;
  • an additional noise scenario.

Simulation 1: associated mixed variables

Data-generating idea

  1. Generate numerical variables from cluster-specific multivariate normal distributions.

    • different cluster means;
    • different correlation structures.

  2. Generate binary categorical variables conditionally on the numerical variables.

    • the first categorical variable depends on the numerical block;
    • later categorical variables depend on the numerical block and on previous categorical variables.

the design induces numerical correlations, categorical associations, and numerical–categorical dependence

Simulation 1: results

Simulation 2: interaction-driven clusters

Design

  • \(I = 500, 1000\)
  • six continuous variables and three categorical variables
  • \(V_4,V_5,V_6\) generated independently from \(N(0,1)\)
  • \(V_1,V_2,V_3\) generated conditionally on \(C_1\) and \(C_2\)

Main feature

The clusters are not defined by continuous variables alone or categorical variables alone.

They are defined by their cross-type interaction.

this is the setting where \(\Delta^{int}\) should matter

Simulation 2: results

5. Response-aware extensions

Response-aware distances for KNN

KNN is usually described as a lazy learner:

  • store the training data;
  • compute distances from a new observation to the training observations;
  • predict from the responses of the nearest neighbours.

Reframing KNN

The distance is not just a preprocessing choice.
It determines the neighbourhoods used for classification or regression.

in supervised learning, the response can help define these neighbourhoods

Response-aware mixed distance

For mixed-type predictors, use a supervised distance with two components:

\[ D_{i\ell} = D^{(n)}\!\left({\bf x}_i^{(n)},{\bf x}_\ell^{(n)}\right) + D^{(c)}\!\left({\bf x}_i^{(c)},{\bf x}_\ell^{(c)}\right). \]

Numerical part

Use discriminant information from \(y\)
to weight numerical differences.

Categorical part

Use the association between categories and \(y\)
to define category dissimilarities.

Numerical part: discriminant weighting1

For continuous predictors, use the response to weight directions or variables.

Single-variable discriminant weighting

For numerical variable \(j\), define the Fisher score

\[ \omega_j = \frac{B_j}{W_j}, \]

where \(B_j\) and \(W_j\) are the between- and within-group variances.

Then a supervised Manhattan-type distance is

\[ D^{(n)}\!\left({\bf x}_i^{(n)},{\bf x}_\ell^{(n)}\right) = \sum_{j=1}^{Q_n} \sqrt{\omega_j} \left|x_{ij}^{(n)}-x_{\ell j}^{(n)}\right|. \]

Categorical part: supervised TVD

For categorical predictors, compare categories through their response profiles.

Let \({\bf Z}_y\) be the indicator matrix of the response.
The supervised profile matrix is

\[ {\bf R}_s = {\bf P}_d^{-1} {\bf Z}^{\sf T}{\bf Z}_y. \]

The supervised category dissimilarity is

\[ \delta_s^j(a,b) = \frac{1}{2} \sum_{\ell=1}^{q_y} \left| {\bf r}_{a\ell}^{j y} - {\bf r}_{b\ell}^{j y} \right|. \]

categories are close if they show similar response distributions

Response-aware KNN: Carseats example

Data

The Carseats data are used to predict whether sales are high.

  • 7 numerical predictors
  • 3 categorical predictors
  • categorical predictors with 2 or 3 categories
  • binary response: high vs. low sales

Compared distances

  • gower: robust Manhattan + matching
  • naive: Euclidean on scaled numerical variables and dummies
  • sup: supervised numerical weighting + supervised TVD
  • sup_add: additive supervised version
  • supf: full supervised version

Response-aware KNN: Carseats results

  • response-aware distances improve nearest-neighbour classification accuracy
  • sup, sup_add, and supf are clearly above gower and naive
  • the response helps define neighbourhoods that better reflect the class structure

Building distance-based pipelines

Building distance-based pipelines

manydist

A package to construct, diagnose, and use distances for continuous, categorical, and mixed-type data.

Distance construction

  • mdist()
    • presets and custom specifications
    • response-aware distances
  • step_mdist()
    • integrates distance construction into tidymodels workflows

Diagnostics

  • lovo_mdist()
  • compare_lovo_mdist()
  • benchmark_mdist()

Learning: model specs

Unsupervised learning

  • pam_dist()
  • spectral_dist()

Supervised learning

  • nearest_neighbor_dist()

manydist: socio-economic country profiles

Data

A 2022 World Bank / WDI snapshot of country-level socio-economic indicators.

  • observations: countries
  • numerical variables: GDP per capita, life expectancy, unemployment, urban population, population growth
  • categorical variables: world region and income group

Use manydist to build a mixed-type distance between countries and diagnose which variables shape the resulting dissimilarity structure.

Preparing the WDI data

Country Region Income group World Bank lending category GDP per capita (k USD) Life expectancy (years) Unemployment (%) Urban population (% total) Population growth (%)
Tajikistan Europe & Central Asia Lower middle income IDA 1.1 71.6 7.1 26.2 2.14
West Bank and Gaza Middle East, North Africa, Afghanistan & Pakistan Lower middle income Not classified 3.8 76.7 24.4 86.6 2.43
Belarus Europe & Central Asia Upper middle income IBRD 8.0 74.1 3.6 78.5 -0.80
United Arab Emirates Middle East, North Africa, Afghanistan & Pakistan High income Not classified 50.8 80.5 2.9 85.5 5.09
El Salvador Latin America & Caribbean Upper middle income IBRD 5.1 72.0 3.0 74.1 0.39
New Zealand East Asia & Pacific High income Not classified 49.1 82.0 3.3 83.9 -0.06
Cyprus Europe & Central Asia High income Not classified 33.2 80.4 6.8 66.7 1.06
Zambia Sub-Saharan Africa Lower middle income IDA 1.4 65.3 6.0 44.6 2.76

Constructing a mixed-type distance using a preset

Directly with mdist()

wdi_x <- wdi_data |> 
  dplyr::select(-country)

d_preset <- mdist(
  wdi_x,
  preset = "u_dep"
)

Inside a recipe with step_mdist()

rec_preset <- recipe(~ ., data = wdi_x) |>
  step_mdist(
    all_predictors(),
    preset = "u_dep"
  )

d_preset_step <- rec_preset |>
  prep(training = wdi_x) |>
  bake(new_data = NULL)
all.equal(
  unname(as.matrix(d_preset$distance)),
  unname(as.matrix(d_preset_step))
)
[1] TRUE

step_mdist() embeds the same distance specification into a modelling workflow

Constructing a custom mixed-type distance

Directly with mdist()

wdi_x <- wdi_data |> 
  dplyr::select(-country)

d_custom <- mdist(
  wdi_x,
  method_cat  = "eskin",
  method_num  = "std",
  commensurable = FALSE
)

Inside a recipe with step_mdist()

rec_custom <- recipe(~ ., data = wdi_x) |>
  step_mdist(
    all_predictors(),
    method_cat  = "eskin",
    method_num  = "std",
    commensurable = FALSE
  )

d_custom_step <- rec_custom |>
  prep(training = wdi_x) |>
  bake(new_data = NULL)
all.equal(
  unname(as.matrix(d_custom$distance)),
  unname(as.matrix(d_custom_step))
)
[1] TRUE

Comparing distance constructions

lovo_mdist_compare()

The same leave-one-variable-out diagnostic can be computed for several distance definitions and compared in one display.

Distance-based classification pipeline

set.seed(123)
wdi_region <- wdi_data |>
  dplyr::filter(region != "North America") |>
  dplyr::mutate(
    region = droplevels(region)
  )
wdi_split <- initial_split(
  wdi_region,
  strata = region
)
wdi_train <- training(wdi_split)
wdi_test  <- testing(wdi_split)
wdi_rec <- recipe(region ~ ., data = wdi_train) |>
  update_role(country, new_role = "id") |>
  step_mdist(
    all_predictors(),
    preset = "u_dep"
  )
knn_spec <- nearest_neighbor_dist(
  mode = "classification",
  neighbors = tune()
)
wdi_wf <- workflow() |>
  add_recipe(wdi_rec) |>
  add_model(knn_spec)

Data

Prepare the classification task.

Distance-based classification pipeline

set.seed(123)
wdi_region <- wdi_data |>
  dplyr::filter(region != "North America") |>
  dplyr::mutate(
    region = droplevels(region)
  )
wdi_split <- initial_split(
  wdi_region,
  strata = region
)
wdi_train <- training(wdi_split)
wdi_test  <- testing(wdi_split)
wdi_rec <- recipe(region ~ ., data = wdi_train) |>
  update_role(country, new_role = "id") |>
  step_mdist(
    all_predictors(),
    preset = "u_dep"
  )
knn_spec <- nearest_neighbor_dist(
  mode = "classification",
  neighbors = tune()
)
wdi_wf <- workflow() |>
  add_recipe(wdi_rec) |>
  add_model(knn_spec)

Data

Prepare the classification task.

Split

Create training and test sets.

Distance-based classification pipeline

set.seed(123)
wdi_region <- wdi_data |>
  dplyr::filter(region != "North America") |>
  dplyr::mutate(
    region = droplevels(region)
  )
wdi_split <- initial_split(
  wdi_region,
  strata = region
)
wdi_train <- training(wdi_split)
wdi_test  <- testing(wdi_split)
wdi_rec <- recipe(region ~ ., data = wdi_train) |>
  update_role(country, new_role = "id") |>
  step_mdist(
    all_predictors(),
    preset = "u_dep"
  )
knn_spec <- nearest_neighbor_dist(
  mode = "classification",
  neighbors = tune()
)
wdi_wf <- workflow() |>
  add_recipe(wdi_rec) |>
  add_model(knn_spec)

Data

Prepare the classification task.

Split

Create training and test sets.

Recipe

Use step_mdist() to construct the distance representation.

Distance-based classification pipeline

set.seed(123)
wdi_region <- wdi_data |>
  dplyr::filter(region != "North America") |>
  dplyr::mutate(
    region = droplevels(region)
  )
wdi_split <- initial_split(
  wdi_region,
  strata = region
)
wdi_train <- training(wdi_split)
wdi_test  <- testing(wdi_split)
wdi_rec <- recipe(region ~ ., data = wdi_train) |>
  update_role(country, new_role = "id") |>
  step_mdist(
    all_predictors(),
    preset = "u_dep"
  )
knn_spec <- nearest_neighbor_dist(
  mode = "classification",
  neighbors = tune()
)
wdi_wf <- workflow() |>
  add_recipe(wdi_rec) |>
  add_model(knn_spec)

Data

Prepare the classification task.

Split

Create training and test sets.

Recipe

Use step_mdist() to construct the distance representation.

Model

Specify a distance-based KNN classifier.

Distance-based classification pipeline

set.seed(123)
wdi_region <- wdi_data |>
  dplyr::filter(region != "North America") |>
  dplyr::mutate(
    region = droplevels(region)
  )
wdi_split <- initial_split(
  wdi_region,
  strata = region
)
wdi_train <- training(wdi_split)
wdi_test  <- testing(wdi_split)
wdi_rec <- recipe(region ~ ., data = wdi_train) |>
  update_role(country, new_role = "id") |>
  step_mdist(
    all_predictors(),
    preset = "u_dep"
  )
knn_spec <- nearest_neighbor_dist(
  mode = "classification",
  neighbors = tune()
)
wdi_wf <- workflow() |>
  add_recipe(wdi_rec) |>
  add_model(knn_spec)

Data

Prepare the classification task.

Split

Create training and test sets.

Recipe

Use step_mdist() to construct the distance representation.

Model

Specify a distance-based KNN classifier.

Workflow

Combine preprocessing and model specification.

Tuning a distance-based KNN pipeline

set.seed(123)
wdi_folds <- vfold_cv(
  wdi_train,
  v = 5,
  strata = region
)
knn_grid <- tibble(
  neighbors = c(1, 3, 5, 7, 9, 11, 15)
)
knn_tuned <- tune_grid(
  wdi_wf,
  resamples = wdi_folds,
  grid = knn_grid,
  metrics = metric_set(accuracy)
)
best_k <- select_best(
  knn_tuned,
  metric = "accuracy"
)
final_wf <- finalize_workflow(
  wdi_wf,
  best_k
)
final_res <- last_fit(
  final_wf,
  split = wdi_split,
  metrics = metric_set(accuracy)
)

Resample

Create cross-validation folds on the training set.

Tuning a distance-based KNN pipeline

set.seed(123)
wdi_folds <- vfold_cv(
  wdi_train,
  v = 5,
  strata = region
)
knn_grid <- tibble(
  neighbors = c(1, 3, 5, 7, 9, 11, 15)
)
knn_tuned <- tune_grid(
  wdi_wf,
  resamples = wdi_folds,
  grid = knn_grid,
  metrics = metric_set(accuracy)
)
best_k <- select_best(
  knn_tuned,
  metric = "accuracy"
)
final_wf <- finalize_workflow(
  wdi_wf,
  best_k
)
final_res <- last_fit(
  final_wf,
  split = wdi_split,
  metrics = metric_set(accuracy)
)

Resample

Create cross-validation folds on the training set.

Grid

Define candidate values for the number of neighbours.

Tuning a distance-based KNN pipeline

set.seed(123)
wdi_folds <- vfold_cv(
  wdi_train,
  v = 5,
  strata = region
)
knn_grid <- tibble(
  neighbors = c(1, 3, 5, 7, 9, 11, 15)
)
knn_tuned <- tune_grid(
  wdi_wf,
  resamples = wdi_folds,
  grid = knn_grid,
  metrics = metric_set(accuracy)
)
best_k <- select_best(
  knn_tuned,
  metric = "accuracy"
)
final_wf <- finalize_workflow(
  wdi_wf,
  best_k
)
final_res <- last_fit(
  final_wf,
  split = wdi_split,
  metrics = metric_set(accuracy)
)

Resample

Create cross-validation folds on the training set.

Grid

Define candidate values for the number of neighbours.

Tune

Evaluate each candidate value by cross-validation.

Tuning a distance-based KNN pipeline

set.seed(123)
wdi_folds <- vfold_cv(
  wdi_train,
  v = 5,
  strata = region
)
knn_grid <- tibble(
  neighbors = c(1, 3, 5, 7, 9, 11, 15)
)
knn_tuned <- tune_grid(
  wdi_wf,
  resamples = wdi_folds,
  grid = knn_grid,
  metrics = metric_set(accuracy)
)
best_k <- select_best(
  knn_tuned,
  metric = "accuracy"
)
final_wf <- finalize_workflow(
  wdi_wf,
  best_k
)
final_res <- last_fit(
  final_wf,
  split = wdi_split,
  metrics = metric_set(accuracy)
)

Resample

Create cross-validation folds on the training set.

Grid

Define candidate values for the number of neighbours.

Tune

Evaluate each candidate value by cross-validation.

Select

Choose the best-performing number of neighbours.

Tuning a distance-based KNN pipeline

set.seed(123)
wdi_folds <- vfold_cv(
  wdi_train,
  v = 5,
  strata = region
)
knn_grid <- tibble(
  neighbors = c(1, 3, 5, 7, 9, 11, 15)
)
knn_tuned <- tune_grid(
  wdi_wf,
  resamples = wdi_folds,
  grid = knn_grid,
  metrics = metric_set(accuracy)
)
best_k <- select_best(
  knn_tuned,
  metric = "accuracy"
)
final_wf <- finalize_workflow(
  wdi_wf,
  best_k
)
final_res <- last_fit(
  final_wf,
  split = wdi_split,
  metrics = metric_set(accuracy)
)

Resample

Create cross-validation folds on the training set.

Grid

Define candidate values for the number of neighbours.

Tune

Evaluate each candidate value by cross-validation.

Select

Choose the best-performing number of neighbours.

Finalize

Insert the selected value into the workflow.

Tuning a distance-based KNN pipeline

set.seed(123)
wdi_folds <- vfold_cv(
  wdi_train,
  v = 5,
  strata = region
)
knn_grid <- tibble(
  neighbors = c(1, 3, 5, 7, 9, 11, 15)
)
knn_tuned <- tune_grid(
  wdi_wf,
  resamples = wdi_folds,
  grid = knn_grid,
  metrics = metric_set(accuracy)
)
best_k <- select_best(
  knn_tuned,
  metric = "accuracy"
)
final_wf <- finalize_workflow(
  wdi_wf,
  best_k
)
final_res <- last_fit(
  final_wf,
  split = wdi_split,
  metrics = metric_set(accuracy)
)

Resample

Create cross-validation folds on the training set.

Grid

Define candidate values for the number of neighbours.

Tune

Evaluate each candidate value by cross-validation.

Select

Choose the best-performing number of neighbours.

Finalize

Insert the selected value into the workflow.

Test

Fit the finalized workflow on the training set and evaluate it on the test set.

Tuning results

Test-set performance

Metric Accuracy
accuracy 0.553

Workflow

last_fit() fits the finalized workflow on the full training set and evaluates it once on the held-out test set.

Wrap-up

Increasing awareness works in two directions.

Distances must be data-aware

Construct distances that reflect the structure of the data they are applied to, including variable types, associations, and interactions.

 

Practitioners must be distance-aware

Understand what a chosen distance emphasizes or suppresses, and how that choice shapes downstream results.

 

A package makes this practical

manydist makes distance construction easy to use in learning pipelines. During resampling, step_mdist() fits the transformation on each analysis fold and applies it to the assessment fold, helping avoid data leakage.

Main references

Iodice D’Enza, A., Tortora, C., & Palumbo, F. (2026). Association-based spectral clustering for mixed data with cross-type interactions. Second-Round Review at Statistics and Computing.
van de Velden, M., Iodice D’Enza, A., Markos, A., & Cavicchia, C. (2026). Unbiased mixed-variable distance. Journal of Computational and Graphical Statistics. https://doi.org/10.1080/10618600.2026.2680181
Ghashti, J. S., & Thompson, J. R. J. (2025). Mixed-type distance shrinkage and selection for clustering via kernel metric learning. Journal of Classification, 42(2), 311–334.
Liu, P., Yuan, H., Ning, Y., Chakraborty, B., Liu, N., & Peres, M. A. (2024). A modified and weighted gower distance-based clustering analysis for mixed type data: A simulation and empirical analyses. BMC Medical Research Methodology, 24(305).
van de Velden, M., Iodice D’Enza, A., Markos, A., & Cavicchia, C. (2024). A general framework for implementing distances for categorical variables. Pattern Recognition, 153, 110547.
Mousavi, E., & Sehhati, M. (2023). A generalized multi-aspect distance metric for mixed-type data clustering. Pattern Recognition, 109353.
Hennig, Christian, Viroli, C., & Anderlucci, L. (2019). Quantile-based clustering. Electronic Journal of Statistics, 13(2), 4849–4883. https://doi.org/10.1214/19-EJS1640
Hennig, C., & Liao, T. F. (2013). How to Find an Appropriate Clustering for Mixed-Type Variables with Application to Socio-Economic Stratification. Journal of the Royal Statistical Society Series C: Applied Statistics, 62(3), 309–369.
Kaufman, L., & Rousseeuw, P. J. (2009). Finding groups in data: An introduction to cluster analysis. John Wiley & Sons.
Ben-Israel, A., & Iyigun, C. (2008). Probabilistic d-clustering. Journal of Classification, 25(1), 5–26.
Borg, I., & Groenen, P. J. (2005). Modern multidimensional scaling: Theory and applications. Springer.
Le, S. Q., & Ho, T. B. (2005). An association-based dissimilarity measure for categorical data. Pattern Recognition Letters, 26(16), 2549–2557.
Ng, A. Y., Jordan, M. I., & Weiss, Y. (2002). On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems, 14, 849–856. MIT Press.
Ng, A., Jordan, M., & Weiss, Y. (2001). On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems, 14.
Hastie, T., & Tibshirani, R. (1995). Discriminant adaptive nearest neighbor classification and regression. Advances in Neural Information Processing Systems, 8. MIT Press.

CARME 2027 in Corfu

CARME 2027 logo

June 22–25, 2027

Ionian University
Corfu, Greece

The tenth international CARME conference brings together work on correspondence analysis, categorical data, visualization, clustering, and related methods.

Invited speakers

Mikael Börjesson
Uppsala University

Paul Eilers
Erasmus University Medical Center

Patrick J. F. Groenen
Erasmus School of Economics

Trevor Hastie
Stanford University

Heungsun Hwang
McGill University

Ndèye Niang
CNAM Paris

carme2027corfu.wixsite.com/conference

Contact and slides

Contact

Alfonso Iodice D’Enza
iodicede@unina.it

GitHub Pages

https://alfonsoiodicede.github.io

Package website

QR code for the manydist package website

Open package website

Slides

QR code for the SFC26 presentation slides

Open slides

Backup: distance–density derivation

Just one-way interaction?

Factorization motivates the direction

Let \({\bf x}_i=\left({\bf x}_{i_{n}},{\bf x}_{i_{c}}\right).\)

The joint distribution can always be written as

\[ f({\bf x}_{i_{n}},{\bf x}_{i_{c}}) = f({\bf x}_{i_{n}}) f({\bf x}_{i_{c}}\mid{\bf x}_{i_{n}}). \]

This motivates using the continuous variables to quantify categorical distinctions:

\[ {\bf X}_{n} \quad\longrightarrow\quad \delta^{j}_{int}(a,b). \]

How can this result be connected to pairwise dissimilarities?

Just one-way interaction?

Dissimilarity and reference density

Starting from a non-negative dissimilarity from a prototype,

\[ d_{{\bf S}_k}({\bf x}_i,{\bf c}_k)\geq0, \qquad d_{{\bf S}_k}({\bf c}_k,{\bf c}_k)=0, \]

a distance-based probability model can be constructed, provided that the exponential kernel is normalisable 1:

\[ f({\bf x}_i;{\bf c}_k,{\bf S}_k) = g({\bf c}_k,{\bf S}_k,\eta) \exp\left\{ -\eta d_{{\bf S}_k}({\bf x}_i,{\bf c}_k) \right\}, \qquad \eta>0, \]

where \(g({\bf c}_k,{\bf S}_k,\eta)\) makes \(f\) a proper density.

Just one-way interaction?

Dissimilarity and reference density

Since the density is maximal at the reference point,

\[ M_k = \max_{\bf x}f({\bf x};{\bf c}_k,{\bf S}_k) = f({\bf c}_k;{\bf c}_k,{\bf S}_k) = g({\bf c}_k,{\bf S}_k,\eta). \]

the maximum-normalised kernel is

\[ \overline K_k({\bf x}_i,{\bf c}_k) = \frac{ f({\bf x}_i;{\bf c}_k,{\bf S}_k) }{ M_k } = \exp\left\{ -\eta d_{{\bf S}_k}({\bf x}_i,{\bf c}_k) \right\}. \]

Therefore,

\[ d_{{\bf S}_k}({\bf x}_i,{\bf c}_k) = -\frac{1}{\eta} \log\overline K_k({\bf x}_i,{\bf c}_k). \]

Just one-way interaction?

The pairwise continuous component

For pairwise comparison, let every observation act in turn as the reference prototype:

\[ {\bf c}_k \longrightarrow {\bf x}_{i'_{n}}, \qquad {\bf S}_k \longrightarrow {\bf S}. \]

Use the ordinary Mahalanobis distance and choose \(\eta=1\):

\[ f_{\mathrm{n}} ({\bf x}_{i_{n}};{\bf x}_{i'_{n}},{\bf S}) = g_{\mathrm{n}}({\bf S}) \exp\left\{ -d_{\mathrm{mah}} ({\bf x}_{i_{n}},{\bf x}_{i'_{n}}) \right\}. \] Using the reference maximum \(M_{\mathrm{n}}=g_{\mathrm{n}}({\bf S})\) from the preceding construction

the maximum-normalised continuous kernel is

\[ \overline K_{\mathrm{n}}(i,i') = \frac{ f_{\mathrm{n}} ({\bf x}_{i_{n}};{\bf x}_{i'_{n}},{\bf S}) }{ M_{\mathrm{n}} } = \exp\left\{ -d_{\mathrm{mah}} ({\bf x}_{i_{n}},{\bf x}_{i'_{n}}) \right\}. \]

\(-\log\overline K_{\mathrm{n}}(i,i')\) recovers the ordinary Mahalanobis distance

Just one-way interaction?

The pairwise interaction-aware categorical component

For categorical variable \(j\), let \(x_{ij_{\mathrm c}}=a\) and \(x_{i'j_{\mathrm c}}=b\).

The interaction-aware category dissimilarity is

\[ \widetilde{\delta}^{\,j}(a,b) = (1-\alpha)\delta^{j}_{tvd}(a,b) + \alpha\delta^{j}_{int}(a,b). \]

Assume that \(\widetilde{\delta}^{\,j}(a,b)\geq0\) and \(\widetilde{\delta}^{\,j}(b,b)=0\).

Choosing \(\eta_j=1\), define the categorical reference PMF

\[ p_j(a;b) = \operatorname{softmax}_{h\in\mathcal A_j} \left\{ -\widetilde{\delta}^{\,j}(h,b) \right\}_{a} = \frac{ \exp\left\{ -\widetilde{\delta}^{\,j}(a,b) \right\} }{ \displaystyle \sum_{h\in\mathcal A_j} \exp\left\{ -\widetilde{\delta}^{\,j}(h,b) \right\} }. \]

smaller category dissimilarities receive larger reference probabilities

Just one-way interaction?

The pairwise interaction-aware categorical component

Because \(\widetilde{\delta}^{\,j}(b,b)=0\), the maximum reference probability is

\[ M_j(b) = p_j(b;b) = \frac{1}{ \displaystyle \sum_{h\in\mathcal A_j} \exp\left\{ -\widetilde{\delta}^{\,j}(h,b) \right\} }. \]

The maximum-normalised categorical kernel

\[ \overline K_j(a,b) = \frac{p_j(a;b)}{M_j(b)} = \exp\left\{ -\widetilde{\delta}^{\,j}(a,b) \right\}, \]

\(-\log\overline K_j(a,b)=\widetilde{\delta}^{\,j}(a,b)\) recovers the interaction-aware dissimilarity

Just one-way interaction?

Combining the kernels

For the whole categorical vector, define the separable maximum-normalised kernel

\[ \begin{aligned} \overline K_{\mathrm{c}}^{(int)}(i,i') &= \prod_{j=1}^{Q_c} \overline K_j (x_{ij_{c}},x_{i'j_{c}}) \\ &= \exp\left\{ -\sum_{j=1}^{Q_c} \widetilde{\delta}^{\,j} (x_{ij_{c}},x_{i'j_{c}}) \right\} \\ &= \exp\left\{ -d_{\mathrm{c}}^{(int)} ({\bf x}_{i_{c}},{\bf x}_{i'_{c}}) \right\}. \end{aligned} \]

Just one-way interaction?

Combining the kernels

Combine the continuous and categorical kernels:

\[ \overline K_{\mathrm{mix}}^{(int)}(i,i') = \overline K_{\mathrm{n}}(i,i') \, \overline K_{\mathrm{c}}^{(int)}(i,i')= \exp\left\{ - d_{\mathrm{mah}} ({\bf x}_{i_{n}},{\bf x}_{i'_{n}}) - d_{\mathrm{c}}^{(int)} ({\bf x}_{i_{c}},{\bf x}_{i'_{c}}) \right\} \]

\(-log \overline{K}_{\mathrm{mix}}^{(int)}(i,i')=d_{\mathrm{mah}}({\bf x}_{i_{n}},{\bf x}_{i'_{n}}) +d_{\mathrm{c}}^{(int)}({\bf x}_{i_{c}},{\bf x}_{i'_{c}})\)

recovers the overall dissimilarity

 

in algebraic form

\({\bf D}_{mix}^{(int)} ={\bf D}_{mah}+{\bf Z}\widetilde{\Delta}{\bf Z}^{\sf T}.\)