Hot-deck matching imputation - MicroImpute documentation

This notebook demonstrates how to use Microimpute’s Matching imputer to impute values using the hot-deck matching approach.

Variable type support¶

The matching model can handle any variable type—numerical, categorical, boolean, or mixed. Since it transfers actual observed values from similar donor records rather than generating model-based predictions, it naturally preserves the original data type and distribution of each variable.

The matching model supports iterative imputation with a single object and workflow. Pass a list of imputed_variables with all variables that you hope to impute for and the model will do so without needing to fit and predict for each separately.

Matching class¶

class Matching(
    matching_hotdeck: MatchingHotdeckFn = nnd_hotdeck_using_rpy2,
    log_level: Optional[str] = "WARNING"
)

Parameter	Type	Default used	Description
matching_hotdeck	MatchingHotdeckFn	nnd_hotdeck_using_rpy2	Function that performs hot deck matching
log_level	str	“WARNING”	Logging verbosity level

fit() method¶

def fit(
    X_train: pd.DataFrame,
    predictors: List[str],
    imputed_variables: List[str],
    weight_col: Optional[str] = None,
    tune_hyperparameters: bool = False,
    constrained: bool = False,
    constr_alg: str = "lpSolve",
    dist_fun: str = "Gower",
    k: int = 5,
) -> Union[MatchingResults, Tuple[MatchingResults, Dict]]

Parameter	Type	Default used	Description
X_train	pd.DataFrame	-	Donor data with predictors and target variables
predictors	List[str]	-	Column names to use as matching variables
imputed_variables	List[str]	-	Column names to impute
weight_col	str	None	Column name for sampling weights
tune_hyperparameters	bool	False	Enable automatic hyperparameter tuning
constrained	bool	False	Use constrained matching (each donor used at most once)
constr_alg	str	“lpSolve”	Algorithm for constrained matching: “lpSolve” or “hungarian”
dist_fun	str	“Gower”	Distance function: “Gower”, “Manhattan”, “Euclidean”, etc.
k	int	5	Number of nearest neighbors to consider

It returns a MatchingResults object (or tuple with best hyperparameters if tuning enabled).

MatchingResults.predict() method¶

def predict(
    X_test: pd.DataFrame,
    quantiles: Optional[List[float]] = None
) -> Dict[float, pd.DataFrame]

Parameter	Type	Default used	Description
X_test	pd.DataFrame	-	Data to impute (with predictors)
quantiles	List[float]	None	Quantiles for output format (values are identical across quantiles)

It returns a dictionary mapping quantiles to DataFrames of imputed values. Note that matching returns the same values for all quantiles since it transfers actual donor values rather than generating distributional predictions.

# Import needed libraries and setup R environment
import warnings

warnings.filterwarnings("ignore")

import logging

logging.getLogger("rpy2").setLevel(logging.ERROR)

import sys
import os
import pandas as pd
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
from rpy2.robjects import pandas2ri
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

pd.set_option("display.width", 600)
pd.set_option("display.max_columns", 10)
pd.set_option("display.expand_frame_repr", False)

# Import Microimpute tools
from microimpute.evaluations import cross_validate_model
from microimpute.models import Matching
from microimpute.config import QUANTILES, RANDOM_STATE
from microimpute.visualizations import model_performance_results
from microimpute.utils.data import preprocess_data

# Load the diabetes dataset
diabetes = load_diabetes()
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)

# Add random boolean variable
df["bool"] = np.random.choice([True, False], size=len(df))
# Add synthetic weights
df["wgt"] = range(1, len(df) + 1)

# Display the first few rows of the dataset
df

# Define variables for the model
predictors = ["age", "sex", "bmi", "bp"]
imputed_variables = [
    "s1",
    "s4",
    "bool",
]  # We'll impute 's1' (total serum cholesterol), 's4' (total cholesterol/HDL ratio), and the random boolean variable
weights = ["wgt"]

# Create a subset with only needed columns
diabetes_df = df[predictors + imputed_variables + weights]

# Display summary statistics
diabetes_df.describe()

warnings.filterwarnings("ignore")

# Split data into training and testing sets, preprocessing data types all in one (this function also supports normalization)
X_train, X_test = preprocess_data(
    diabetes_df,
    full_data=False,
    test_size=0.2,
    normalize=False,
)

# Let's see how many records we have in each set
print(f"Training set size: {X_train.shape[0]} records")
print(f"Testing set size: {X_test.shape[0]} records")

Training set size: 353 records
Testing set size: 89 records

Simulating missing data¶

For this example, we’ll simulate missing data in our test set by removing the values we want to impute.

# Create a copy of the test set with missing values
X_test_missing = X_test.copy()

# Store the actual values for later comparison
actual_values = X_test_missing[imputed_variables].copy()

# Remove the values to be imputed
X_test_missing[imputed_variables] = np.nan

X_test_missing.head()

Training and using the Matching imputer¶

Now we’ll train the Matching imputer and use it to impute the missing values in our test set.

# Define quantiles we want to model
# We'll use the default quantiles from the config module
print(f"Modeling these quantiles: {QUANTILES}")

Modeling these quantiles: [np.float64(0.05), np.float64(0.1), np.float64(0.15), np.float64(0.2), np.float64(0.25), np.float64(0.3), np.float64(0.35), np.float64(0.4), np.float64(0.45), np.float64(0.5), np.float64(0.55), np.float64(0.6), np.float64(0.65), np.float64(0.7), np.float64(0.75), np.float64(0.8), np.float64(0.85), np.float64(0.9), np.float64(0.95)]

# Initialize the Matching imputer
matching_imputer = Matching()

# Fit the model with our training data
# This trains a linear regression model
fitted_matching_imputer = matching_imputer.fit(
    X_train,
    predictors,
    imputed_variables,
    weight_col="wgt",  # weights will be used for sampling the training data
)

# Impute values in the test set
# This uses the trained Matching model to predict missing values
imputed_values = fitted_matching_imputer.predict(X_test_missing, QUANTILES)

# Display the first few imputed values at the median (0.5 quantile)
imputed_values[0.5].head()

Evaluating the imputation results¶

Now let’s compare the imputed values with the actual values to evaluate the performance of our imputer. Matching does not have the ability to adapt its prediction to specific quantiles, which means no matter which quantile we select, we will obtain the same results

# Define your quantiles
quantiles = list(imputed_values.keys())

# Convert imputed_values dict to a 3D array: (n_samples, n_quantiles)
pred_matrix = np.stack([imputed_values[q].values.flatten() for q in quantiles], axis=1)

# Actual values flattened
actual = actual_values.values.flatten()

# Compute absolute error matrix: shape (n_samples, n_quantiles)
abs_error = np.abs(pred_matrix - actual[:, None])

# Find index of closest prediction for each sample
closest_indices = abs_error.argmin(axis=1)

# Select the closest predictions
closest_predictions = np.array(
    [pred_matrix[i, idx] for i, idx in enumerate(closest_indices)]
)

# Wrap as DataFrame for plotting
closest_df = pd.DataFrame(
    {
        "Actual": actual,
        "ClosestPrediction": closest_predictions,
    }
)

# Extract median predictions for evaluation
median_predictions = imputed_values[0.5]

# Convert data for plotting
plot_df = pd.DataFrame(
    {
        "Actual": actual_values.values.flatten(),
        "Imputed": median_predictions.values.flatten(),
    }
)

# Create the scatter plot
fig = go.Figure()

# Add scatter points
fig.add_trace(
    go.Scatter(
        x=plot_df["Actual"],
        y=plot_df["Imputed"],
        mode="markers",
        name="Data points",
        showlegend=False,
    )
)

# Calculate the full range to ensure square plot
all_values = np.concatenate([plot_df["Actual"], plot_df["Imputed"]])
min_val = all_values.min()
max_val = all_values.max()
padding = (max_val - min_val) * 0.05  # 5% padding

plot_min = min_val - padding
plot_max = max_val + padding

# Add the diagonal line
fig.add_trace(
    go.Scatter(
        x=[plot_min, plot_max],
        y=[plot_min, plot_max],
        mode="lines",
        line=dict(color="red", dash="dash", width=2),
        name="Perfect Prediction",
        showlegend=True,
    )
)

# Update layout with fixed ranges
fig.update_layout(
    title="Comparison of actual vs. imputed values using Matching",
    xaxis_title="Actual values",
    yaxis_title="Imputed values",
    width=750,
    height=700,
    template="plotly_white",
    xaxis=dict(
        range=[plot_min, plot_max],
        scaleanchor="y",  # This ensures the axes have the same scale
        scaleratio=1,
    ),
    yaxis=dict(
        range=[plot_min, plot_max],
    ),
    showlegend=True,
    legend=dict(
        x=0.7,
        y=0.95,
        bgcolor="rgba(255, 255, 255, 0.8)",
        bordercolor="rgba(0, 0, 0, 0.2)",
        borderwidth=1,
    ),
)

fig.show()

This scatter plot presents the performance of a matching-based imputation method by comparing the actual values (x-axis) to the imputed values (y-axis). Each dot represents a data point where a missing value was imputed using the nearest matched donor based on covariates. The red dashed line represents the ideal scenario of perfect prediction, where imputed values would exactly match actual values. Unlike model-based approaches such as Quantile Regression or Random Forests, the matching method shows greater dispersion around the ideal line, with several imputed values either overestimating or underestimating the true values. The scatter reveals that while some imputations are close to accurate, many are not, and the overall alignment with the perfect prediction line is weaker. This suggests that matching may introduce higher imputation error, especially when suitable matches are not available or when the matching algorithm doesn’t capture complex relationships between covariates and the missing variable.

Examining quantile predictions¶

The Matching imputer can also provide predictions at different quantiles, which can be useful for understanding the uncertainty in the imputation.

# Compare predictions at different quantiles for the first 5 records
quantiles_to_show = QUANTILES
comparison_df = pd.DataFrame(index=range(5))

# Add actual values
comparison_df["Actual"] = actual_values.iloc[:5, 0].values

# Add quantile predictions
for q in quantiles_to_show:
    comparison_df[f"Q{int(q * 100)}"] = imputed_values[q].iloc[:5, 0].values

comparison_df

Visualizing prediction intervals¶

By visualizing the prediction intervals of the model’s imputations we can better understand the uncertainty in our imputed values.

# Create a prediction interval plot for the first 10 records
# Number of records to plot
n_records = 10

# Prepare data for plotting
records = list(range(n_records))
actuals = actual_values.iloc[:n_records, 0].values
medians = imputed_values[0.5].iloc[:n_records, 0].values
q30 = imputed_values[0.3].iloc[:n_records, 0].values
q70 = imputed_values[0.7].iloc[:n_records, 0].values
q10 = imputed_values[0.1].iloc[:n_records, 0].values
q90 = imputed_values[0.9].iloc[:n_records, 0].values

# Create the base figure
fig = go.Figure()

# Add 80% prediction interval (Q10-Q90)
for i in range(n_records):
    fig.add_trace(
        go.Scatter(
            x=[i, i],
            y=[q10[i], q90[i]],
            mode="lines",
            line=dict(width=10, color="rgba(173, 216, 230, 0.3)"),
            hoverinfo="none",
            showlegend=False,
        )
    )

# Add 40% prediction interval (Q30-Q70)
for i in range(n_records):
    fig.add_trace(
        go.Scatter(
            x=[i, i],
            y=[q30[i], q70[i]],
            mode="lines",
            line=dict(width=10, color="rgba(70, 130, 180, 0.5)"),
            hoverinfo="none",
            showlegend=False,
        )
    )

# Add actual values
fig.add_trace(
    go.Scatter(
        x=records,
        y=actuals,
        mode="markers",
        marker=dict(color="black", size=8),
        name="Actual",
    )
)

# Add median predictions
fig.add_trace(
    go.Scatter(
        x=records,
        y=medians,
        mode="markers",
        marker=dict(color="red", size=8),
        name="Median (Q50)",
    )
)

# Add dashed line for Q10
fig.add_trace(
    go.Scatter(
        x=[-1, -1],  # Dummy points for legend
        y=[0, 0],  # Dummy points for legend
        mode="lines",
        line=dict(color="rgba(173, 216, 230, 0.3)", width=10),
        name="80% PI (Q10-Q90)",
    )
)

# Add dashed line for Q30
fig.add_trace(
    go.Scatter(
        x=[-1, -1],  # Dummy points for legend
        y=[0, 0],  # Dummy points for legend
        mode="lines",
        line=dict(color="rgba(70, 130, 180, 0.5)", width=10),
        name="40% PI (Q30-Q70)",
    )
)

# Update layout with smaller width to fit in the book layout
fig.update_layout(
    title="Matching imputation prediction intervals",
    xaxis=dict(
        title="Data record index",
        showgrid=True,
        gridwidth=1,
        gridcolor="rgba(211, 211, 211, 0.7)",
    ),
    yaxis=dict(
        title="Total Serum Cholesterol (s1)",
        showgrid=True,
        gridwidth=1,
        gridcolor="rgba(211, 211, 211, 0.7)",
    ),
    width=750,
    height=600,
    template="plotly_white",
    margin=dict(l=50, r=50, t=80, b=50),  # Adjust margins
    legend=dict(yanchor="top", y=0.99, xanchor="right", x=0.99),
)

fig.show()

This plot displays the prediction intervals produced by a Matching model for total serum cholesterol values across ten data records. Each red dot indicates the imputed median value (Q50) for a missing observation, while black dots represent the corresponding true values. Light gray and dark gray vertical bars would represent the 80% (Q10–Q90) and 40% (Q30–Q70) prediction intervals, respectively. Unlike model-based methods, all records lack visible interval bars entirely. This reflects the limited variability inherent in matching methods, where each imputed value is drawn from a single matched donor or a small set of similar units. As a result, the model cannot capture the full uncertainty of the imputed values, as all quantile estimates collapse to the same value. Additionally, many of the imputed medians lie far from the actual values. This highlights a key limitation of matching-based imputation: while simple and interpretable, it may lack the flexibility to accurately quantify uncertainty or represent the underlying distribution, especially in complex or high-variance data.

Assesing the method’s performance¶

To check whether our model is overfitting and ensure robust results we can perform cross-validation and visualize the results.

predictors = ["age", "sex", "bmi", "bp"]
imputed_variables = ["s1", "s4"]

diabetes = load_diabetes()
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)
df["bool"] = np.random.choice([True, False], size=len(df))
df["wgt"] = range(1, len(df) + 1)
diabetes_df = df[predictors + imputed_variables + weights]

diabetes_df = preprocess_data(
    diabetes_df,
    full_data=True,
)

# Run cross-validation on the same data set removing the boolean variable
matching_results = cross_validate_model(
    Matching, diabetes_df, predictors, imputed_variables
)

# Display the results based on variable types
if "quantile_loss" in matching_results:
    print("Quantile loss results for numerical variables:")
    print(matching_results["quantile_loss"]["results"])

if "log_loss" in matching_results:
    print("\nLog loss results for categorical variables:")
    print(f"Mean train: {matching_results['log_loss']['mean_train']:.4f}")
    print(f"Mean test: {matching_results['log_loss']['mean_test']:.4f}")

[Parallel(n_jobs=1)]: Done   1 tasks      | elapsed:    0.1s
[Parallel(n_jobs=1)]: Done   4 tasks      | elapsed:    0.2s

Quantile loss results for numerical variables:
           0.05      0.10      0.15      0.20      0.25  ...     0.75      0.80      0.85      0.90      0.95
train  0.000000  0.000000  0.000000  0.000000  0.000000  ...  0.00000  0.000000  0.000000  0.000000  0.000000
test   0.024143  0.024089  0.024035  0.023982  0.023928  ...  0.02339  0.023336  0.023282  0.023229  0.023175

[2 rows x 19 columns]

Log loss results for categorical variables:
Mean train: nan
Mean test: nan

[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.3s finished

# Plot the results for numerical variables
if "quantile_loss" in matching_results:
    perf_results_viz = model_performance_results(
        results=matching_results,
        model_name="Matching",
        method_name="Cross-validation quantile loss average",
    )
    fig = perf_results_viz.plot(
        title="Matching cross-validation performance",
    )
    fig.show()

We can observe how train loss will be 0 in this case, as each record will find itself in the training data as a perfect match, which is not the case for testing data. Note that when using donor and receiver data sets that are different from each other this will not occur.

Tuning the Matching model¶

The Matching imputer supports various parameters that can be adjusted to improve performance. To set specific values you know increase performance for your specific dataset see below. Additionally, automatic hyperparameter tunning specific to the target dataset is enabled by setting the parameter tune_hyperparameters to True.

predictors = ["age", "sex", "bmi", "bp"]
imputed_variables = ["s1", "s4"]

diabetes = load_diabetes()
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)
df["bool"] = np.random.choice([True, False], size=len(df))
df["wgt"] = range(1, len(df) + 1)
diabetes_df = df[predictors + imputed_variables + weights]

X_train, X_test = preprocess_data(
    diabetes_df,
    full_data=False,
)

# To set specific hyperparameters pass them when fitting the model
fitted_matching_imputer = matching_imputer.fit(
    X_train=X_train,
    predictors=predictors,
    imputed_variables=imputed_variables,
    constrained=True,  # Use constrained matching
)

predictors = ["age", "sex", "bmi", "bp"]
imputed_variables = ["s1", "s4"]

diabetes = load_diabetes()
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)
df["bool"] = np.random.choice([True, False], size=len(df))
df["wgt"] = range(1, len(df) + 1)
diabetes_df = df[predictors + imputed_variables + weights]

X_train, X_test = preprocess_data(
    diabetes_df,
    full_data=False,
)

# To automatically tune hyperparameters to the specific dataset at hand
fitted_matching_imputer, best_tuned_params = matching_imputer.fit(
    X_train=X_train,
    predictors=predictors,
    imputed_variables=imputed_variables,
    tune_hyperparameters=True,
)

print(best_tuned_params)

{'dist_fun': 'Manhattan', 'k': 6}