Advancing Binary Classification Through Fuzzy Logic: A Novel Framework for Handling Imbalance and Separation

Logistic regression stands as one of the most widely used statistical methods for binary classification across fields ranging from healthcare diagnostics to financial risk assessment. However, two fundamental challenges continue to compromise its performance: imbalance in response variables and separation in predictors. According to our most recent research, these issues can lead to biased coefficient estimates and unreliable predictions that undermine decision-making in critical applications.

In a groundbreaking study published in Information Sciences, researchers propose an innovative fuzzy logistic regression framework that addresses both challenges simultaneously while maintaining the interpretability that makes logistic regression valuable. Their work demonstrates how incorporating uncertainty through fuzzy logic can produce more robust and accurate classifications than traditional approaches.

The Twin Challenges: Imbalance and Separation

• Class imbalance occurs when one outcome vastly outnumbers the other in binary data. Imagine a disease screening test where only 5% of samples show positive results. Traditional classifiers often struggle with this asymmetry, learning predominantly from the majority class and performing poorly on the minority class that may actually be of greater interest. This problem pervades applications from cyber-attack detection to fraud identification and disease diagnosis.

• Separation presents a different challenge: when predictor variables can perfectly or near-perfectly distinguish between outcomes, especially in smaller samples, maximum likelihood estimation becomes unstable or impossible. While this might seem like an ideal scenario, it typically indicates overfitting that will fail when applied to new data.

Existing solutions, including preprocessing techniques, ensemble methods, and algorithmic modifications, each carry disadvantages. Ensemble approaches, while promising, increase computational complexity and sacrifice interpretability. Meanwhile, most fuzzy logistic regression research has focused on "possibilistic" approaches that transform binary outcomes into continuous possibilities, abandoning the probabilistic foundation that enables odds ratio interpretation.

A Probabilistic Fuzzy Framework

The researchers' approach differs fundamentally by maintaining a truly probabilistic framework while leveraging fuzzy logic's ability to handle uncertainty. Their innovation centers on three key components:

1. Fuzzifying Binary Variables

Rather than converting binary responses into arbitrary possibility scores, the framework transforms them into triangular fuzzy numbers (TFNs) that preserve the binary nature at their core. A response of 0 or 1 becomes the "vertex" of a fuzzy number, with left and right boundaries capturing uncertainty. This maintains the essential binary character while allowing for gradations of confidence, similar to acknowledging that while a patient either has or doesn't have a condition, diagnostic uncertainty exists around that determination.

2. Monte Carlo Optimization

The framework employs Monte Carlo simulation to optimize model coefficients, generating thousands of random coefficient vectors and selecting those that minimize prediction error. This approach proved remarkably effective, with mean absolute error (MAE) demonstrating 56% lower coefficient of variation compared to mean squared error across different scenarios. The researchers found that 10,000 Monte Carlo replications provided optimal balance between accuracy and computational efficiency.

3. Fuzzy Classification Thresholds

Instead of using crisp probability cutoffs (like the traditional 0.5 threshold), the method classifies observations using fuzzy thresholds that themselves are triangular fuzzy numbers. This reduces sensitivity to threshold selection, a known vulnerability in traditional logistic regression, while maintaining classification accuracy.

Three Model Structures for Different Scenarios

The research examined three distinct model configurations:

• Case II: Crisp predictors with fuzzy coefficients and fuzzy outputs

• Case III: Fuzzy predictors with crisp coefficients and fuzzy outputs

• Case IV: Fully fuzzy approach with fuzzy predictors, coefficients, and outputs

Demonstrating Superior Performance

The framework's effectiveness was validated through rigorous testing on synthetic datasets with controlled imbalance (ranging from 50-50 balance to 85% imbalance) and separation conditions, as well as five diverse real-world datasets including immunotherapy treatment response, cervical cancer screening, autism diagnosis, and wildfire prediction.

When benchmarked against seven machine learning methods (including Support Vector Machines, Neural Networks, XGBoost, and K-Nearest Neighbors), the fuzzy framework showed remarkable robustness. While traditional methods frequently exhibited characteristic signs of imbalance sensitivity, such as zero specificity and perfect sensitivity scores, the fuzzy approaches maintained balanced performance across all metrics.

For the immunotherapy dataset with 79% imbalance, six of seven traditional machine learning methods showed complete sensitivity/specificity imbalance. The Case II fuzzy model achieved 77.5% specificity and 86.2% sensitivity with strong F1 and MCC scores, while maintaining interpretability through odds ratios.

In datasets with separation issues, machine learning methods produced suspiciously perfect scores (all measures exceeding 0.9) in 35-50% of cases, indicating overfitting. The fuzzy framework showed such "perfect" results less than 3% of the time, demonstrating genuine robustness rather than memorization.

Practical Implementation and Interpretability

A crucial advantage of this framework is that it retains logistic regression's interpretability. After optimization, fuzzy coefficients are defuzzified using the center-of-gravity method, allowing practitioners to calculate and interpret odds ratios just as they would with traditional logistic regression. This means analysts can both achieve superior classification performance and gain insights into which predictors matter most and by how much.

The computational requirements proved reasonable for practical application. Processing datasets ranging from 72 to 243 observations with 5-19 predictors required under one minute for 1,000 Monte Carlo replications on standard hardware. For larger datasets, the number of replications can be reduced with minimal accuracy loss, decreasing from 1,000 to 500 replications yielded 44% efficiency gain with only 1% accuracy loss.

The complete research, including technical details and R code, is available in: Charizanos, G., Demirhan, H., & İçen, D. (2024). A Monte Carlo fuzzy logistic regression framework against imbalance and separation. Information Sciences, 655, 119893.