Abstract—Intrusion Detection Systems (IDS) must operate under dynamic network conditions where attack distributions change over time and previously unseen threats may emerge after deployment. However, many machine-learning-based IDS studies still rely on random train-test splitting, which can lead to overly optimistic performance estimates. To address this limitation, this paper presents a leakage-aware evaluation framework for intrusion detection using the Communications Security Establishment–Canadian Institute for Cybersecurity Intrusion Detection System 2018 (CSE-CIC-IDS2018) dataset under a time-based train-test protocol. The proposed framework includes dataset curation, temporal partitioning, leakage-free feature cleaning, baseline and class-weighted Extreme Gradient Boosting (XGBoost) classification, confidence-threshold sensitivity analysis, and duplicate-filtered robustness checking. The results show that the dataset represents a severely imbalanced multi-class detection problem. Under the time-based split, the baseline model achieves high overall accuracy but fails on unseen attack classes and remains unstable on several rare-known attacks. Class-weighted training improves detection for rare-known classes represented during training, but does not improve performance on unseen attacks. In addition, confidence-based rejection fails to identify unseen attacks because the classifier remains highly confident even when making incorrect predictions on previously unseen traffic. Threshold-sensitivity analysis further shows that stricter rejection thresholds do not provide a useful trade-off, while duplicate-filtered evaluation confirms that the unseen-attack failure is not explained by exact train-test feature overlap. Overall, the findings show that reliable intrusion detection requires more than strong benchmark accuracy. It requires leakage-aware evaluation, class-sensitive analysis, robustness checks against evaluation artifacts, and explicit mechanisms for handling unseen attacks under realistic temporal shift.

Index Terms—Intrusion Detection Systems (IDS), leakage-aware evaluation, Extreme Gradient Boosting (XGBoost), confidence-based rejection, explainable artificial intelligence

Cite: Ameen Shaheen, Wael Alzyadat, and Aysh Alhroob, "Leakage-Aware Evaluation of Intrusion Detection Systems for Rare and Unseen Attacks under Temporal Distribution Shift," International Journal of Electrical and Electronic Engineering & Telecommunications, vol. 15, no. 5, pp. 367-377, 2026. doi: 10.18178/ijeetc.15.5.367-377

Copyright © 2026 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).