A Statistical-Finance Benchmark for Same-Day Directional Stock Prediction: Walk-Forward Evidence from SPY

Abstract

We study statistical predictability in daily U.S. equity prices using only information available at the market open. Using SPY from February 1, 1993 through March 15, 2024, we benchmark XGBoost against Random Forest, LightGBM, Logistic Regression, and naive baselines under expanding-window walk-forward validation. After observing the current day's opening price and two lagged target-specific prices, the task is to predict whether the same day's close will be above or below the previous day's close. On the last 800 trading days, Logistic Regression attains the highest close-direction accuracy (71.09%), Random Forest reaches 61.20%, and XGBoost reaches 58.45% with a 95% bootstrap confidence interval of [54.94%, 62.08]. For XGBoost, close-direction accuracy rises to 72.7% when the predicted move exceeds 1%, but the usable sample falls to 154 observations. We also report Diebold-Mariano and McNemar tests, regime-specific results, SHAP feature importance, and an auxiliary 541-equity screen. The evidence supports a narrow statistical-finance conclusion: simple daily equity features contain detectable same-day directional information, but the result should be interpreted with careful sample-size accounting and limited economic claims.

7 figures, 7 tables, includes appendices