Part 2 · Cleaning and Features
Goal: Turn raw prices into stationary return features, split the data without breaking time order (70/15/15), and fit StandardScaler on the training set only, so nothing leaks.
Key ideas:
- Missing values and closed markets: Exchanges are closed at weekends and on public holidays. When filling missing values we only carry the last known past value forward (
ffill). Carrying a future value backwards (bfill) is copying the future. - Why log returns instead of prices? 6 TL in 2015 and 300 TL in 2026 are not on the same scale. The log return $r_t = \ln(P_t / P_{t-1})$ makes the series scale-free and stationary.
- Chronological split and scaler leakage: A time series is never split at random (
shuffle=True). And ifStandardScaler.fit()sees all the data, the mean and standard deviation of 2026 leak into the 2018 training data.
Catch-up cell: 🔄 Catch up: Part 2 (run this if you skipped the previous part)
Step-by-step code walkthrough:
- TODO 2 (
sec2-feat-code): We compute the daily log return (log_return), the 5-day moving average of returns (roll_mean_5) and the 10-day volatility (roll_std_10):
feat_df = raw_df[["Open", "High", "Low", "Close", "Volume"]].ffill().dropna().copy()
# TODO 2 solution:
feat_df["log_return"] = np.log(feat_df["Close"] / feat_df["Close"].shift(1))
feat_df["roll_mean_5"] = feat_df["log_return"].rolling(window=5).mean()
feat_df["roll_std_10"] = feat_df["log_return"].rolling(window=10).std()
feat_df["rsi_14"] = compute_rsi(feat_df["Close"], period=14) / 100.0
feat_df["up_day"] = (feat_df["log_return"] > 0.0).astype(np.float32)
feat_df = feat_df.dropna().copy()- TODO 3 (
sec2-split-code): We split the feature table in time order into 70% train, 15% validation and 15% test:
# TODO 3 solution:
train_df, val_df, test_df = chronological_split(feat_df, train_ratio=0.70, val_ratio=0.15)Train: 2106 days (2015-01-21 -> 2023-04-05), Val : 451 days (2023-04-06 -> 2025-01-24), Test : 452 days (2025-01-27 -> 2026-11-16) (trading days).- TODO 4 (
sec2-scaler-code): We callfit_transformontrain_dfonly, and onlytransformonval_dfandtest_df:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
# TODO 4 solution:
x_train = scaler.fit_transform(train_df[FEATURE_COLS].values.astype(np.float32))
x_val = scaler.transform(val_df[FEATURE_COLS].values.astype(np.float32))
x_test = scaler.transform(test_df[FEATURE_COLS].values.astype(np.float32))Troubleshooting (if this step fails):
KeyError: 'log_return': You probably jumped to the split cell before runningsec2-feat-code. Run the cells in order, or use the🔄 Catch up: Part 2cell at the top of the part.ValueError: Found array with 0 sample(s): A column name in the rolling-window code is probably misspelled beforedropna(). Check the three lines above.