omarcevi.dev / workshops / pytorch-finance-workshop / 2 · Cleaning and Features

Part 2 · Cleaning and Features

Goal: Turn raw prices into stationary return features, split the data without breaking time order (70/15/15), and fit StandardScaler on the training set only, so nothing leaks.

Key ideas:

  • Missing values and closed markets: Exchanges are closed at weekends and on public holidays. When filling missing values we only carry the last known past value forward (ffill). Carrying a future value backwards (bfill) is copying the future.
  • Why log returns instead of prices? 6 TL in 2015 and 300 TL in 2026 are not on the same scale. The log return $r_t = \ln(P_t / P_{t-1})$ makes the series scale-free and stationary.
  • Chronological split and scaler leakage: A time series is never split at random (shuffle=True). And if StandardScaler.fit() sees all the data, the mean and standard deviation of 2026 leak into the 2018 training data.

Catch-up cell: 🔄 Catch up: Part 2 (run this if you skipped the previous part)

Step-by-step code walkthrough:

  1. TODO 2 (sec2-feat-code): We compute the daily log return (log_return), the 5-day moving average of returns (roll_mean_5) and the 10-day volatility (roll_std_10):
notebook_cell.py
feat_df = raw_df[["Open", "High", "Low", "Close", "Volume"]].ffill().dropna().copy()

# TODO 2 solution:
feat_df["log_return"] = np.log(feat_df["Close"] / feat_df["Close"].shift(1))
feat_df["roll_mean_5"] = feat_df["log_return"].rolling(window=5).mean()
feat_df["roll_std_10"] = feat_df["log_return"].rolling(window=10).std()

feat_df["rsi_14"] = compute_rsi(feat_df["Close"], period=14) / 100.0
feat_df["up_day"] = (feat_df["log_return"] > 0.0).astype(np.float32)
feat_df = feat_df.dropna().copy()
  1. TODO 3 (sec2-split-code): We split the feature table in time order into 70% train, 15% validation and 15% test:
notebook_cell.py
# TODO 3 solution:
train_df, val_df, test_df = chronological_split(feat_df, train_ratio=0.70, val_ratio=0.15)
THYAO.IS chronological train, validation and test regions
Expected output: Train: 2106 days (2015-01-21 -> 2023-04-05), Val : 451 days (2023-04-06 -> 2025-01-24), Test : 452 days (2025-01-27 -> 2026-11-16) (trading days).
  1. TODO 4 (sec2-scaler-code): We call fit_transform on train_df only, and only transform on val_df and test_df:
notebook_cell.py
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

# TODO 4 solution:
x_train = scaler.fit_transform(train_df[FEATURE_COLS].values.astype(np.float32))
x_val = scaler.transform(val_df[FEATURE_COLS].values.astype(np.float32))
x_test = scaler.transform(test_df[FEATURE_COLS].values.astype(np.float32))

Troubleshooting (if this step fails):

  • KeyError: 'log_return': You probably jumped to the split cell before running sec2-feat-code. Run the cells in order, or use the 🔄 Catch up: Part 2 cell at the top of the part.
  • ValueError: Found array with 0 sample(s): A column name in the rolling-window code is probably misspelled before dropna(). Check the three lines above.