ML Experiment Logging Notes

实验日志在使比较成本低廉时最有用。目标不是记录一切。目标是保留解释结果的决策。

Minimum Useful Fields

每次运行应该记录：

• Hypothesis：你在测试什么假设？
• Dataset Version：数据集的版本号或哈希值
• Configuration：超参数和设置
• Metrics：目标指标和次要指标
• Interpretation：简短的解释

解释很重要，因为指标本身很少能解释什么发生了变化。

Organize by Question

不要只按日期组织实验，而是按它们回答的问题分组：

experiments/
├── does-feature-x-help/
│   ├── run-001-baseline/
│   ├── run-002-with-feature/
│   └── run-003-ablation/
├── learning-rate-sensitivity/
│   ├── run-001-lr-1e-3/
│   ├── run-002-lr-1e-4/
│   └── run-003-lr-1e-5/
└── architecture-comparison/
    ├── run-001-transformer/
    └── run-002-rnn/

What to Log

Essential

• Git commit hash
• Random seeds
• Hardware info (GPU type)
• Training time
• Final metrics

Useful

• Loss curves
• Learning rate schedule
• Model architecture
• Data preprocessing

Optional

• Intermediate checkpoints
• Attention visualizations
• Gradient statistics

Tools

Weights & Biases

import wandb

wandb.init(project="my-project", config={
    "learning_rate": 1e-4,
    "batch_size": 32,
    "architecture": "transformer"
})

for epoch in range(num_epochs):
    loss = train_epoch()
    wandb.log({"loss": loss, "epoch": epoch})

MLflow

import mlflow

mlflow.start_run()
mlflow.log_params({"lr": 1e-4, "batch_size": 32})
mlflow.log_metrics({"loss": 0.5, "accuracy": 0.85})
mlflow.end_run()

Simple File-Based

import json
from pathlib import Path

def log_experiment(name, config, metrics):
    run_dir = Path("experiments") / name
    run_dir.mkdir(parents=True, exist_ok=True)
    
    with open(run_dir / "config.json", "w") as f:
        json.dump(config, f, indent=2)
    
    with open(run_dir / "metrics.json", "w") as f:
        json.dump(metrics, f, indent=2)

Comparing Runs

创建一个简单的比较脚本：

import pandas as pd
from pathlib import Path

def compare_runs(experiment_dir):
    runs = []
    for run_dir in Path(experiment_dir).iterdir():
        if run_dir.is_dir():
            config = json.load(open(run_dir / "config.json"))
            metrics = json.load(open(run_dir / "metrics.json"))
            runs.append({**config, **metrics, "run": run_dir.name})
    
    return pd.DataFrame(runs)

Key Principles

1. Reproducibility：记录足够信息来重现结果
2. Comparability：使不同运行之间的比较容易
3. Interpretability：记录为什么做出某些决策
4. Minimalism：只记录真正重要的信息

Summary

好的实验日志系统应该使记录成本低，比较成本低。专注于记录决策和结果，而不是一切。

---

This approach works well with tools like Weights & Biases or MLflow.