Experiment Framework: A Guide to Structuring Your AI Experiments

Introduction 

Running AI experiments effectively requires a structured approach to ensure reproducibility, proper logging, and efficient result analysis. This tutorial provides a well-organized Python framework to conduct experiments smoothly by handling configuration, data loading, training, and exporting results. -

• Why Use a Structured Experiment Framework? 

Without a structured framework, AI experiments can become disorganized, leading to difficulties in tracking parameters, reproducing results, and analyzing performance. A well-structured approach ensures: 

• Consistency: Using fixed configurations and seeds for reproducibility.
• Automation: Automating training, evaluation, and result exporting.
• Scalability: Easy adaptation for different datasets and models.
• Efficiency: Reducing redundant code and saving results systematically. 

Implementing the Experiment Framework

1. Configuration and Setup 

We start by defining a configuration class to manage experimental settings like experiment name, random seed, and output directory.

 

import os
import json
import pandas as pd
import numpy as np
import random
import torch
from datetime import datetime

# Configuration Class
class ExperimentConfig:
    def __init__(self, name="default_experiment", seed=42, output_dir="results"):
        self.name = name
        self.seed = seed
        self.output_dir = output_dir
        os.makedirs(self.output_dir, exist_ok=True)
        self.timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
        self.experiment_path = os.path.join(self.output_dir, f"{self.name}_{self.timestamp}")
        os.makedirs(self.experiment_path, exist_ok=True)
        self.save_config()
        
    def save_config(self):
        """Save experiment configuration to a JSON file."""
        config_path = os.path.join(self.experiment_path, "config.json")
        with open(config_path, "w") as f:
            json.dump(self.__dict__, f, indent=4)

 

 This class ensures that each experiment is saved in a unique directory with a timestamp, making it easy to track and compare multiple runs. 

2. Ensuring Reproducibility 

Setting a fixed random seed ensures that results remain consistent across runs.

 

# Set seed for reproducibility
def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(seed)

 

 3. Loading Data For demonstration, 

we generate a synthetic dataset. In real cases, you would load actual data from a file or database. 

 

# Data Loading (Placeholder Function)
def load_data():
    """Simulates loading a dataset."""
    return pd.DataFrame(np.random.randn(100, 5), columns=[f"feature_{i}" for i in range(5)])

 

4. Model Training 

This is a placeholder function to simulate training. You can replace it with your actual model training logic. 

 

# Model Training (Placeholder Function)
def train_model(data):
    """Simulates model training and returns accuracy and loss."""
    return {"accuracy": round(random.uniform(0.8, 0.95), 4), "loss": round(random.uniform(0.1, 0.5), 4)}

 

5. Saving Results After training, 

we save the results in both JSON and CSV formats for easy analysis.

# Save Results
def save_results(results, experiment_path):
    """Save experiment results to a JSON and CSV file."""
    results_path = os.path.join(experiment_path, "results.json")
    with open(results_path, "w") as f:
        json.dump(results, f, indent=4)
    
    df = pd.DataFrame([results])
    df.to_csv(os.path.join(experiment_path, "results.csv"), index=False)

 

6. Running the Experiment 

The main function ties everything together: setting up the configuration, loading data, training the model, and saving results. 

 

# Main Experiment Workflow
def run_experiment():
    """Runs the full experiment workflow."""
    config = ExperimentConfig(name="my_experiment", seed=123)
    set_seed(config.seed)
    data = load_data()
    results = train_model(data)
    save_results(results, config.experiment_path)
    print("Experiment completed. Results saved in:", config.experiment_path)

if __name__ == "__main__":
    run_experiment()

Conclusion

 This tutorial provides a structured way to conduct AI experiments efficiently. By following this approach, you ensure your experiments are: 

✅ Well-documented 

✅ Easily reproducible 

✅ Systematically stored 

 You can extend this framework by integrating actual models, hyperparameter tuning, and logging systems. Happy experimenting!