Replication Package: Human vs. AI Agents in Code Review

This repository provides the replication package for the paper: "Human vs. AI Agents in Code Review: An Empirical Comparison of Review Feedback, Interaction Dynamics, and Suggestion Adoption"

This study empirically compares code review practices across four categories:

• HRH: Human Reviews Human-written code
• HRA: Human Reviews Agent-generated code
• ARH: Agent Reviews Human-written code
• ARA: Agent Reviews Agent-generated code

The analysis spans three research questions:

• RQ1: What are the similarities and differences between the review comments by AI agents and human reviewers?
• RQ2: How do interaction patterns differ between human and AI agent code reviews?
• RQ3: What is the impact of code suggestions from human reviewers and AI agents on code quality?

AI_Vs_Human_Codereview/
├── README.md                          # This file
├── appendix/
│   └── Appendix_RQ_Results.pdf        # Fine-grained taxonomy & cross-agent analysis
├── run_all.py                         # Run all scripts in one command
├── requirements.txt                   # Python dependencies
├── requirements-lock.txt              # Pinned Python dependencies
├── install_packages.R                 # R dependencies
├── data/
│   ├── results/                       # Pre-computed aggregated results
│   │   ├── RQ1/                       #   (reproduces exact paper numbers)
│   │   ├── RQ2/
│   │   └── RQ3/
│   └── sample/                        # Raw data sample (100 rows per file)
│       └── README.md                  #   Data file descriptions
├── scripts/
│   ├── config.py                      # Centralized path configuration
│   ├── setup/                         # Experiment setup artifacts
│   │   ├── 1_feedback_taxonomy.py     # Table II: Feedback type taxonomy
│   │   ├── 2_labeling_prompt.py       # Figure 3: LLM labeling prompt
│   │   └── 3_sampling_stats.py        # Table IX: Sampling statistics
│   ├── RQ1/                           # RQ1: Feedback analysis
│   │   ├── 0_dataset_table.py         # Table I: Dataset overview
│   │   ├── 1_feedback_distribution.py # Figure 4: Feedback type chart
│   │   ├── 2_cd_violin.py             # Figure 5: Comment density violin
│   │   ├── 3_sk_fdbtype_turns.py      # Table III: Feedback by rounds
│   │   └── 3_scott_knott_cd.R         # Figure 5: Scott-Knott CD ranking
│   ├── RQ2/                           # RQ2: Interaction dynamics
│   │   ├── 1_avg_duration.py          # Table IV: Avg comment count
│   │   ├── 2_fsm_visualization.py     # Figure 6: FSM diagrams
│   │   └── 3_scott_knott_feedback.R   # Table III: Scott-Knott feedback
│   └── RQ3/                           # RQ3: Suggestion adoption
│       ├── 1_adoption_rates.py        # Table V: Adoption rates
│       ├── 2_code_metrics_analysis.py # Table VI: Code quality impact
│       ├── 3_unadopted_analysis.py    # Figure 8: Unadopted reasons
│       ├── 6_code_metrics_R.R         # Statistical testing (R)
│       └── 7_scott_knott_metrics.R    # Scott-Knott metric ranking
├── prompts/
│   └── feedback_classification.md     # LLM prompts used for labeling
└── results/                           # Output directory
    ├── tables/                        # Generated CSV tables
    ├── figures/                       # Generated figures (JPEG)
    ├── verify_feedbacktype_label.csv      # Manual validation: feedback-type labels (Cohen's kappa)
    └── verify_qualityattribute_label.csv  # Manual validation: quality-attribute labels

This package supports two levels of reproducibility:

All pre-computed aggregated results are included in data/results/. Python scripts read these files and produce the exact tables and figures from the paper. No full dataset needed.

Download the full raw dataset from HuggingFace, then R scripts run the statistical analysis (Scott-Knott ESD, Wilcoxon, Cliff's delta) from scratch.

Figures 1, 2, 7, 9 are static illustrations/examples (not computed from data).

• Python 3.9+
• R 4.0+ (optional, for full reproducibility only)

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate        # Linux/macOS
# .venv\Scripts\activate         # Windows

# Install Python dependencies (exact versions for reproducibility)
pip install -r requirements-lock.txt
# Or use flexible versions:
# pip install -r requirements.txt

# R dependencies (optional)
Rscript install_packages.R

Full raw dataset is available on Hugging Face: Suzhen/AgentReviewChat

python run_all.py              # Run all scripts
python run_all.py --skip-r     # Skip R scripts (Python only)
python run_all.py --rq1        # Run only RQ1 scripts

python scripts/setup/1_feedback_taxonomy.py    # Table II
python scripts/setup/2_labeling_prompt.py      # Figure 3
python scripts/setup/3_sampling_stats.py       # Table IX

python scripts/RQ1/0_dataset_table.py          # Table I
python scripts/RQ1/1_feedback_distribution.py  # Figure 4
python scripts/RQ1/2_cd_violin.py              # Figure 5
python scripts/RQ1/3_sk_fdbtype_turns.py       # Table III
Rscript scripts/RQ1/3_scott_knott_cd.R         # Figure 5 (from raw data)

python scripts/RQ2/1_avg_duration.py           # Table IV
python scripts/RQ2/2_fsm_visualization.py      # Figure 6
Rscript scripts/RQ2/3_scott_knott_feedback.R   # Table III (from raw data)

python scripts/RQ3/1_adoption_rates.py         # Table V
python scripts/RQ3/2_code_metrics_analysis.py  # Table VI
python scripts/RQ3/3_unadopted_analysis.py     # Figure 8
Rscript scripts/RQ3/6_code_metrics_R.R         # Statistical testing (from raw data)
Rscript scripts/RQ3/7_scott_knott_metrics.R    # Scott-Knott ranking (from raw data)

See appendix/Appendix_RQ_Results.pdf for material that complements the main results:

• Fine-grained technical taxonomy (Table A1) — Turzo and Bosu's sub-categories for Defect Detection and Code Improvement comments, extending the RQ1 comparison.
• Cross-agent analysis (Table A2–A4, Figure A1–A2) — per-agent breakdown across all three RQs for the seven AI reviewers (GitHub Copilot, CodeRabbit, Gemini Code Assist, Cursor, Claude Code, Sourcery AI, Greptile):
  • RQ1: feedback-type distribution, reading-ease ranking, and quality-attribute distribution per agent vs. the aggregate AI-agent baseline.
  • RQ2: interaction patterns (average comment count, multi-comment rate, usefulness) per agent.
  • RQ3: code-metric changes associated with each agent's adopted suggestions vs. the aggregate baseline.

results/ also includes the first-author manual labels used to validate the LLM classifications reported in the paper (Cohen's $\kappa$ agreement):

• results/verify_feedbacktype_label.csv
• results/verify_qualityattribute_label.csv

• Python scripts use pre-computed results from data/results/ and reproduce exact paper outputs.
• R scripts run statistical analysis from raw data in data/sample/ or data/full/.
• Sample data produces correct output structure but values differ from the paper (only 100 rows).
• Full dataset from HuggingFace reproduces exact paper results.
• R scripts require the ScottKnottESD package for non-parametric effect size ranking.
• Code metrics extraction uses SciTools Understand (not included; raw metrics are provided).
• LLM-based labeling uses GPT-4.1-mini; prompts are documented in prompts/.