Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models

Synopsis

24-participant within-subjects study; Copilot does not always speed up task completion but most prefer it as a starting point.

Keywords

·Copilot ·usability ·within-subjects ·task completion

Related entries

• Human-AI Collaboration in Software Development: A Mixed-Methods Study of Developers' Use of GitHub Copilot and ChatGPT
  2025 · FSE Companion 2025
• A Study on Developer Behaviors for Validating and Repairing LLM-Generated Code Using Eye Tracking and IDE Actions
  May 25, 2024 · VL/HCC 2024
• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
  February 13, 2023 · arXiv
• Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
  October 25, 2022 · CHI 2024
• Grounded Copilot: How Programmers Interact with Code-Generating Models
  June 30, 2022 · OOPSLA 2023
• The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers
  2024 · Management Science 2026