Human-In-The-Loop Software Development Agents: Challenges and Future Directions

Jirat Pasuksmit; Wannita Takerngsaksiri; Patanamon Thongtanunam; Chakkrit Tantithamthavorn; Ruixiong Zhang; Shiyan Wang; Fan Jiang; Jing Li; Evan Cook; Kun Chen; Ming Wu

doi:10.1109/MSR66628.2025.00112

Human-In-The-Loop Software Development Agents: Challenges and Future Directions

Jirat Pasuksmit
, Wannita Takerngsaksiri
, Patanamon Thongtanunam
, Chakkrit Tantithamthavorn
, Ruixiong Zhang
, Shiyan Wang
, Fan Jiang
, Jing Li
, Evan Cook
, Kun Chen
, Ming Wu

Department of Software Systems & Cybersecurity

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Other

3 Citations (Scopus)

Abstract

Multi-agent LLM-driven systems for software development are rapidly gaining traction, offering new opportunities to enhance productivity. At Atlassian, we deployed Human-in-the-Loop Software Development Agents to resolve Jira work items and evaluated the generated code quality using functional correctness testing and GPT-based similarity scoring. This paper highlights two major challenges: the high computational costs of unit testing and the variability in LLM-based evaluations. We also propose future research directions to improve evaluation frameworks for Human-In-The-Loop software development tools.

Original language	English
Title of host publication	Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025
Editors	Dong Wang, Jinqiu Yang
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	756-757
Number of pages	2
ISBN (Electronic)	9798331501839
ISBN (Print)	9798331501846
DOIs	https://doi.org/10.1109/MSR66628.2025.00112
Publication status	Published - 2025
Event	IEEE International Working Conference on Mining Software Repositories 2025 - Ottawa, Canada Duration: 28 Apr 2025 → 29 Apr 2025 Conference number: 22nd https://ieeexplore.ieee.org/xpl/conhome/11025548/proceeding (Proceedings) https://2025.msrconf.org/ (Website)

Publication series

Name	Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025
Publisher	IEEE, Institute of Electrical and Electronics Engineers
ISSN (Print)	2574-3848
ISSN (Electronic)	2574-3864

Conference

Conference	IEEE International Working Conference on Mining Software Repositories 2025
Abbreviated title	MSR 2025
Country/Territory	Canada
City	Ottawa
Period	28/04/25 → 29/04/25
Internet address	https://ieeexplore.ieee.org/xpl/conhome/11025548/proceeding (Proceedings) https://2025.msrconf.org/ (Website)

Keywords

atlassian
code-generation
hula
human-in-the-loop
jira
llm

Access to Document

10.1109/MSR66628.2025.00112

Cite this

Pasuksmit, J., Takerngsaksiri, W., Thongtanunam, P., Tantithamthavorn, C., Zhang, R., Wang, S., Jiang, F., Li, J., Cook, E., Chen, K., & Wu, M. (2025). Human-In-The-Loop Software Development Agents: Challenges and Future Directions. In D. Wang, & J. Yang (Eds.), Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025 (pp. 756-757). (Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/MSR66628.2025.00112

Pasuksmit, Jirat ; Takerngsaksiri, Wannita ; Thongtanunam, Patanamon et al. / Human-In-The-Loop Software Development Agents : Challenges and Future Directions. Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025. editor / Dong Wang ; Jinqiu Yang. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2025. pp. 756-757 (Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025).

@inproceedings{d560d216c9cf4d708d18ea851120cf6f,

title = "Human-In-The-Loop Software Development Agents: Challenges and Future Directions",

abstract = "Multi-agent LLM-driven systems for software development are rapidly gaining traction, offering new opportunities to enhance productivity. At Atlassian, we deployed Human-in-the-Loop Software Development Agents to resolve Jira work items and evaluated the generated code quality using functional correctness testing and GPT-based similarity scoring. This paper highlights two major challenges: the high computational costs of unit testing and the variability in LLM-based evaluations. We also propose future research directions to improve evaluation frameworks for Human-In-The-Loop software development tools.",

keywords = "atlassian, code-generation, hula, human-in-the-loop, jira, llm",

author = "Jirat Pasuksmit and Wannita Takerngsaksiri and Patanamon Thongtanunam and Chakkrit Tantithamthavorn and Ruixiong Zhang and Shiyan Wang and Fan Jiang and Jing Li and Evan Cook and Kun Chen and Ming Wu",

note = "Publisher Copyright: {\textcopyright} 2025 IEEE.; IEEE International Working Conference on Mining Software Repositories 2025, MSR 2025 ; Conference date: 28-04-2025 Through 29-04-2025",

year = "2025",

doi = "10.1109/MSR66628.2025.00112",

language = "English",

isbn = "9798331501846",

series = "Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "756--757",

editor = "Dong Wang and Jinqiu Yang",

booktitle = "Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025",

address = "United States of America",

url = "https://ieeexplore.ieee.org/xpl/conhome/11025548/proceeding, https://2025.msrconf.org/",

}

Pasuksmit, J, Takerngsaksiri, W, Thongtanunam, P, Tantithamthavorn, C, Zhang, R, Wang, S, Jiang, F, Li, J, Cook, E, Chen, K & Wu, M 2025, Human-In-The-Loop Software Development Agents: Challenges and Future Directions. in D Wang & J Yang (eds), Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025. Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 756-757, IEEE International Working Conference on Mining Software Repositories 2025, Ottawa, Ontario, Canada, 28/04/25. https://doi.org/10.1109/MSR66628.2025.00112

Human-In-The-Loop Software Development Agents: Challenges and Future Directions. / Pasuksmit, Jirat; Takerngsaksiri, Wannita; Thongtanunam, Patanamon et al.
Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025. ed. / Dong Wang; Jinqiu Yang. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2025. p. 756-757 (Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Other

TY - GEN

T1 - Human-In-The-Loop Software Development Agents

T2 - IEEE International Working Conference on Mining Software Repositories 2025

AU - Pasuksmit, Jirat

AU - Takerngsaksiri, Wannita

AU - Thongtanunam, Patanamon

AU - Tantithamthavorn, Chakkrit

AU - Zhang, Ruixiong

AU - Wang, Shiyan

AU - Jiang, Fan

AU - Li, Jing

AU - Cook, Evan

AU - Chen, Kun

AU - Wu, Ming

N1 - Conference code: 22nd

PY - 2025

Y1 - 2025

N2 - Multi-agent LLM-driven systems for software development are rapidly gaining traction, offering new opportunities to enhance productivity. At Atlassian, we deployed Human-in-the-Loop Software Development Agents to resolve Jira work items and evaluated the generated code quality using functional correctness testing and GPT-based similarity scoring. This paper highlights two major challenges: the high computational costs of unit testing and the variability in LLM-based evaluations. We also propose future research directions to improve evaluation frameworks for Human-In-The-Loop software development tools.

AB - Multi-agent LLM-driven systems for software development are rapidly gaining traction, offering new opportunities to enhance productivity. At Atlassian, we deployed Human-in-the-Loop Software Development Agents to resolve Jira work items and evaluated the generated code quality using functional correctness testing and GPT-based similarity scoring. This paper highlights two major challenges: the high computational costs of unit testing and the variability in LLM-based evaluations. We also propose future research directions to improve evaluation frameworks for Human-In-The-Loop software development tools.

KW - atlassian

KW - code-generation

KW - hula

KW - human-in-the-loop

KW - jira

KW - llm

UR - https://www.scopus.com/pages/publications/105009162123

U2 - 10.1109/MSR66628.2025.00112

DO - 10.1109/MSR66628.2025.00112

M3 - Conference Paper

AN - SCOPUS:105009162123

SN - 9798331501846

T3 - Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025

SP - 756

EP - 757

BT - Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025

A2 - Wang, Dong

A2 - Yang, Jinqiu

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

Y2 - 28 April 2025 through 29 April 2025

ER -

Pasuksmit J, Takerngsaksiri W, Thongtanunam P, Tantithamthavorn C, Zhang R, Wang S et al. Human-In-The-Loop Software Development Agents: Challenges and Future Directions. In Wang D, Yang J, editors, Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2025. p. 756-757. (Proceedings - 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories, MSR 2025). doi: 10.1109/MSR66628.2025.00112

Human-In-The-Loop Software Development Agents: Challenges and Future Directions

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Cite this