Graphical Abstract Figure

Elicitron’s architecture for requirements elicitation using LLMs: First, LLM agents are generated within a design context in either serial and parallel fashion (incorporating diversity sampling to represent varied user perspectives). These agents then engage in simulated product experience scenarios, documenting each step (Action, Observation, Challenge) in detail. Following this, they undergo an agent interview process, where questions are asked and answered to surface latent user needs. In the final stage, latent needs are identified using an LLM on a provided criteria, and finally a report is generated from the identified latent needs.

Graphical Abstract Figure

Elicitron’s architecture for requirements elicitation using LLMs: First, LLM agents are generated within a design context in either serial and parallel fashion (incorporating diversity sampling to represent varied user perspectives). These agents then engage in simulated product experience scenarios, documenting each step (Action, Observation, Challenge) in detail. Following this, they undergo an agent interview process, where questions are asked and answered to surface latent user needs. In the final stage, latent needs are identified using an LLM on a provided criteria, and finally a report is generated from the identified latent needs.

Close modal

Abstract

Requirement elicitation, a critical, yet time-consuming and challenging step in product development, often fails to capture the full spectrum of user needs. This may lead to products that fall short of user expectations. This article introduces a novel framework that leverages large language models (LLMs) to automate and enhance the requirement elicitation process. LLMs are used to generate a vast array of simulated users (LLM agents), enabling the exploration of a much broader range of user needs and unforeseen use cases. These agents engage in product experience scenarios, explaining their actions, observations, and challenges. Subsequent agent interviews and analysis uncover valuable user needs, including latent ones. We validate our framework with three experiments. First, we explore different methodologies for the challenge of diverse agent generation, discussing their advantages and shortcomings. We measure the diversity of identified user needs and demonstrate that context-aware agent generation leads to greater diversity. Second, we show how our framework effectively mimics empathic lead user interviews, identifying a greater number of latent needs than conventional human interviews. Third, we show that LLMs can be used to analyze interviews, capture needs, and classify them as latent or not. Our work highlights the potential of using LLMs to accelerate early-stage product development with minimal costs and increase innovation.

References

1.
Zave
,
P.
,
1997
, “
Classification of Research Efforts in Requirements Engineering
,”
ACM Comput. Surv. (CSUR)
,
29
(
4
), pp.
315
321
.
2.
Berry
,
D. M.
,
2007
, “
Ambiguity in Natural Language Requirements Documents
,”
Monterey Workshop
,
Monterey, CA
,
Sept. 10–13
,
Springer
, pp.
1
7
.
3.
Brown
,
T.
,
Mann
,
B.
,
Ryder
,
N.
,
Subbiah
,
M.
,
Kaplan
,
J. D.
,
Dhariwal
,
P.
, and
Neelakantan
,
A.
, et al.
2020
,
Language Models Are Few-Shot Learners
,”
Advances in Neural Information Processing Systems
,
H.
Larochelle
,
M.
Ranzato
,
R.
Hadsell
,
M. F.
Balcan
, and
H.
Lin
, eds.,
Curran Associates, Inc.
,
New Orleans, LA
, pp.
1877
1901
.
4.
Vaswani
,
A.
,
Shazeer
,
N.
,
Parmar
,
N.
,
Uszkoreit
,
J.
,
Jones
,
L.
,
Gomez
,
A. N.
,
Kaiser
,
Ł.
, and
Polosukhin
,
I.
,
2017
, “
Attention Is All You Need
,”
Advances in Neural Information Processing Systems
, Vol.
30
, pp.
6000
6010
.
5.
Wei
,
J.
,
Wang
,
X.
,
Schuurmans
,
D.
,
Bosma
,
M.
,
Xia
,
F.
,
Chi
,
Ed.
,
Le
,
Q. V.
, and
Zhou
,
D.
,
2022
, “
Chain-of-thought Prompting Elicits Reasoning in Large Language Models
,”
Adv. Neural Inf. Process. Syst.
,
35
, pp.
24824
24837
.
6.
Maalej
,
W.
,
Nayebi
,
M.
,
Johann
,
T.
, and
Ruhe
,
G.
,
2015
, “
Toward Data-Driven Requirements Engineering
,”
IEEE Softw.
,
33
(
1
), pp.
48
54
.
7.
Lin
,
J.
, and
Seepersad
,
C. C.
,
2007
, “
Empathic Lead Users: The Effects of Extraordinary User Experiences on Customer Needs Analysis and Product Redesign
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Las Vegas, NV
,
Sept. 4–7
, pp.
289
296
.
8.
Lingard
,
L.
,
2023
, “
Writing With ChatGPT: An Illustration of Its Capacity, Limitations & Implications for Academic Writers
,”
Perspect. Med. Educ.
,
12
(
1
), p.
261
.
9.
Htet
,
A.
,
Liana
,
S. R.
,
Aung
,
T.
, and
Bhaumik
,
A.
,
2024
, “ChatGPT in Content Creation: Techniques, Applications, and Ethical Implications,”
Advanced Applications of Generative AI and Natural Language Processing Models
,
A. J.
Obaid
, et al
, eds.,
IGI Global
,
Hershey, PA
, pp.
43
68
.
10.
Subagja
,
A. D.
,
Ausat
,
A. M. A.
,
Sari
,
A. R.
,
Wanof
,
M. I.
, and
Suherlan
,
S.
,
2023
, “
Improving Customer Service Quality in MSMEs Through the Use of ChatGPT
,”
J. Minfo Polgan
,
12
(
2
), pp.
380
386
.
11.
Li
,
Y.
,
Choi
,
D.
,
Chung
,
J.
,
Kushman
,
N.
,
Schrittwieser
,
J.
,
Leblond
,
R.
, and
Eccles
,
T.
, et al.
2022
, “
Competition-Level Code Generation With AlphaCode
,”
Science
,
378
(
6624
), pp.
1092
1097
.
12.
Doris
,
A. C.
,
Grandi
,
D.
,
Tomich
,
R.
,
Alam
,
M. F.
,
Ataei
,
M.
,
Cheong
,
H.
, and
Ahmed
,
F.
,
2024
, “
DesignQA: A Multimodal Benchmark for Evaluating Large Language Models’ Understanding of Engineering Documentation
,” arXiv preprint arXiv:2404.07917.
13.
Etesam
,
Y.
,
Cheong
,
H.
,
Ataei
,
M.
, and
Jayaraman
,
P. K.
,
2024
, “
Deep Generative Model for Mechanical System Configuration Design
,” arXiv preprint arXiv:2409.06016.
14.
Shanahan
,
M.
,
McDonell
,
K.
, and
Reynolds
,
L.
,
2023
, “
Role Play With Large Language Models
,”
Nature
,
623
(
7987
), pp.
493
498
.
15.
Csepregi
,
L. M.
,
2021
, “
The Effect of Context-Aware LLM-Based NPC Conversations on Player Engagement in Role-Playing Video Games
,” Unpublished Manuscript.
16.
Zhu
,
A.
,
Martin
,
L.
,
Head
,
A.
, and
Callison-Burch
,
C.
,
2023
, “
CALYPSO: LLMs as Dungeon Master’s Assistants
,”
Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment
,
Salt Lake City, UT
,
Oct. 8–12
, pp.
380
390
.
17.
Gray
,
C.
,
Yilmaz
,
S.
,
McKilligan
,
S.
,
Daly
,
S.
,
Seifert
,
C.
, and
Gonzalez
,
R.
,
2015
, “
Idea Generation Through Empathy: Reimagining the ‘Cognitive Walkthrough
,”
ASEE Annual Conference & Exposition
,
Seattle, WA
,
June 14–17
.
18.
Schmitt
,
E.
, and
Morkos
,
B.
,
2016
, “
Teaching Students Designer Empathy in Senior Capstone Design
,”
Capstone Design Conference
,
Columbus, OH
,
June 6–8
.
19.
Walther
,
J.
,
Miller
,
S. E.
, and
Kellam
,
N. N.
,
2012
, “
Exploring the Role of Empathy in Engineering Communication Through a Transdisciplinary Dialogue
,”
2012 ASEE Annual Conference & Exposition
,
San Antonio, TX
,
June 10–13
, pp.
25
622
.
20.
Hannukainen
,
P.
, and
Hol̈tta-Otto
,
K.
,
2006
, “
Identifying Customer Needs: Disabled Persons as Lead Users
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Philadelphia, PA
,
Sept. 10–13
, pp.
243
251
.
21.
Leonard
,
D.
, and
Rayport
,
J. F.
,
1997
, “
Spark Innovation Through Empathic Design
,”
Harv. Bus. Rev.
,
75
, pp.
102
115
.
22.
Otto
,
K. N.
,
2003
,
Product Design: Techniques in Reverse Engineering and New Product Development
,
Tsinghua University Press Co. Ltd
,
Hoboken, NJ
.
23.
Ulrich
,
K. T.
, and
Eppinger
,
S. D.
,
2016
,
Product Design and Development
,
McGraw-Hill
,
New York
.
24.
Suh
,
N. P.
,
1990
,
The Principles of Design
,
Oxford University Press
,
New York
.
25.
Von Hippel
,
E.
,
1986
, “
Lead Users: A Source of Novel Product Concepts
,”
Manage. Sci.
,
32
(
7
), pp.
791
805
.
26.
Urban
,
G. L.
, and
Von Hippel
,
E.
,
1988
, “
Lead User Analyses for the Development of New Industrial Products
,”
Manage. Sci.
,
34
(
5
), pp.
569
582
.
27.
Issa
,
N. Md.
,
Sasaki
,
H.
,
Okamura
,
N.
,
Yahya
,
W. J.
,
Rahman
,
M. A. A.
,
Ariff
,
M. H. M.
, and
Koga
,
T.
,
2023
, “
Proposition and Verification of a Design Method to Discover Latent Needs Based on Empathy, Experiences, and Working Prototype by Designing Autonomous Childcare Vehicle
,”
J. Adv. Vehicle Syst.
,
14
(
1
), pp.
19
34
.
28.
Qiu
,
Y.
, and
Jin
,
Y.
,
2023
, “
Document Understanding-Based Design Support: Application of Language Model for Design Knowledge Extraction
,”
ASME J. Mech. Des.
,
145
(
12
), p.
121401
.
29.
Chen
,
L.
,
Jing
,
Q.
,
Tsang
,
Y.
,
Wang
,
Q.
,
Sun
,
L.
, and
Luo
,
Ji.
,
2024
, “
DesignFusion: Integrating Generative Models for Conceptual Design Enrichment
,”
ASME J. Mech. Des.
,
146
(
11
), p.
111703
.
30.
Zhu
,
Q.
, and
Luo
,
J.
,
2023
, “
Toward Artificial Empathy for Human-Centered Design: A Framework
.”
31.
Zhu
,
Q.
,
Chong
,
L.
,
Yang
,
M.
, and
Luo
,
J.
,
2024
, “
Reading Users’ Minds From What They Say: An Investigation Into LLM-Based Empathic Mental Inference
,” arXiv preprint arXiv:2403.13301.
32.
Barandoni
,
S.
,
Chiarello
,
F.
,
Cascone
,
L.
,
Marrale
,
E.
, and
Puccio
,
S.
,
2024
, “
Automating Customer Needs Analysis: A Comparative Study of Large Language Models in the Travel Industry
,” arXiv preprint arXiv:2404.17975.
33.
Strobel
,
J.
,
Hess
,
J.
,
Pan
,
R.
, and
Wachter Morris
,
C. A.
,
2013
, “
Empathy and Care Within Engineering: Qualitative Perspectives From Engineering Faculty and Practicing Engineers
,”
Eng. Stud.
,
5
(
2
), pp.
137
159
.
34.
Raviselvam
,
S.
,
Sanaei
,
R.
,
Blessing
,
L.
,
Hölttä-Otto
,
K.
, and
Wood
,
K. L.
,
2017
, “
Demographic Factors and Their Influence on Designer Creativity and Empathy Evoked Through User Extreme Conditions
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Cleveland, OH
,
Aug. 6–9
,
American Society of Mechanical Engineers
, p.
V007T06A011
.
35.
Surma-Aho
,
A.
,
Björklund
,
T.
, and
Hölttä-Otto
,
K.
,
2018
, “
An Analysis of Designer Empathy in the Early Phases of Design Projects
,” DS 91: Proceedings of NordDesign 2018, Aug. 14–17, Linköping, Sweden.
36.
Tang
,
X.
,
2018
, “
From ‘Empathic Design’ to ‘Empathic Engineering’: Toward a Genealogy of Empathy in Engineering Education
,” 2018
ASEE Annual Conference & Exposition
,
Salt Lake City, UT
,
June 23–27
.
37.
Boden
,
M. A.
,
2009
, “
Computer Models of Creativity
,”
AI Mag.
,
30
(
3
), pp.
23
23
.
38.
Amabile
,
T. M.
,
1988
, “
A Model of Creativity and Innovation in Organizations
,”
Res. Organ. Behav.
,
10
(
1
), pp.
123
167
.
39.
Amabile
,
T. M.
,
1982
, “
Social Psychology of Creativity: A Consensual Assessment Technique
,”
J. Personal. Soc. Psychol.
,
43
(
5
), p.
997
.
40.
Miller
,
S. R
,
Hunter
,
S. T.
,
Starkey
,
E.
,
Ramachandran
,
S.
,
Ahmed
,
F.
, and
Fuge
,
M.
,
2021
, “
How Should We Measure Creativity in Engineering Design? A Comparison Between Social Science and Engineering Approaches
,”
ASME J. Mech. Des.
,
143
(
3
), p.
031404
.
41.
Amabile
,
T. M.
,
2018
,
Creativity in Context: Update to the Social Psychology of Creativity
,
Routledge
,
New York
.
42.
Regenwetter
,
L.
,
Srivastava
,
A.
,
Gutfreund
,
D.
, and
Ahmed
,
F.
,
2023
, “
Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design
,”
Comput. Aid. Des.
,
165
, p.
103609
.
43.
Ma
,
K.
,
Grandi
,
D.
,
McComb
,
C.
, and
Goucher-Lambert
,
K.
,
2023
, “
Conceptual Design Generation Using Large Language Models
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Boston, MA
,
Aug. 20–23
.
44.
Jiralerspong
,
M.
,
Bose
,
J.
,
Gemp
,
I.
,
Qin
,
C.
,
Bachrach
,
Y.
, and
Gidel
,
G.
,
2024
, “Feature Likelihood Score: Evaluating the Generalization of Generative Models Using Samples,”
Advances in Neural Information Processing Systems
, Vol.
36
.
45.
Sarica
,
S.
, and
Luo
,
J.
,
2023
, “
Innovation Slowdown: Decelerating Concept Creation and Declining Originality in New Technological Concepts
,” arXiv preprint arXiv:2303.13300.
46.
Picard
,
C.
,
Schiffmann
,
J.
, and
Ahmed
,
F.
,
2023
, “
DATED: Guidelines for Creating Synthetic Datasets for Engineering Design Applications
,” arXiv preprint arXiv:2305.09018.
47.
Regenwetter
,
L.
,
Abu Obaideh
,
Y.
, and
Ahmed
,
F.
,
2023
, “
Counterfactuals for Design: A Model-agnostic Method for Design Recommendations
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Boston, MA
,
Aug. 20–23
,
American Society of Mechanical Engineers
, p.
V03AT03A008
.
48.
Bagazinski
,
N. J.
, and
Ahmed
,
F.
,
2023
, “
ShipGen: A Diffusion Model for Parametric Ship Hull Generation With Multiple Objectives and Constraints
,”
J. Marine Sci. Eng.
,
11
(
12
), pp.
2215
.
49.
Fan
,
J.
,
Vuaille
,
L.
,
Bäck
,
T.
, and
Wang
,
H.
,
2023
, “
On the Noise Scheduling for Generating Plausible Designs With Diffusion Models
,” arXiv preprint arXiv:2311.11207.
50.
Rousseeuw
,
P. J.
,
1987
, “
Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis
,”
J. Comput. Appl. Math.
,
20
, pp.
53
65
.
51.
Podani
,
J.
,
2009
, “
Convex Hulls, Habitat Filtering, and Functional Diversity: Mathematical Elegance Versus Ecological Interpretability
,”
Commun. Ecol.
,
10
(
2
), pp.
244
250
.
52.
Mueller
,
C. T.
, and
Ochsendorf
,
J. A.
,
2015
, “
Combining Structural Performance and Designer Preferences in Evolutionary Design Space Exploration
,”
Autom. Construct.
,
52
, pp.
70
82
.
53.
Brown
,
N. C.
, and
Mueller
,
C. T.
,
2019
, “
Quantifying Diversity in Parametric Design: A Comparison of Possible Metrics
,”
AI EDAM
,
33
(
1
), pp.
40
53
.
54.
Zanitti
,
M.
,
Sørensen
,
J.
,
Terolli
,
E.
, and
Kosta
,
S.
,
2022
,
Exploiting Consumption Diversity and neighbour Similarity Trade-offs in Recommender Systems: a User-Centric Offline Evaluation of Diversity Objectives
.
55.
Chaudhuri
,
A.
,
Sarma
,
M.
, and
Samanta
,
D.
,
2019
, “
Advanced Feature Identification Towards Research Article Recommendation: A Machine Learning Based Approach
,”
TENCON 2019-2019 IEEE Region 10 Conference (TENCON)
,
Kerala, India
,
Oct. 17–20
,
IEEE
, pp.
7
12
.
56.
Holbrook III
,
H.
,
1990
, “
A Scenario-Based Methodology for Conducting Requirements Elicitation
,”
ACM SIGSOFT Softw. Eng. Notes
,
15
(
1
), pp.
95
104
.
57.
OpenAI
,
Achiam
,
J.
,
Adler
,
S.
,
Agarwal
,
S.
,
Ahmad
,
L.
,
Akkaya
,
I.
,
Aleman
,
F. L.
, and
Almeida
,
D.
,
2024
, “
GPT-4 Technical Report
,” URL 2303.08774. https://arxiv.org/abs/2303.08774
58.
Van der Maaten
,
L.
, and
Hinton
,
G.
,
2008
, “
Visualizing Data Using t-SNE
,”
J. Mach. Learn. Res.
,
9
(
11
), pp.
2579
2605
.
59.
Hripcsak
,
G.
, and
Rothschild
,
A. S.
,
2005
, “
Agreement, the f-Measure, and Reliability in Information Retrieval
,”
J. Am. Med. Inf. Assoc.
,
12
(
3
), pp.
296
298
.
You do not currently have access to this content.