Evaluating large language models on the core cognitive capabilities expected of technology managers: a hierarchical benchmark

Zhipeng ZHU; Guo LI; Mark Xuefang ZHU; Shanshan KONG; Zhenyu WANG; Wenting XIONG

doi:10.1631/ENG.ITEE.2026.0168

Your Location：

Home >

Browse articles >

Evaluating large language models on the core cognitive capabilities expected of technology managers: a hierarchical benchmark

Regular Papers | Updated：2026-11-05

- Evaluating large language models on the core cognitive capabilities expected of technology managers: a hierarchical benchmark
- Evaluating large language models on the core cognitive capabilities expected of technology managers: a hierarchical benchmark
- ENGINEERING Information Technology & Electronic Engineering Vol. 27, Issue 10, Pages: 1-19(2026)
- Affiliations：
  
  School of Information Management, Nanjing University, Nanjing 210023, China
- Author bio：
  
  ✉ Mark Xuefang ZHU, xfzhu@nju.edu.cn
- Funds：
- DOI：10.1631/ENG.ITEE.2026.0168
  CLC： TP18;G311
- Received：02 June 2026，
  
  Revised：2026-08-09，
  
  Published：01 October 2026
- Accepted：
Scan QR Code
Zhipeng ZHU, Guo LI, Mark Xuefang ZHU, et al. Evaluating large language models on the core cognitive capabilities expected of technology managers: a hierarchical benchmark[J]. ENGINEERING Information Technology & Electronic Engineering, 2026, 27(10): 1-19.
DOI：

Zhipeng ZHU, Guo LI, Mark Xuefang ZHU, et al. Evaluating large language models on the core cognitive capabilities expected of technology managers: a hierarchical benchmark[J]. ENGINEERING Information Technology & Electronic Engineering, 2026, 27(10): 1-19. DOI： 10.1631/ENG.ITEE.2026.0168.

摘要

Abstract

Technology managers are instrumental in facilitating technology transfer. While large language models (LLMs) have demonstrated potential in this domain

evaluating them on the core cognitive capabilities expected of technology managers remains challenging due to the lack of domain-specific benchmarks. To address this challenge

we construct TMBench

a domain-specific benchmark for evaluating the cognitive capabilities expected of technology managers. It comprises a collection of 3474 questions spanning diverse scenarios

covering three tasks and five question types. We integrate typical metrics with LLM-as-a-Judge using both fixed and entropy-based weights to calculate the corresponding dimensional and composite scores. Leveraging this mechanism

we evaluate nine representative models under varying prompt configurations and temperature settings to investigate their capabilities

characterize their error patterns

and benchmark them against human experts. Our results show heterogeneous capabilities

with no single model ranking first across all three capability levels. For higher-order capabilities

LLMs show more favorable point estimates than the human baseline

but the differences are not statistically significant. Chain-of-thought-based prompting is associated with prompt-sensitive capability profiles

with more favorable open-weight performance in the two higher-order tasks. Error analysis identifies knowledge base

retrieval

and application deficits as the most prevalent error category. Building on these findings

this study provides a reference for model selection

prompt engineering

and targeted optimization when applying LLMs in technology transfer.

关键词

Keywords

references

Alibaba Group , 2025 . Qwen3-Next-80B: instruction-tuned models for RAG workflows . https://qwen.ai/blog/qwen3-next-ultimate-efficiency/ https://qwen.ai/blog/qwen3-next-ultimate-efficiency/ [Accessed on Jan. 20, 2026 ] .

Altermatt FR , Neyem A , Sumonte NI , et al. , 2025 . Evaluating the performance of large language models on the CONACEM anesthesiology certification exam: a comparison with human participants . Appl Sci , 15 ( 11 ): 6245 . https://doi.org/10.3390/app15116245 https://doi.org/10.3390/app15116245

Anthropic , 2026 . Introducing Claude Haiku 4.5: high-efficiency iterations of the Claude family . https://www.anthropic.com/news/claude-haiku-4-5 https://www.anthropic.com/news/claude-haiku-4-5 [Accessed on Jan. 20, 2026 ] .

Association of University Technology Managers , 2024 . Technology Transfer Practice Manual . https://autm.net/surveys-and-tools/tech-transferpractices-manual https://autm.net/surveys-and-tools/tech-transferpractices-manual [Accessed on Jan. 20, 2026 ] .

Badenes-Olmedo C , Eyzaguirre-Barreda P , Chu-Artzt N , et al. , 2026 . ESQAD: a curriculum-aligned dataset for question answer generation in Spanish . Knowl-Based Syst , 335 : 115255 . https://doi.org/10.1016/j.knosys.2025.115255 https://doi.org/10.1016/j.knosys.2025.115255

Bastani H , Karger E , Chen YH , et al. , 2025 . ForecastBench: a dynamic benchmark of AI forecasting capabilities . Proc 13 th Int Conf on Learning Representations .

BenYoash N , Brief M , Ovadia O , et al. , 2025 . SECQUE: a benchmark for evaluating real-world financial analysis capabilities . Proc 4 th Workshop on Generation, Evaluation and Metrics , p. 212 - 230 .

Betancourt J , Coral A , Fraga A , et al. , 2025 . Intelligent virtual assistant for calculating technology readiness levels using large language models (LLM) . IEEE Access , 13 : 137709 - 137723 . https://doi.org/10.1109/access.2025.3595699 https://doi.org/10.1109/access.2025.3595699

Binegde GN , Zhang HP , 2026 . Exploring cultural commonsense in multilingual large language models: a survey . Inform Syst , 138 : 102649 . https://doi.org/10.1016/j.is.2025.102649 https://doi.org/10.1016/j.is.2025.102649

Busch K , Leopold H , 2024 . Towards a benchmark for large language models for business process management tasks . https://doi.org/10.48550/arXiv.2410.03255 https://doi.org/10.48550/arXiv.2410.03255

Center for AI Safety , Scale AI , HLE Contributors Consortium , 2026 . A benchmark of expert-level academic questions to assess AI capabilities . Nature , 649 ( 8099 ): 1139 - 1146 . https://doi.org/10.1038/s41586-025-09962-4 https://doi.org/10.1038/s41586-025-09962-4

Cetindamar D , Phaal R , Probert DR , 2016 . Technology management as a profession and the challenges ahead . J Eng Technol Manag , 41 : 1 - 13 . https://doi.org/10.1016/j.jengtecman.2016.05.001 https://doi.org/10.1016/j.jengtecman.2016.05.001

Chadli K , Botterweck G , Saber T , 2026 . PEARL: performance and energy aware routing for LLMs . Fut Gener Comput Syst , 176 : 108218 . https://doi.org/10.1016/j.future.2025.108218 https://doi.org/10.1016/j.future.2025.108218

Chang CC , Chen CY , Lee HS , et al. , 2024 . Benchmarking cognitive domains for LLMs: insights from Taiwanese Hakka culture . Proc 27 th Conf of the Oriental COCOSDA Int Committee , p. 1 - 6 . https://doi.org/10.1109/O-COCOSDA64382.2024.10800594 https://doi.org/10.1109/O-COCOSDA64382.2024.10800594

Chang YP , Wang X , Wang JD , et al. , 2024 . A survey on evaluation of large language models . ACM Trans Intell Syst Technol , 15 ( 3 ): 39 . https://doi.org/10.1145/3641289 https://doi.org/10.1145/3641289

China Association of Science and Technology Evaluation and Management of Scientific and Technical Achievement (CASTEM) , National Center for Science and Technology Evaluation (NCSTE) , 2025 . Advanced Tutorial Technical Manager . Science and Technology of China Press , Beijing, China (in Chinese) .

Chiu YY , Jiang LW , Lin BY , et al. , 2025 . CulturalBench: a robust, diverse and challenging benchmark for measuring LMs' cultural knowledge through human-AI red-teaming . Proc 63 rd Annual Meeting of the Association for Computational Linguistics , p. 25663 - 25701 . https://doi.org/10.18653/v1/2025.acl-long.1247 https://doi.org/10.18653/v1/2025.acl-long.1247

Chowa SS , Alvi R , Rahman SS , et al. , 2026 . From language to action: a review of large language models as autonomous agents and tool users . Artif Intell Rev , 59 ( 2 ): 71 . https://doi.org/10.1007/s10462-025-11471-9 https://doi.org/10.1007/s10462-025-11471-9

COGR , 2021 . The evolution of technology transfer . https://www.cogr.edu/sites/default/files/COGR%20Tech%20Transfer10621.pdf https://www.cogr.edu/sites/default/files/COGR%20Tech%20Transfer10621.pdf [Accessed on Jan. 20, 2026 ] .

Dai H , Teehan R , Ren MY , 2025 . Are LLMs prescient? A continuous evaluation using daily news as the oracle . https://doi.org/10.48550/arXiv.2411.08324 https://doi.org/10.48550/arXiv.2411.08324

Decoupes R , Interdonato R , Roche M , et al. , 2025 . Evaluation of geographical distortions in language models . Mach Learn , 114 ( 12 ): 1 - 32 . https://doi.org/10.1007/s10994-025-06916-9 https://doi.org/10.1007/s10994-025-06916-9

DeepSeek AI , 2025 . DeepSeek-V3.2: high-compute optimization and reasoning performance . https://api-docs.deepseek.com/news/news251201/ https://api-docs.deepseek.com/news/news251201/ [Accessed on Jan. 20, 2026 ] .

Deroy A , Ghosh K , Ghosh S , 2026 . Investigating legal question generation using large language models . Artif Intell Law , 34 : 775 - 813 . https://doi.org/10.1007/s10506-025-09452-y https://doi.org/10.1007/s10506-025-09452-y

DeVore S , 2025 . Exploring the ability of LLMs to classify written proficiency levels . Comput Speech Lang , 90 : 101745 . https://doi.org/10.1016/j.csl.2024.101745 https://doi.org/10.1016/j.csl.2024.101745

Fan ZY , He JL , Li XQ , et al. , 2025 . CAMIEval: enhancing NLG evaluation through multidimensional comparative instruction-following analysis . Proc Conf of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , p. 8708 - 8733 . https://doi.org/10.18653/v1/2025.naacl-long.438 https://doi.org/10.18653/v1/2025.naacl-long.438

Gao MQ , Hu XY , Yin XJ , et al. , 2025 . LLM-based NLG evaluation: current status and challenges . Comput Ling , 51 ( 2 ): 661 - 687 . https://doi.org/10.1162/coli_a_00561 https://doi.org/10.1162/coli_a_00561

Genç O , 2026 . Is conversational AI ready for engineering licensure? A ChatGPT performance benchmark and prompting strategy evaluation . J Asian Archit Build Eng , p. 1 - 11 . https://doi.org/10.1080/13467581.2025.2574556 https://doi.org/10.1080/13467581.2025.2574556

Georgiev P , Lei VL , Burnell R , et al. , 2024 . Gemini 1.5: unlocking multimodal understanding across millions of tokens of context . https://arXiv.org/abs/2403.05530 https://arXiv.org/abs/2403.05530 .

Gignac GE , Ilić D , 2025 . Psychometrically derived 60-question benchmarks: substantial efficiencies and the possibility of human-AI comparisons . Intelligence , 110 : 101922 . https://doi.org/10.1016/j.intell.2025.101922 https://doi.org/10.1016/j.intell.2025.101922

Google DeepMind , 2025 . Gemini 3 Flash: frontier intelligence built for speed . https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/ https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/ [Accessed on Jan. 20, 2026 ] .

Gorschek T , Garre P , Larsson S , et al. , 2006 . A model for technology transfer in practice . IEEE Softw , 23 ( 6 ): 88 - 95 . https://doi.org/10.1109/MS.2006.147 https://doi.org/10.1109/MS.2006.147

Guo SK , Shariatmadari AH , Xiong GZ , et al. , 2025 . IDEABench: benchmarking large language models for research idea generation . Proc 31 st ACM SIGKDD Conf on Knowledge Discovery and Data Mining V.2 , p. 5888 - 5899 . https://doi.org/10.1145/3711896.3737419 https://doi.org/10.1145/3711896.3737419

Guo X , Xia HT , Liu ZW , et al. , 2025 . FinEval: a Chinese financial domain knowledge evaluation benchmark for large language models . Proc Conf of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , p. 6258 - 6292 . https://doi.org/10.18653/v1/2025.naacl-long.318 https://doi.org/10.18653/v1/2025.naacl-long.318

Halawi D , Zhang F , Chen YH , et al. , 2024 . Approaching human-level forecasting with language models . Proc 38 th Int Conf on Neural Information Processing Systems , p. 50426 - 50468 . https://doi.org/10.52202/079017-1598 https://doi.org/10.52202/079017-1598

Harrison W , 2006 . Technology transfer and the tech broker . IEEE Softw , 23 ( 5 ): 5 - 7 . https://doi.org/10.1109/MS.2006.142 https://doi.org/10.1109/MS.2006.142

Huang L , Yu WJ , Ma WT , et al. , 2025 . A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions . ACM Trans Inform Syst , 43 ( 2 ): 42 . https://doi.org/10.1145/3703155 https://doi.org/10.1145/3703155

Hunan University , National Center for Science and Technology Evaluation (NCSTE) , 2025 . Elementary Tutorial Technical Manager . Science and Technology of China Press , Beijing, China (in Chinese) .

Jiang LK , Zhang CQ , Scherz PA , et al. , 2025 . Can large language models generate high-quality patent claims ? Findings of the Association for Computational Linguistics , p. 1272 - 1287 . https://doi.org/10.18653/v1/2025.findings-naacl.70 https://doi.org/10.18653/v1/2025.findings-naacl.70

Jung J , Lu M , Benker SC , et al. , 2025 . How model size, temperature, and prompt style affect LLM-human assessment score alignment . Proc Artificial Intelligence in Measurement and Education Conf , p. 265 - 273 .

Kampelopoulos D , Tsanousa A , Vrochidis S , et al. , 2025 . A review of LLMs and their applications in the architecture, engineering and construction industry . Artif Intell Rev , 58 ( 8 ): 250 . https://doi.org/10.1007/s10462-025-11241-7 https://doi.org/10.1007/s10462-025-11241-7

Khera B , Alamian R , Scherz PA , et al. , 2025 . Can large language models understand as well as apply patent regulations to pass a hands-on patent attorney test ? https://doi.org/10.48550/arXiv.2507.10576 https://doi.org/10.48550/arXiv.2507.10576

Kim J , Podlasek A , Shidara K , et al. , 2025 . Limitations of large language models in clinical problem-solving arising from inflexible reasoning . Sci Rep , 15 ( 1 ): 39426 . https://doi.org/10.1038/s41598-025-22940-0 https://doi.org/10.1038/s41598-025-22940-0

Kim JS , Kim BK , 2022 . Examining different technology transfer capabilities and their counterpart works from two different positions . Technol Soc , 68 : 101856 . https://doi.org/10.1016/j.techsoc.2021.101856 https://doi.org/10.1016/j.techsoc.2021.101856

Krathwohl DR , 2002 . A revision of Bloom's taxonomy: an overview . Theory Pract , 41 ( 4 ): 212 - 218 . https://doi.org/10.1207/s15430421tip4104_2 https://doi.org/10.1207/s15430421tip4104_2

Lane JP , 1999 . Understanding technology transfer . Assist Technol , 11 ( 1 ): 5 - 19 . https://doi.org/10.1080/10400435.1999.10131981 https://doi.org/10.1080/10400435.1999.10131981

Li HT , Chen Y , Ai QY , et al. , 2024 . LexEval: a comprehensive Chinese legal benchmark for evaluating large language models . Proc 38 th Int Conf on Neural Information Processing Systems , p. 25061 - 25094 . https://doi.org/10.52202/079017-0790 https://doi.org/10.52202/079017-0790

Li JW , Gao YZ , Yang Y , et al. , 2025 . Fundamental capabilities and applications of large language models: a survey . ACM Comput Surv , 58 ( 2 ): 38 . https://doi.org/10.1145/3735632 https://doi.org/10.1145/3735632

Li KY , Jiang MH , Fu DY , et al. , 2025 . DatasetResearch: benchmarking agent systems for demand-driven dataset discovery . https://doi.org/10.48550/arXiv.2508.06960 https://doi.org/10.48550/arXiv.2508.06960

Li SW , Jiang J , Yang XD , 2025 . Preliminary assessment of large language models’ performance in answering questions on developmental dysplasia of the hip . J Child Orthop , 19 ( 3 ): 207 - 212 . https://doi.org/10.1177/18632521251331772 https://doi.org/10.1177/18632521251331772

Li YJ , Sun Y , 2026 . A fine-grained evaluation framework for language models: combining pointwise grading and pairwise comparison . Inform Process Manag , 63 ( 1 ): 104270 . https://doi.org/10.1016/j.ipm.2025.104270 https://doi.org/10.1016/j.ipm.2025.104270

Liang C , Huang ZQ , Wang HF , et al. , 2026 . AECBench: a hierarchical benchmark for knowledge evaluation of large language models in the AEC field . Adv Eng Inform , 71 : 104314 . https://doi.org/10.1016/j.aei.2026.104314 https://doi.org/10.1016/j.aei.2026.104314

Lin CB , Liang HY , Su TY , et al. , 2022 . Multi-criteria decision-making approach for analyzing competency model of technology managers . Math Probl Eng , 2022 ( 1 ): 1909851 . https://doi.org/10.1155/2022/1909851 https://doi.org/10.1155/2022/1909851

Liu AX , Feng B , Xue B , et al. , 2025 . DeepSeek-v3 technical report . https://doi.org/10.48550/arXiv.2412.19437 https://doi.org/10.48550/arXiv.2412.19437

Liu QY , Zou T , Xu CH , et al. , 2025 . Shanghai Service Manual about Technology Commercialization . Shanghai Jiao Tong University Press , Shanghai, China (in Chinese) .

Longpre S , Perisetla K , Chen A , et al. , 2021 . Entity-based knowledge conflicts in question answering . Proc Conf on Empirical Methods in Natural Language Processing , p. 7052 - 7063 . https://doi.org/10.18653/v1/2021.emnlp-main.565 https://doi.org/10.18653/v1/2021.emnlp-main.565

MacMillan IC , McGrath RG , 2004 . Nine new roles for technology managers . Res-Technol Manag , 47 ( 3 ): 16 - 26 . https://doi.org/10.1080/08956308.2004.11671626 https://doi.org/10.1080/08956308.2004.11671626

Manifold , 2026 . Manifold Markets: The World’s Largest Prediction Market Platform . https://manifold.markets/ https://manifold.markets/ [Accessed on Jan. 20, 2026 ] .

Matrinex Academy of International Innovation & Strategy (MAIIS) , National Center for Science and Technology Evaluation (NCSTE) , 2025 . Intermediate Tutorial Technical Manager . Science and Technology of China Press , Beijing, China (in Chinese) .

Meta AI , 2025 . Llama 4 Maverick: sparse mixture-of-experts (MoE) architectures . https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/ [Accessed on Jan. 20, 2026 ] .

Miao ZY , Sun QY , Wang JY , et al. , 2025 . Easy Dataset: a unified and extensible framework for synthesizing LLM fine-tuning data from unstructured documents . Proc Conf on Empirical Methods in Natural Language Processing: System Demonstrations , p. 960 - 968 . https://doi.org/10.18653/v1/2025.emnlp-demos.75 https://doi.org/10.18653/v1/2025.emnlp-demos.75

Mistral AI , 2025 . Mistral Small 3.1: latency-optimized 24B parameters model . https://mistral.ai/news/mistral-small-3-1/ https://mistral.ai/news/mistral-small-3-1/ [Accessed on Jan. 20, 2026 ] .

Moonshot AI , 2025 . Kimi-K2-0905: scaling 1T-parameter MoE architectures . https://www.moonshot.cn/news/kimi-k2-announcement/ https://www.moonshot.cn/news/kimi-k2-announcement/ [Accessed on Jan. 20, 2026 ] .

Morrissey MT , Almonacid S , 2005 . Rethinking technology transfer . J Food Eng , 67 ( 1-2 ): 135 - 145 . https://doi.org/10.1016/j.jfoodeng.2004.05.057 https://doi.org/10.1016/j.jfoodeng.2004.05.057

NASA , 2020 . Technology readiness assessment best practices guide . NASA Special Publication , NASA/SP-20205003605 . https://ntrs.nasa.gov/citations/20205003605 https://ntrs.nasa.gov/citations/20205003605 [Accessed on Jan. 20, 2026 ] .

NASA , 2026 . TechPort 101 . https://techport.nasa.gov/help/TechPort101 https://techport.nasa.gov/help/TechPort101 [Accessed on Jan. 20, 2026 ] .

Naveed H , Khan AU , Qiu S , et al. , 2025 . A comprehensive overview of large language models . ACM Trans Intell Syst Technol , 16 ( 5 ): 106 . https://doi.org/10.1145/3744746 https://doi.org/10.1145/3744746

Ni YQ , Huang JS , Song LL , 2024 . Guidelines for Enhancing the Capacity of National Technology Transfer Professionals . University of Science and Technology of China Press , Hefei, China (in Chinese) .

Nie Y , Yan BW , Guo TY , et al. , 2025 . CFinBench: a comprehensive Chinese financial benchmark for large language models . Proc Conf of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , p. 876 - 891 . https://doi.org/10.18653/v1/2025.naacl-long.40 https://doi.org/10.18653/v1/2025.naacl-long.40

OpenAI , 2026 . GPT-5-mini technical report . https://openai.com/index/introducing-gpt-5-for-developers/ https://openai.com/index/introducing-gpt-5-for-developers/ [Accessed on Jan. 20, 2026 ] .

Park J , Mendes E , Stanovsky G , et al. , 2026 . Look before you leap: estimating LLM benchmark scores from descriptions . https://arxiv.org/abs/2509.20645v2 https://arxiv.org/abs/2509.20645v2

Pei DS , Wu YW , He JH , et al. , 2025 . Methodology and benchmark for automated driving theory test of large language models . IEEE Trans Intell Transp Syst , 26 ( 10 ): 17013 - 17026 . https://doi.org/10.1109/TITS.2025.3571213 https://doi.org/10.1109/TITS.2025.3571213

Peters U , Chin-Yee B , 2025 . Generalization bias in large language model summarization of scientific research . R Soc Open Sci , 12 ( 4 ): 241776 . https://doi.org/10.1098/rsos.241776 https://doi.org/10.1098/rsos.241776

Pham DK , Vo BQ , 2024 . Towards reliable medical question answering: techniques and challenges in mitigating hallucinations in language models . https://doi.org/10.48550/arXiv.2408.13808 https://doi.org/10.48550/arXiv.2408.13808

Pombal J , Yoon D , Fernandes P , et al. , 2025 . M-Prometheus: a suite of open multilingual LLM judges . https://doi.org/10.48550/arXiv.2504.04953 https://doi.org/10.48550/arXiv.2504.04953

Rahman ABMA , Anwar S , Usman M , et al. , 2025 . DefAn: definitive answer dataset for LLM hallucination evaluation . Information , 16 ( 11 ): 937 . https://doi.org/10.3390/info16110937 https://doi.org/10.3390/info16110937

Reese ML , Smirnova A , 2024 . Comparing ChatGPT and humans on world knowledge and common-sense reasoning tasks: a case study of the Japanese Winograd schema challenge . Extended Abstracts of the CHI Conf on Human Factors in Computing Systems , p. 1 - 9 . https://doi.org/10.1145/3613905.3650975 https://doi.org/10.1145/3613905.3650975

Renze M , 2024 . The effect of sampling temperature on problem solving in large language models . Findings of the Association for Computational Linguistics , p. 7346 - 7356 . https://doi.org/10.18653/v1/2024.findings-emnlp.432 https://doi.org/10.18653/v1/2024.findings-emnlp.432

Rifkin KI , Fineman M , Ruhnke CH , 1999 . The human side: developing technical managers—first you need a competency model . Res-Technol Manag , 42 ( 2 ): 53 - 57 . https://doi.org/10.1080/08956308.1999.11671273 https://doi.org/10.1080/08956308.1999.11671273

Rogers EM , Takegami S , Yin J , 2001 . Lessons learned about technology transfer . Technovation , 21 ( 4 ): 253 - 261 . https://doi.org/10.1016/S0166-4972(00)00039-9 https://doi.org/10.1016/S0166-4972(00)00039-9

Saxena A , Aggarwal D , Badathala N , et al. , 2025 . A framework to synthetically generate fine-grained hallucinated data . Lang Resour Eval , 59 ( 4 ): 4223 - 4252 . https://doi.org/10.1007/s10579-025-09864-x https://doi.org/10.1007/s10579-025-09864-x

Schaufeld J , 2015 . The commercialization model: the value of a process . In: Commercializing Innovation: Turning Technology Breakthroughs into Products . Apress , Berkeley, CA , p. 17 - 30 . https://doi.org/10.1007/978-1-4302-6353-1_2 https://doi.org/10.1007/978-1-4302-6353-1_2

Schoenegger P , Tuminauskaite I , Park PS , 2024 . Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy . Sci Adv , 10 ( 45 ): eadp1528 . https://doi.org/10.1126/sciadv.adp1528 https://doi.org/10.1126/sciadv.adp1528

Servantez S , Barrow J , Hammond K , et al. , 2024 . Chain of logic: rule-based reasoning with large language models . Findings of the Association for Computational Linguistics , p. 2721 - 2733 . https://doi.org/10.18653/v1/2024.findings-acl.159 https://doi.org/10.18653/v1/2024.findings-acl.159

Sheft J , 2008 . Technology transfer and idea commercialization . Nat Biotechnol , 26 ( 6 ): 711 - 712 . https://doi.org/10.1038/nbt0608-711 https://doi.org/10.1038/nbt0608-711

Song PY , Han PR , Goodman N , 2026 . Large language model reasoning failures . https://doi.org/10.48550/arXiv.2602.06176 https://doi.org/10.48550/arXiv.2602.06176

Srivastava A , Rastogi A , Rao A , et al. , 2023 . Beyond the imitation game: quantifying and extrapolating the capabilities of language models . https://doi.org/10.48550/arXiv.2206.04615 https://doi.org/10.48550/arXiv.2206.04615

Sun LT , Han Y , Zhao ZH , et al. , 2024 . SciEval: a multi-level large language model evaluation benchmark for scientific research . Proc 38 th AAAI Conf on Artificial Intelligence , p. 19053 - 19061 . https://doi.org/10.1609/aaai.v38i17.29872 https://doi.org/10.1609/aaai.v38i17.29872

Tak R , Pele DT , 2025 . LLM-driven stock prediction: capturing market trends with LLaMA . Proc Int Conf on Busin Excell , 19:529 - 540 . https://doi.org/10.2478/picbe-2025-0043 https://doi.org/10.2478/picbe-2025-0043

Tao YC , Xu HT , 2026 . TheoremForge: scaling up formal data synthesis with low-budget agentic workflow . https://doi.org/10.48550/arXiv.2601.17332 https://doi.org/10.48550/arXiv.2601.17332

Ting YS , Nguyen TD , Ghosal T , et al. , 2025 . AstroMLab 1: who wins astronomy jeopardy! ? Astron Comput , 51 : 100893 . https://doi.org/10.1016/j.ascom.2024.100893 https://doi.org/10.1016/j.ascom.2024.100893

Tofeeq K , Naseer A , Wali A , 2025 . Large language models in healthcare: a systematic evaluation on medical Q/A datasets . Health Inform Sci Syst , 14 ( 1 ): 2 . https://doi.org/10.1007/s13755-025-00397-9 https://doi.org/10.1007/s13755-025-00397-9

Toroghi A , Guo W , Sanner S , 2025 . CoLoTa: a dataset for entity-based commonsense reasoning over long-tail knowledge . Proc 48 th Int ACM SIGIR Conf on Research and Development in Information Retrieval , p. 3444 - 3454 . https://doi.org/10.1145/3726302.3730291 https://doi.org/10.1145/3726302.3730291

Vu T , Iyyer M , Wang XZ , et al. , 2024 . FreshLLMs: refreshing large language models with search engine augmentation . Findings of the Association for Computational Linguistics , p. 13697 - 13720 . https://doi.org/10.18653/v1/2024.findings-acl.813 https://doi.org/10.18653/v1/2024.findings-acl.813

Wang MS , Li ZQ , Zhao XG , et al. , 2026 . Eliminate-then-select: a human-centric reasoning framework for educational question answering with LLMs . Inform Process Manag , 63 ( 2 ): 104422 . https://doi.org/10.1016/j.ipm.2025.104422 https://doi.org/10.1016/j.ipm.2025.104422

Wang QY , Chen GH , Wang HB , et al. , 2025 . IPBench: benchmarking the knowledge of large language models in intellectual property . https://doi.org/10.48550/arXiv.2504.15524 https://doi.org/10.48550/arXiv.2504.15524

Wang RQ , Guo JY , Gao CY , et al. , 2025 . Can LLMs replace human evaluators? An empirical study of LLM-as-a-Judge in software engineering . Proc ACM Softw Eng , 2 ( ISSTA ): 1955 - 1977 . https://doi.org/10.1145/3728963 https://doi.org/10.1145/3728963

Wang XZ , Wei J , Schuurmans D , et al. , 2023 . Self-consistency improves chain of thought reasoning in language models . https://doi.org/10.48550/arXiv.2203.11171 https://doi.org/10.48550/arXiv.2203.11171

Wei KL , Paskov P , Dev S , et al. , 2025 . Recommendations and reporting checklist for rigorous & transparent human baselines in model evaluations . https://doi.org/10.48550/arXiv.2506.13776 https://doi.org/10.48550/arXiv.2506.13776

xAI , 2025 . Grok 4.1 Fast and Agent Tools API . https://x.ai/news/grok-4-1-fast https://x.ai/news/grok-4-1-fast [Accessed on Jan. 20, 2026 ] .

Xu K , Lian ZY , 2026 . WaferSAGE: large language model-powered wafer defect analysis via synthetic data generation and rubric-guided reinforcement learning . https://doi.org/10.48550/arXiv.2604.27629 https://doi.org/10.48550/arXiv.2604.27629

Xu LQ , Li QW , Peng TS , et al. , 2025 . Can large language models be good language teachers ? Proc Conf on Empirical Methods in Natural Language Processing , p. 23957 - 23971 . https://doi.org/10.18653/v1/2025.emnlp-main.1222 https://doi.org/10.18653/v1/2025.emnlp-main.1222

Yang QC , Mahns S , Li SD , et al. , 2025 . LLM-as-a-Prophet: understanding predictive intelligence with Prophet Arena . https://doi.org/10.48550/arXiv.2510.17638 https://doi.org/10.48550/arXiv.2510.17638

Yao SY , Yu D , Zhao J , et al. , 2023 . Tree of thoughts: deliberate problem solving with large language models . Proc 37 th Int Conf on Neural Information Processing Systems , p. 11809 - 11822 .

Yu JF , Wang XZ , Tu SQ , et al. , 2024 . KoLA: carefully benchmarking world knowledge of large language models . https://doi.org/10.48550/arXiv.2306.09296 https://doi.org/10.48550/arXiv.2306.09296

Zhang S , Huang QC , Zhang Q , et al. , 2025 . ElecBench: a large language model benchmark in electric power domain . Eng Appl Artif Intell , 162 : 112310 . https://doi.org/10.1016/j.engappai.2025.112310 https://doi.org/10.1016/j.engappai.2025.112310

Zhao XK , Huang H , Yang T , et al. , 2025 . Urban planning in the age of large language models: assessing OpenAI o1's performance and capabilities across 556 tasks . Comput Environ Urban Syst , 121 : 102332 . https://doi.org/10.1016/j.compenvurbsys.2025.102332 https://doi.org/10.1016/j.compenvurbsys.2025.102332

Zhao ZH , Huang ZX , Li JR , et al. , 2025 . SUPERChem: a multimodal reasoning benchmark in chemistry . https://doi.org/10.48550/arXiv.2512.01274 https://doi.org/10.48550/arXiv.2512.01274

Zheng LM , Chiang WL , Sheng Y , et al. , 2023 . Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena . Proc 37 th Int Conf on Neural Information Processing Systems , p. 46595 - 46623 .

Zografos G , Moussiades L , 2025 . Beyond the benchmark: a customizable platform for real-time, preference-driven LLM evaluation . Electronics , 14 ( 13 ): 2577 . https://doi.org/10.3390/electronics14132577 https://doi.org/10.3390/electronics14132577

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Relation-aware place recognition network for large-scale point clouds

Dynamic trend perception for fine-grained unmanned aerial vehicle anomaly detection in flight data

A comprehensive survey of data classification based on evidence theory

Decoupled prompt-guided mixture-of-experts dynamic distillation for multimodal recommendation

Stacked intelligent metasurface-aided multi-static sensing and cooperative communication

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰