Construction of the SAFE Database Specialized for Korean Structural Engineering and Performance Evaluation of a Retrieval Augmented Generation System Using the MAXIM Method

Seok-Jae Heo; Won-Jun Choi; Sang-Hyun Lee

doi:10.7734/COSEIK.2025.38.4.247

All Issue

2025 Vol.38, Issue 4 Preview Page Next Page

Research Paper

Construction of the SAFE Database Specialized for Korean Structural Engineering and Performance Evaluation of a Retrieval Augmented Generation System Using the MAXIM Method 한국 건축구조공학 분야 특화 SAFE 데이터베이스 구축과 MAXIM 검색 기법을 활용한 RAG 시스템 성능 평가

31 August 2025. pp. 247-255

PDF XML

Abstract

The rapid advancement of large language models (LLMs) presents both significant opportunities and notable challenges to structural‐engineering practice. To evaluate their reliability within a Korean context, we introduce safety-oriented AI framework for engineering (SAFE)—a domain-specific benchmark that integrates a 377 K-snippet knowledge base with 4,200 expert-designed questions drawn from five representative datasets (MMLU-Struct, StructQA-KO, SPED, StructMCQA, StructCase-Y/N). We used SAFE to analyze several Retrieval‑Augmented Generation (RAG) pipelines that differed in corpus structure, retriever architecture, and LLM family. The optimal configuration—a StructCorpus combined with a StructCPT retriever (MAXIM) feeding an ensemble GPT‑4o generator—achieved an overall accuracy of 89.1%, representing a 3.87-percentage‑point improvement over the strongest baseline LLM employing chain‑of‑thought prompting alone (85.2%). Performance peaked at 94.9% on the StructCase‑Y/N task, which assesses compliance with domestic building standards through practical, case‑based questions. Ablation studies revealed that retrieving approximately 32 snippets per query offers the optimal balance between accuracy and latency, whereas larger context windows yield diminishing returns. These results confirm that a well-curated structural‐engineering knowledge base, combined with a domain‑specialized retriever, significantly reduces hallucination and enhances answer fidelity. Accordingly, SAFE represents the first large‑scale evaluation framework specifically designed for Korean structural engineering and provides a reproducible platform for advancing research on safety‑critical AI decision support.

Keywords

large language models

retrieval-augmented generation

structural engineering

knowledge base

MAXIM retrieval

SAFE benchmark

본 연구는 한국 건축･구조공학 도메인에 특화된 SAFE(Safetyoriented AI Framework for Engineering) 지식베이스와 이를 활용한 검색 증강 생성(RAG) 시스템을 제안한다. SAFE는 전문용어집, 설계 기준, 교과서, 프로젝트 보고서에서 추출한 37.7만개 스니펫을 통합하여 국내 구조설계기준(KDS)과 최신 실무 사례를 포괄한다. SAFE 기반 파이프라인은 5개 대표 과업(MMLUStruct, Struct QAKO, SPED, StructMCQA, StructCaseY/N)으로 구성된 4,200문항 벤치마크에서 전체 정확도 89.1%를 기록하여, 체인오브생각(CoT) 방식의 최고 성능 LLM 대비 3.87%p 향상 효과를 나타냈다. 특히 국내 기준･실무 판정 과업인 StructCaseY/N에서 94.9%의 정확도를 달성하였다. 절편 분석 결과, 질의당 32개 스니펫을 투입할 때 정확도와 응답 지연 간 최적 균형점이 형성되며, 그 이상에서는 성능 개선 대비 비용이 급격히 감소함을 확인하였다. 또한 질문 유형별로 최적 정보 출처가 상이함을 규명하여, 도메인 맞춤형 코퍼스 가중치 조정의 필요성을 제시하였다. 본 연구는 국내 최초의 구조공학 RAG 평가 체계를 확립함으로써, 안전 중심 AI 의사결정 지원 도구의 실무 적용 가능성을 입증하고 향후 연구의 기반을 마련하였다.

키워드

대규모 언어 모델

검색 증강 생성

데이터베이스

MAXIM 검색

SAFE 벤치마크

References

Abubakar, S.M., Karimi, M.U., Mustafa, S.J., Ahmad, B. (2024) Structural Engineering Applications Using Artificial Intelligence and Machine Learning: A Review, Int. J. Advan. Nat. Sci. & Eng. Res., 8(5), pp.140~145.

Friel, R., Belyi, M., Sanyal, A. (2024) Ragbench: Explainable Benchmark for Retrieval-Augmented Generation Systems, arXiv preprint arXiv:2407.11005.

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., Wang, H. (2023) Retrieval-Augmented Generation for Large Language Models: A Survey, arXiv preprint arXiv:2312.10997.

Homolak, J. (2023) Opportunities and Risks of ChatGPT in Medicine, Science, and Academic Publishing: A Modern Promethean Dilemma, Croat. Med. J., 64(1), pp.1~3.

10.3325/cmj.2023.64.136864812PMC10028563

Hope, B., Bracey, J., Choukir, S., Warner, D. (2025) Assessment of ChatGPT for Engineering Statics Analysis, arXiv preprint arXiv:2502.00562.

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., Fung, P. (2023) Survey of Hallucination in Natural Language Generation, ACM Comput. Surv., 55(12), pp.1~38.

10.1145/3571730

Jin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W.J., Lu, Z. (2023) Medcpt: Contrastive Pre-Trained Transformers with Large-Scale Pubmed Search Logs for Zero-Shot Biomedical Information Retrieval, Bioinform., 39(11), btad651.

10.1093/bioinformatics/btad65137930897PMC10627406

Joren, H., Zhang, J., Ferng, C.S., Juan, D.C., Taly, A., Rashtchian, C. (2024) Sufficient Context: A New Lens on Retrieval Augmented Generation Systems, arXiv preprint arXiv:2411. 06037.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D. (2020) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Adv. Neural Inf. Proc. Syst. (NeurIPS), 33, arXiv:2005.11401.

Liang, H., Talebi Kalaleh, M., Mei, Q. (2025) Integrating Large Language Models for Automated Structural Analysis, arXiv preprint arXiv:2504.09754.

Naser, M.Z. (2023) Machine Learning for Civil Engineers: From Theory to Application, Wiley.

Park, M., Oh, H., Choi, E., Hwang, W. (2025) LARGE: Legal Retrieval Augmented Generation Evaluation Tool, arXiv preprint arXiv:2504.01840.

Qin, S. Z., Liao, W. J., Huang, S. N., Hu, K. G., Tan, Z., Gao, Y., Lu, X.Z. (2024) AIstructure-Copilot: Assistant for Generative AI-Driven Intelligent Design of Building Structures, Smart Constr., 1(1), pp.1~21.

10.55092/sc20240001

Sebastian, K., Huaccha, M., Rosales, B., Santa Maria, G.L., Delgadillo, R.M. (2024) Application of AI for Modelling and Structural Analysis of a Parametric 2D Frame with Voice Assistant, In E3S Web of Conferences, 586, p.02003.

10.1051/e3sconf/202458602003

Tamber, M.S., Kazi, S., Sourabh, V., Lin, J. (2025) Teaching Dense Retrieval Models to Specialize with Listwise Distillation and LLM Data Augmentation, arXiv preprint arXiv:2502.19712.

Tian, J., Hou, J., Wu, Z., Shu, P., Liu, Z., Xiang, Y., Gu, B., Filla, N., Li, Y., Liu, N., Chen, X., Tang, K., Liu, T., Wang, X. (2024) Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understanding, arXiv preprint arXiv:2401.12983.

Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., Li, J. (2024a) Prompt Engineering in Consistency and Reliability with the Evidence-based Guideline for LLMs, NPJ Digital Med., 7(1), 41.

10.1038/s41746-024-01029-438378899PMC10879172

Wang, S., Liu, J., Song, S., Cheng, J., Fu, Y., Guo, P., Fang, K., Zhu, Y., Dou, Z. (2024b) Domainrag: A Chinese Benchmark for Evaluating Domain-Specific Retrieval-Augmented Generation, arXiv preprint arXiv:2406.05654.

Wang, S., Tan, J., Dou, Z., Wen, J.R. (2024c) OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain, arXiv preprint arXiv:2412.13018.

Yu, S., Cheng, M., Yang, J., Ouyang, J. (2024) A Knowledge- Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation, arXiv preprint arXiv:2409. 13694.

Zhang, T., Patil, S. G., Jain, N., Shen, S., Zaharia, M., Stoica, I., Gonzalez, J.E. (2024a) Raft: Adapting Language Model to Domain Specific RAG, In First Conference on Language Modeling, arXiv preprint arXiv:2403.10131.

Zhang, X., Song, Y., Wang, Y., Tang, S., Li, X., Zeng, Z., Wu, Z., Ye, W., Xu, W., Zhang, Y., Dai, X., Zhang, S., Wen, Q. (2024b) Raglab: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation, arXiv preprint arXiv:2408.11381.

10.18653/v1/2024.emnlp-demo.43

Information

Publisher :Computational Structural Engineering Institute of Korea
Publisher(Ko) :한국전산구조공학회
Journal Title :Journal of the Computational Structural Engineering Institute of Korea
Journal Title(Ko) :한국전산구조공학회 논문집
Volume : 38
No :4
Pages :247-255
Received Date : 2025-04-21
Revised Date : 2025-05-12
Accepted Date : 2025-05-20
DOI :https://doi.org/10.7734/COSEIK.2025.38.4.247

Journal of the Computational Structural Engineering Institute of Korea ISSN:1229-3059(Print) 2287-2302(Online) 한국전산구조공학회 논문집

All Issue