diff --git a/docs/design/706-smilegate-qa-history/README.md b/docs/design/706-smilegate-qa-history/README.md
new file mode 100644
index 0000000..2fd8104
--- /dev/null
+++ b/docs/design/706-smilegate-qa-history/README.md
@@ -0,0 +1,70 @@
+# 설계서: 스마일게이트 고객 질답 검증 이력
+
+## 추적성
+
+- Redmine: #706 `[Smilegate] 고객 엑셀 질답 검증 이력 및 실행 화면`
+- 기준 질답서: `/Users/joungminko/claude-workspace/oci-data-flow-aidp/docs/reports/sgmp-select-ai-full-qa-term-dict-final-v2-20260721.md`
+- 기준 데이터: 표준 DW 샘플 28건 + 카제나 샘플 19건 = 47건
+- 대상 스키마: `SGMP_POC`
+- 대상 화면: `poc4_active_source_20260714/apps/poc4/mcp_discovery_ui.py`
+
+## 프로젝트 개요
+
+`vpd-permission-poc`은 Oracle Autonomous Database의 게임 데이터와 Select AI/MCP를 연결해 자연어 데이터 질의를 검증하는 PoC다. 이번 기능은 고객이 제공한 Excel 기반 질답서를 실행 가능한 기준 시나리오로 바꾸고, 데모 중 실제 답변 품질을 설명 가능하게 남긴다.
+
+## 목표
+
+1. 고객 Excel에서 정리한 47개 질문을 질문 마스터로 보관한다.
+2. 기준 답변, 기준 SQL, 과거 검증 결과와 이후 실행 결과를 모두 순차 이력으로 보관한다.
+3. 사용자가 후보 테이블에서 질문을 고르거나 자유 질의를 입력해 즉시 실행할 수 있게 한다.
+4. 후보 질문은 SQL 의미 검증과 실행 결과로 `PASS`, `WARN`, `FAIL`을 표시한다. 정답 기준이 없는 자유 질의는 `REVIEW`로 표시한다.
+
+## 데이터 모델
+
+테이블은 사용자 요청에 따라 두 개만 둔다.
+
+| 테이블 | 키 | 역할 |
+| --- | --- | --- |
+| `SG_AI_QA_QUESTION` | `QUESTION_ID` | 고객 Excel 질문, 출처, 기대 포인트, 원본 샘플 SQL, 기준 SQL/답변, SQL 판정 규칙을 보관한다. 자유 질의도 해시 기준으로 이 테이블에 한 번만 등록한다. |
+| `SG_AI_QA_ANSWER` | `ANSWER_SEQ` | 질문별 실행 이력이다. 과거 47건도 `HISTORICAL`로 적재하고, 포털 실행은 `LIVE`로 계속 추가한다. |
+
+`SG_AI_QA_ANSWER.QUESTION_ID`는 질문 마스터를 참조한다. 실행 결과는 JSON, 생성 SQL·답변·판정 근거는 CLOB으로 저장한다. 따라서 질문 기준은 바뀌어도 이미 실행된 이력의 원문과 당시 판정을 보존한다.
+
+## 판정 규칙
+
+1. 기준 시나리오는 `required_sql_terms`와 `recommended_sql_terms`를 사용한다.
+2. 필수 테이블·컬럼·집계·기간 규칙이 빠지거나 모델 오류 문구가 SQL에 섞이면 `FAIL`이다.
+3. 권장 필터가 빠졌거나 지원 범위가 일부인 경우 `WARN`이다.
+4. 미지원 게임 질문은 별칭 조회를 거치지 않고 임의 게임 ID나 테이블을 만들어 내면 `FAIL`이다. 안전하게 거절하거나 별칭 조회 결과가 0건이면 `PASS`이다.
+5. 월간 NRU/AU, 재화 보유/사용 등 기존 질답서의 개별 보정 규칙은 같은 판정기에 반영한다.
+6. 자유 질의는 기준 질문을 선택하지 않은 경우 `REVIEW`로 저장한다. 실행 성공을 정답으로 표시하지 않는다.
+
+문장 표현의 유사도만으로 정답을 판정하지 않는다. 집계값, 생성 SQL, 실행 결과가 근거가 되므로 고객에게 왜 통과 또는 실패인지 보여줄 수 있다.
+
+## 화면 흐름
+
+1. `검증 시나리오` 탭에서 47개 후보를 표 형태로 표시한다. 케이스, 구분, 제목, 질문, 기대 포인트, 최근 판정, 최근 실행 시각을 보여 준다.
+2. 행을 선택하면 질문 입력란이 채워지고, 우측 또는 하단에 기준 답변·기준 SQL·원본 Excel 출처를 표시한다.
+3. 사용자는 선택된 기준 질문을 그대로 실행하거나 자유 텍스트를 작성한다.
+4. 실행 뒤에는 현재 답변, 생성 SQL, 조회 행, 판정, 판정 근거를 표시하고 `SG_AI_QA_ANSWER`에 저장한다.
+5. 같은 질문의 과거 답변은 최신 순 표로 보여 주며, 과거 기준 검증과 현재 실행을 구분한다.
+
+## 적재 기준
+
+- 기준 원본은 `sgmp-select-ai-full-qa-term-dict-final-v2-20260721.md`와 동시 생성된 JSON이다.
+- JSON의 `STD-05` 실행 출력은 비정상적으로 크므로, 이력 조회 안정성을 위해 저장 시 안전한 길이로 절단하고 원본 보고서 경로를 질문에 남긴다.
+- 과거 레코드는 `HISTORICAL`, 포털에서 수행하는 새 레코드는 `LIVE`로 구분한다.
+
+## 완료 기준
+
+- ADB에 질문 마스터 47건과 과거 답변 이력 47건이 있다.
+- 답변 이력 키는 증가하는 `ANSWER_SEQ`이며 질문 외래키가 유효하다.
+- 후보 선택, 자유 질의, 기준 답변/SQL, 과거 이력, PASS/WARN/FAIL/REVIEW 표기가 한 화면에서 작동한다.
+- 생성 SQL의 핵심 규칙을 바꾼 실패 케이스가 `FAIL`로 판정되는 단위 테스트가 있다.
+- 실제 포털 실행 한 건이 ADB 이력에 저장되는 것을 확인한다.
+
+## 비범위
+
+- 이 기능은 Select AI의 정답을 하드코딩해 바꾸지 않는다.
+- 과거 대화 SQLite 저장소를 이번 작업에서 전면 이전하지 않는다. 고객 질답 검증 이력만 ADB의 두 테이블에 저장한다.
+- 자유 질의에 임의의 정답을 부여하지 않는다.
diff --git a/poc4_active_source_20260714/apps/poc4/mcp_discovery_ui.py b/poc4_active_source_20260714/apps/poc4/mcp_discovery_ui.py
index cfc78c1..4f8de74 100644
--- a/poc4_active_source_20260714/apps/poc4/mcp_discovery_ui.py
+++ b/poc4_active_source_20260714/apps/poc4/mcp_discovery_ui.py
@@ -57,7 +57,8 @@ from src.agent_console.presentation import (
render_login_brand,
)
from src.agent_console.profile import AppProfile, AppProfileError, load_app_profile
-from src.poc4.scenarios import ScenarioConfigError, load_demo_scenarios
+from src.poc4.qa_history import QaJudgment, QaQuestion, evaluate_sql, load_benchmark_questions
+from src.poc4.qa_history_store import QaHistoryStore, QaHistoryStoreError
LOG = logging.getLogger(__name__)
@@ -77,7 +78,7 @@ DEFAULT_QUERY_MODEL_PROFILE = "gpt54_mini_oci"
ENV_FILE = ROOT / ".env"
MCP_SERVERS_FILE = ROOT / "config" / "mcp_servers.json"
VPD_TOKEN_PRESETS_FILE = ROOT / "config" / "vpd_token_presets.json"
-DEMO_SCENARIOS_FILE = ROOT / "config" / "smilegate_demo_scenarios.json"
+QA_BENCHMARK_FILE = ROOT / "config" / "smilegate_qa_benchmark.json"
APP_PROFILE_FILE = ROOT / "config" / "app_profile.json"
CHAT_DB_FILE = ROOT / "data" / "poc4_mcp_chat.sqlite3"
DEFAULT_VPD_USER_ID = "1001"
@@ -1889,12 +1890,180 @@ def new_conversation_id() -> str:
def _question_label(question: object) -> str:
- question_id = str(getattr(question, "question_id"))
- category = str(getattr(question, "category"))
- title = str(getattr(question, "title", getattr(question, "text")))
+ question_id = str(
+ getattr(question, "question_code", getattr(question, "question_id", ""))
+ )
+ category = str(getattr(question, "category", ""))
+ title = str(
+ getattr(question, "title", getattr(question, "question_text", ""))
+ )
return f"{question_id} · {category} · {title}"
+@st.cache_resource(show_spinner=False)
+def _qa_history_store() -> QaHistoryStore:
+ return QaHistoryStore()
+
+
+def _load_qa_questions() -> tuple[list[QaQuestion], QaHistoryStore | None, str]:
+ try:
+ store = _qa_history_store()
+ questions = store.list_questions(limit=200)
+ if questions:
+ return questions, store, ""
+ return [], store, "질답 이력 DB에 아직 적재된 기준 질문이 없습니다."
+ except QaHistoryStoreError as exc:
+ try:
+ fallback = list(load_benchmark_questions(QA_BENCHMARK_FILE))
+ except Exception:
+ fallback = []
+ return fallback, None, str(exc)
+
+
+def _judgment_label(status: str) -> str:
+ labels = {
+ "PASS": "통과",
+ "WARN": "검토",
+ "FAIL": "실패",
+ "REVIEW": "수동 검토",
+ }
+ return labels.get(str(status or "").upper(), "미실행")
+
+
+def _qa_answer_preview(value: object, limit: int = 180) -> str:
+ text = str(value or "").strip().replace("\n", " ")
+ return text if len(text) <= limit else text[:limit] + "..."
+
+
+def _render_qa_benchmark_panel(
+ questions: list[QaQuestion],
+ store: QaHistoryStore | None,
+ store_error: str,
+) -> QaQuestion | None:
+ st.markdown(
+ ''
+ "고객 질답 검증 시나리오"
+ "
",
+ unsafe_allow_html=True,
+ )
+ st.caption(
+ "고객 제공 Excel에서 정리한 47개 후보입니다. 후보를 선택하면 기준 답변과 과거 검증 이력을 확인하고 바로 실행할 수 있습니다."
+ )
+ if store_error:
+ st.warning(
+ "현재는 파일 기준 후보만 표시합니다. ADB 이력 저장은 연결 복구 후 사용할 수 있습니다. "
+ + store_error
+ )
+ if not questions:
+ st.error("표시할 고객 질답 기준이 없습니다.")
+ return None
+
+ category_options = tuple(sorted({question.category for question in questions}))
+ category = st.selectbox(
+ "시나리오 구분",
+ options=("ALL", *category_options),
+ format_func=lambda item: "전체 47개" if item == "ALL" else item,
+ key="poc4_qa_category_filter",
+ )
+ visible_questions = [
+ question for question in questions if category == "ALL" or question.category == category
+ ]
+ st.dataframe(
+ [
+ {
+ "케이스": question.question_code,
+ "구분": question.category,
+ "제목": question.title,
+ "질문": question.question_text,
+ "기대 기준": _qa_answer_preview(question.expected_focus, 150),
+ "최근 판정": _judgment_label(question.last_judgment_status),
+ "최근 실행": question.last_evaluated_at or "과거 기준",
+ }
+ for question in visible_questions
+ ],
+ hide_index=True,
+ width="stretch",
+ height=min(520, 110 + 36 * len(visible_questions)),
+ column_config={
+ "질문": st.column_config.TextColumn(width="large"),
+ "기대 기준": st.column_config.TextColumn(width="large"),
+ "최근 실행": st.column_config.TextColumn(width="medium"),
+ },
+ )
+ by_code = {question.question_code: question for question in visible_questions}
+ selected_code = st.selectbox(
+ "실행할 검증 후보 선택",
+ options=("", *by_code),
+ format_func=lambda code: "선택 안 함 · 자유 질의" if not code else _question_label(by_code[code]),
+ key="poc4_qa_question_code",
+ )
+ selected = by_code.get(selected_code)
+ if selected is None:
+ st.info("자유 텍스트 질문도 실행하고 이력으로 남길 수 있습니다. 자유 질의는 정답 기준이 없어 수동 검토로 표시됩니다.")
+ return None
+
+ with st.expander("선택한 질문의 정답 기준·원본·과거 답변", expanded=True):
+ left, right = st.columns(2)
+ with left:
+ st.markdown("**정답 기준**")
+ st.write(selected.expected_focus or "정답 기준 설명이 없습니다.")
+ st.markdown("**기준 답변**")
+ st.code(selected.baseline_answer or "기준 실행 결과가 없습니다.", language=None)
+ with right:
+ st.markdown("**고객 원본 출처**")
+ st.write(
+ f"{selected.source_document} / {selected.source_sheet} / 행 {selected.source_row or '-'}"
+ )
+ if selected.source_scenario:
+ st.caption(selected.source_scenario)
+ st.markdown("**기준 SQL**")
+ st.code(selected.baseline_sql or selected.sample_sql or "기준 SQL이 없습니다.", language="sql")
+
+ if store is not None and selected.question_id is not None:
+ try:
+ answers = store.list_answers(selected.question_id, limit=30)
+ except QaHistoryStoreError as exc:
+ st.warning(str(exc))
+ answers = []
+ if answers:
+ st.markdown("**과거 답변 이력**")
+ st.dataframe(
+ [
+ {
+ "순번": item["answer_seq"],
+ "구분": "과거 기준" if item["answer_kind"] == "HISTORICAL" else "이번 실행",
+ "판정": _judgment_label(item["judgment_status"]),
+ "실행 상태": item["execution_status"],
+ "실행 시각": item["requested_at"],
+ "답변": _qa_answer_preview(item["answer_text"]),
+ }
+ for item in answers
+ ],
+ hide_index=True,
+ width="stretch",
+ )
+ answer_by_seq = {str(item["answer_seq"]): item for item in answers}
+ detail_seq = st.selectbox(
+ "답변 이력 상세",
+ options=tuple(answer_by_seq),
+ format_func=lambda value: (
+ f"#{value} · {_judgment_label(answer_by_seq[value]['judgment_status'])} · "
+ f"{answer_by_seq[value]['requested_at']}"
+ ),
+ key=f"poc4_qa_answer_detail_{selected.question_id}",
+ )
+ detail = answer_by_seq[detail_seq]
+ st.markdown("**판정 근거**")
+ st.write(detail["judgment_reason"] or "판정 근거가 없습니다.")
+ st.markdown("**생성 SQL**")
+ st.code(detail["generated_sql"] or "생성 SQL이 없습니다.", language="sql")
+ st.markdown("**실행 답변**")
+ st.write(detail["answer_text"] or "답변이 없습니다.")
+ else:
+ st.info("아직 저장된 답변 이력이 없습니다.")
+ return selected
+
+
def _apply_console_theme(profile: AppProfile) -> None:
apply_console_theme(st, profile)
@@ -6334,6 +6503,89 @@ def _render_vpd_operations_tab() -> None:
)
+def _select_ai_execution_summary(payload: Any) -> dict[str, Any]:
+ if not isinstance(payload, Mapping):
+ return {
+ "generated_sql": "",
+ "execution_status": "UNKNOWN",
+ "execution_succeeded": False,
+ "result": {},
+ }
+ generated_sql = str(
+ payload.get("generatedSql") or payload.get("generated_sql") or ""
+ ).strip()
+ status = str(payload.get("status") or "").strip().upper()
+ execution = str(payload.get("execution") or "").strip().upper()
+ execution_succeeded = (
+ status == "SHOWSQL_AND_EXECUTED" and execution == "READ_ONLY_EXECUTED"
+ )
+ result = {
+ key: payload.get(key)
+ for key in (
+ "status",
+ "execution",
+ "rowCount",
+ "truncated",
+ "columns",
+ "items",
+ "generatedSql",
+ )
+ if key in payload
+ }
+ return {
+ "generated_sql": generated_sql,
+ "execution_status": status or execution or "UNKNOWN",
+ "execution_succeeded": execution_succeeded,
+ "result": result,
+ }
+
+
+def _record_qa_history(
+ *,
+ store: QaHistoryStore | None,
+ benchmark_question: QaQuestion | None,
+ question_text: str,
+ conversation_id: str,
+ requested_by: str,
+ model_profile: str,
+ answer_text: str,
+ mcp_result: Any,
+ duration_ms: int,
+) -> tuple[QaQuestion | None, QaJudgment | None]:
+ if store is None:
+ return benchmark_question, None
+ try:
+ history_question = benchmark_question or store.find_or_create_free_text_question(
+ question_text
+ )
+ execution = _select_ai_execution_summary(mcp_result)
+ judgment = evaluate_sql(
+ benchmark_question,
+ execution["generated_sql"],
+ execution_succeeded=bool(execution["execution_succeeded"]),
+ error_text=_bounded_json(mcp_result, max_chars=8000),
+ )
+ store.record_answer(
+ question_id=int(history_question.question_id or 0),
+ answer_kind="LIVE",
+ conversation_id=conversation_id,
+ requested_by=requested_by,
+ model_profile=model_profile,
+ generated_sql=execution["generated_sql"],
+ answer_text=answer_text,
+ result=execution["result"],
+ execution_output=_bounded_json(mcp_result, max_chars=24_000),
+ execution_status=str(execution["execution_status"]),
+ judgment_status=judgment.status,
+ judgment_reason=judgment.reason,
+ duration_ms=duration_ms,
+ )
+ return history_question, judgment
+ except QaHistoryStoreError as exc:
+ st.warning(f"질답 이력 저장에 실패했습니다: {exc}")
+ return benchmark_question, None
+
+
def _process_submitted_question(
*,
question: str,
@@ -6349,6 +6601,8 @@ def _process_submitted_question(
limit: int,
selected_token_preset: VpdTokenPreset | None,
execution_mode_override: str,
+ benchmark_question: QaQuestion | None,
+ qa_history_store: QaHistoryStore | None,
) -> None:
normalized_question = question.strip()
if not normalized_question:
@@ -6891,6 +7145,32 @@ def _process_submitted_question(
],
"discovery_failures": discovery_failures,
}
+ history_question, history_judgment = _record_qa_history(
+ store=qa_history_store,
+ benchmark_question=benchmark_question,
+ question_text=normalized_question,
+ conversation_id=conversation_id,
+ requested_by=(
+ selected_token_preset.user_id
+ if selected_token_preset is not None
+ else "portal-user"
+ ),
+ model_profile=answer_model_profile,
+ answer_text=str(assistant_message["content"]),
+ mcp_result=answer_source,
+ duration_ms=round((perf_counter() - process_started) * 1000),
+ )
+ if history_judgment is not None:
+ assistant_message["details"]["qa_history"] = {
+ "question_id": history_question.question_id if history_question else None,
+ "question_code": history_question.question_code if history_question else "",
+ "judgment_status": history_judgment.status,
+ "judgment_reason": history_judgment.reason,
+ }
+ st.info(
+ "질답 검증 판정: "
+ f"{_judgment_label(history_judgment.status)} · {history_judgment.reason}"
+ )
refresh_progress(98, "질의 결과를 저장하고 있습니다.")
save_chat_turn(
conversation_id=conversation_id,
@@ -6945,27 +7225,13 @@ def main() -> None:
if not st.session_state.get(PORTAL_AUTHENTICATED_KEY, False):
_render_portal_login(profile)
return
- try:
- questions = load_demo_scenarios(DEMO_SCENARIOS_FILE)
- except ScenarioConfigError as exc:
- st.error(str(exc))
- return
- scenario_key = "poc4_mcp_discovery_scenario"
+ qa_questions, qa_history_store, qa_store_error = _load_qa_questions()
question_text_key = "poc4_mcp_discovery_question_text"
- loaded_scenario_key = "poc4_mcp_discovery_loaded_scenario_id"
+ loaded_qa_question_key = "poc4_mcp_discovery_loaded_qa_question_code"
conversation_id_key = "poc4_mcp_discovery_conversation_id"
chat_page_key = "poc4_mcp_discovery_page"
query_progress_notice_key = "poc4_query_progress_notice"
mcp_cache_generation_key = "poc4_mcp_tools_cache_generation"
- selected_scenario_state = st.session_state.get(scenario_key)
- scenario_ids = {question.question_id for question in questions}
- if str(getattr(selected_scenario_state, "question_id", "")).strip().upper() not in (
- "",
- *scenario_ids,
- ):
- st.session_state.pop(scenario_key, None)
- st.session_state.pop(loaded_scenario_key, None)
- st.session_state[question_text_key] = DEFAULT_QUESTION
init_chat_store()
if conversation_id_key not in st.session_state:
st.session_state[conversation_id_key] = new_conversation_id()
@@ -7254,7 +7520,7 @@ def main() -> None:
except (OSError, UnicodeError, ValueError):
st.warning("MCP 설정 JSON을 읽지 못했습니다.")
st.caption(f"config: {MCP_SERVERS_FILE}")
- st.caption(f"scenario config: {DEMO_SCENARIOS_FILE}")
+ st.caption(f"QA benchmark: {QA_BENCHMARK_FILE}")
st.caption(f"token presets: {VPD_TOKEN_PRESETS_FILE}")
st.caption(f"selected LLM model: {selected_query_model_profile}")
st.caption(f"default model: {default_query_model_profile}")
@@ -7272,35 +7538,36 @@ def main() -> None:
_render_architecture_tab()
with scenario_tab:
- st.markdown(
- ''
- "질문 입력"
- "
",
- unsafe_allow_html=True,
- )
- selected_scenario = st.selectbox(
- "업무 데모 질의 샘플",
- options=(None, *questions),
- format_func=lambda item: (
- "선택 안 함 · 직접 질문" if item is None else _question_label(item)
- ),
- key=scenario_key,
- )
- selected_scenario_id = (
- None
- if selected_scenario is None
- else str(getattr(selected_scenario, "question_id"))
+ selected_qa_question = _render_qa_benchmark_panel(
+ qa_questions,
+ qa_history_store,
+ qa_store_error,
)
if question_text_key not in st.session_state:
st.session_state[question_text_key] = DEFAULT_QUESTION
- if st.session_state.get(loaded_scenario_key, "") != str(
- selected_scenario_id or ""
- ):
- if selected_scenario is not None:
- st.session_state[question_text_key] = str(
- getattr(selected_scenario, "text")
- )
- st.session_state[loaded_scenario_key] = str(selected_scenario_id or "")
+ selected_qa_code = (
+ selected_qa_question.question_code if selected_qa_question is not None else ""
+ )
+ if st.session_state.get(loaded_qa_question_key, "") != selected_qa_code:
+ st.session_state[question_text_key] = (
+ selected_qa_question.question_text
+ if selected_qa_question is not None
+ else DEFAULT_QUESTION
+ )
+ st.session_state[loaded_qa_question_key] = selected_qa_code
+
+ st.markdown(
+ ''
+ "질문 실행"
+ "
",
+ unsafe_allow_html=True,
+ )
+ if selected_qa_question is not None:
+ st.caption(
+ "선택한 후보 문장을 수정하지 않고 실행하면 자동 판정합니다. 문장을 수정하면 자유 질의로 저장되어 수동 검토 대상이 됩니다."
+ )
+ else:
+ st.caption("자유 질의는 실행 이력으로 남기며, 고객 기준 정답과 자동 비교하지 않습니다.")
with st.form("poc4_mcp_question_form"):
question = st.text_area(
@@ -7309,7 +7576,7 @@ def main() -> None:
key=question_text_key,
height=120,
max_chars=1_000,
- placeholder="업무 질문을 직접 입력하거나 위 질의 샘플을 선택하세요.",
+ placeholder="고객 검증 후보를 선택하거나 업무 질문을 직접 입력하세요.",
)
submitted = st.form_submit_button(
"질문 전송",
@@ -7386,6 +7653,13 @@ def main() -> None:
limit=int(limit),
selected_token_preset=selected_token_preset,
execution_mode_override=execution_mode_override,
+ benchmark_question=(
+ selected_qa_question
+ if selected_qa_question is not None
+ and question.strip() == selected_qa_question.question_text.strip()
+ else None
+ ),
+ qa_history_store=qa_history_store,
)
diff --git a/poc4_active_source_20260714/config/smilegate_qa_benchmark.json b/poc4_active_source_20260714/config/smilegate_qa_benchmark.json
new file mode 100644
index 0000000..0f967e8
--- /dev/null
+++ b/poc4_active_source_20260714/config/smilegate_qa_benchmark.json
@@ -0,0 +1,1832 @@
+{
+ "version": 1,
+ "source_report": "/Users/joungminko/claude-workspace/oci-data-flow-aidp/docs/reports/sgmp-select-ai-full-qa-term-dict-final-v2-20260721.json",
+ "source_redmine": 690,
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "sample_base_date": "2026-07-15",
+ "sample_game_id": "STOVE_CHAOSZERO",
+ "scenarios": [
+ {
+ "case_id": "STD-01",
+ "title": "%GAME% 테이블명 바인딩",
+ "category": "STANDARD",
+ "scenario_no": 1,
+ "question": "버블리즈 유저 마스터 조회해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 4,
+ "scenario": "표준 테이블 사용\n%GAME% 변수 → 테이블명 바인딩",
+ "natural_question": "버블리즈 유저 마스터 조회해줘",
+ "sample_query": "SELECT *\n FROM DW.BBZ_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)",
+ "note": "게임명 & 테이블 PREPIX 매핑 테이블 생성 필요\n기본 조회시 row 제한 프롬프트 적용 필요\n파티션 기본 조회 조건 추가"
+ },
+ "expected_focus": "BBZ_COMN_USER_MST는 현재 SGMP_POC object list에 없으므로 임의 테이블 생성 없이 거절/실패해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n a.\"GAME_ID\",\n a.\"GAME_NM\",\n a.\"GAME_PREFIX\"\nFROM\n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\nWHERE\n a.\"USE_YN\" = 'Y'\n AND UPPER(a.\"GAME_NM\") LIKE UPPER('%버블리즈%')\n OR a.\"USE_YN\" = 'Y'\n AND UPPER(a.\"GAME_ALIAS_NM\") LIKE UPPER('%버블리즈%')\n OR a.\"USE_YN\" = 'Y'\n AND UPPER(a.\"GAME_PREFIX\") LIKE UPPER('%버블리즈%')\n OR a.\"USE_YN\" = 'Y'\n AND UPPER(a.\"GAME_ID\") LIKE UPPER('%버블리즈%')",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 50212
+ }
+ },
+ {
+ "case_id": "STD-02",
+ "title": "한글 게임명 AU",
+ "category": "STANDARD",
+ "scenario_no": 2,
+ "question": "버블리즈 AU 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 5,
+ "scenario": "표준 테이블 사용\n%게임% 한글 변수 인식",
+ "natural_question": "버블리즈 AU 알려줘",
+ "sample_query": "SELECT COUNT(1)\n FROM DW.BBZ_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND EXPT_USER_YN = 'N'\n AND AU_FLAG = 1",
+ "note": "AU 라고만 하면 표준 AU를 제공할 수 있도록 프롬프트 제공\n유저 기준 사용시 EXPT_USER_YN = 'N' 조건 사용할 수 있도록 프롬프트 제공\n날짜 제공하지 않을시 기본적으로 기준일(어제) 기준으로 제공할 수 있도록 프롬프트 제공\n날짜 변수를 어떻게 받아오는지 확인 필요"
+ },
+ "expected_focus": "버블리즈는 현재 object list에 없으므로 임의 game_id/table 사용 없이 거절/실패해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = (SELECT MAX(\"BASE_DT\") FROM \"SGMP_POC\".\"CZN_COMN_USER_MST\")\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%버블리즈%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%버블리즈%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%버블리즈%')\n OR UPPER(g.\"GAME_ID\") LIKE UPPER('%버블리즈%')\n )\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "AU_COUNT=0",
+ "execution_output": "AU_COUNT\n----------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 48811
+ }
+ },
+ {
+ "case_id": "STD-03",
+ "title": "게임명 미입력",
+ "category": "STANDARD",
+ "scenario_no": 3,
+ "question": "공통유저마스터에서 유저 수 알려줘. 게임명은 언급하지 않았어.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 6,
+ "scenario": "표준 테이블 사용\n변수 미입력 시 처리",
+ "natural_question": "공통유저마스터에서 유저 수 알려줘\n(게임명 미언급)",
+ "sample_query": "별도 샘플 쿼리 없음",
+ "note": "TEXT2SQL 공간 생성에 따라 다름\n- 전체 게임 조회 가능할 시: 어떤 게임 확인이 필요한지 역질문 필요\n- 특정 게임만 조회 가능할 시: 해당 게임에 대한 답변 필요"
+ },
+ "expected_focus": "게임 특정이 불가능하면 임의 게임 선택 없이 clarification 또는 제한적 SQL이어야 한다.",
+ "support_level": "PARTIAL",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS USER_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"STD_USER_YN\" = 'Y'\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "USER_COUNT=10",
+ "execution_output": "USER_COUNT\n----------\n 10",
+ "execution_status": "PASS",
+ "judgment_status": "WARN",
+ "judgment_reason": "Required SQL terms are present.\nPartial scenario contains unsupported scope; generated SQL should be reviewed before demo use.",
+ "duration_ms": 47670
+ }
+ },
+ {
+ "case_id": "STD-04",
+ "title": "영문 게임명 BUBBLYZ",
+ "category": "STANDARD",
+ "scenario_no": 4,
+ "question": "BUBBLYZ 유저 마스터 조회해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 7,
+ "scenario": "한글/영문 게임명 혼용",
+ "natural_question": "BUBBLYZ 유저 마스터 조회해줘",
+ "sample_query": "SELECT *\n FROM DW.BBZ_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)",
+ "note": ""
+ },
+ "expected_focus": "BUBBLYZ는 현재 object list에 없으므로 임의 테이블 생성 없이 거절/실패해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n a.\"GAME_ID\",\n a.\"GAME_NM\",\n a.\"GAME_PREFIX\"\nFROM\n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\nWHERE\n a.\"USE_YN\" = 'Y'\n AND (\n UPPER(a.\"GAME_NM\") LIKE UPPER('%BUBBLYZ%')\n OR UPPER(a.\"GAME_ALIAS_NM\") LIKE UPPER('%BUBBLYZ%')\n OR UPPER(a.\"GAME_PREFIX\") LIKE UPPER('%BUBBLYZ%')\n OR UPPER(a.\"GAME_ID\") LIKE UPPER('%BUBBLYZ%')\n )",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 48491
+ }
+ },
+ {
+ "case_id": "STD-05",
+ "title": "게임 약어/FULL NAME",
+ "category": "STANDARD",
+ "scenario_no": 5,
+ "question": "로나 유저 마스터 조회해줘. 그리고 카오스 제로 나이트메어 유저 마스터도 조회해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 8,
+ "scenario": "게임 약어 / FULL NAME 사용",
+ "natural_question": "1. 로나 유저 마스터 조회해줘\n2. 카오스 제로 나이트메어 유저 마스터 조회해줘",
+ "sample_query": "SELECT *\n FROM DW.LORDNINE_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n\nSELECT *\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)",
+ "note": ""
+ },
+ "expected_focus": "로드나인은 미지원, 카오스 제로 나이트메어는 CZN_COMN_USER_MST로 처리 가능하다.",
+ "support_level": "PARTIAL",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n a.\"GUID\",\n a.\"AUID\",\n a.\"BASE_DT\",\n a.\"SNAPSHOT_DT\",\n a.\"STD_USER_YN\",\n a.\"EXPT_USER_YN\",\n a.\"AU_FLAG\",\n a.\"NRU_FLAG\",\n a.\"RU_FLAG\",\n a.\"ACM_CONN_DCNT\",\n a.\"CONTINUE_CONN_DCNT\",\n a.\"CONTINUE_NON_CONN_DCNT\",\n a.\"ACM_PAYMT_AMT\",\n a.\"TDAY_PAYMT_AMT\",\n a.\"TDAY_PLAY_TIME\",\n a.\"MAX_LEVEL\",\n a.\"LAST_CONN_DT\",\n a.\"LAST_CONN_COUNTRY_CD\",\n a.\"USER_TYPE_ITG_CD\"\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" a\nWHERE\n EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%로나%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%로나%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%로나%')\n OR UPPER(g.\"GAME_ID\") LIKE UPPER('%로나%')\n )\n AND g.\"GAME_PREFIX\" = 'CZN'\n )\nUNION ALL\nSELECT\n b.\"GUID\",\n b.\"AUID\",\n b.\"BASE_DT\",\n b.\"SNAPSHOT_DT\",\n b.\"STD_USER_YN\",\n b.\"EXPT_USER_YN\",\n b.\"AU_FLAG\",\n b.\"NRU_FLAG\",\n b.\"RU_FLAG\",\n b.\"ACM_CONN_DCNT\",\n b.\"CONTINUE_CONN_DCNT\",\n b.\"CONTINUE_NON_CONN_DCNT\",\n b.\"ACM_PAYMT_AMT\",\n b.\"TDAY_PAYMT_AMT\",\n b.\"TDAY_PLAY_TIME\",\n b.\"MAX_LEVEL\",\n b.\"LAST_CONN_DT\",\n b.\"LAST_CONN_COUNTRY_CD\",\n b.\"USER_TYPE_ITG_CD\"\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" b\nWHERE\n EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%카오스 제로 나이트메어%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카오스 제로 나이트메어%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%카오스 제로 나이트메어%')\n OR UPPER(g.\"GAME_ID\") LIKE UPPER('%카오스 제로 나이트메어%')\n )\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "GUID=AUID",
+ "execution_output": "GUID\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n...[historical output truncated]",
+ "execution_status": "PASS",
+ "judgment_status": "WARN",
+ "judgment_reason": "Required SQL terms are present.\nPartial scenario contains unsupported scope; generated SQL should be reviewed before demo use.",
+ "duration_ms": 62484
+ }
+ },
+ {
+ "case_id": "STD-06",
+ "title": "복수 게임 AU",
+ "category": "STANDARD",
+ "scenario_no": 6,
+ "question": "로드나인이랑 카제나의 2026년 7월 15일 AU를 각각 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 9,
+ "scenario": "복수 게임 변수 동시 처리",
+ "natural_question": "로드나인이랑 카제나 AU 각각 알려줘",
+ "sample_query": "SELECT COUNT(1)\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND AU_FLAG = 1\n AND EXPT_USER_YN = 'N'\n\nSELECT COUNT(1)\n FROM DW.LORDNINE_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND AU_FLAG = 1\n AND EXPT_USER_YN = 'N'",
+ "note": ""
+ },
+ "expected_focus": "로드나인은 미지원, 카제나는 CZN_COMN_USER_MST AU_FLAG 집계 가능하다.",
+ "support_level": "PARTIAL",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n g1.\"GAME_NM\" AS GAME_NAME_1,\n COUNT(DISTINCT u1.\"GUID\") AS AU_COUNT_1,\n g2.\"GAME_NM\" AS GAME_NAME_2,\n COUNT(DISTINCT u2.\"GUID\") AS AU_COUNT_2\nFROM\n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g1,\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u1,\n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g2,\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u2\nWHERE\n g1.\"USE_YN\" = 'Y'\n AND UPPER(g1.\"GAME_NM\") LIKE UPPER('%로드나인%')\n AND u1.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u1.\"AU_FLAG\" = 1\n AND u1.\"EXPT_USER_YN\" = 'N'\n AND g2.\"USE_YN\" = 'Y'\n AND UPPER(g2.\"GAME_NM\") LIKE UPPER('%카제나%')\n AND u2.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u2.\"AU_FLAG\" = 1\n AND u2.\"EXPT_USER_YN\" = 'N'\nGROUP BY\n g1.\"GAME_NM\",\n g2.\"GAME_NM\"",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "WARN",
+ "judgment_reason": "Required SQL terms are present.\nPartial scenario contains unsupported scope; generated SQL should be reviewed before demo use.",
+ "duration_ms": 49075
+ }
+ },
+ {
+ "case_id": "STD-07",
+ "title": "존재하지 않는 게임명",
+ "category": "STANDARD",
+ "scenario_no": 7,
+ "question": "테스트게임 유저마스터 조회해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 10,
+ "scenario": "존재하지 않는 게임명 입력",
+ "natural_question": "테스트게임 유저마스터 조회해줘",
+ "sample_query": "별도 샘플 쿼리 없음",
+ "note": ""
+ },
+ "expected_focus": "미등록 게임명은 임의 테이블명 생성 없이 거절/실패해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n a.\"GAME_ID\",\n a.\"GAME_NM\",\n a.\"GAME_PREFIX\"\nFROM\n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\nWHERE\n a.\"USE_YN\" = 'Y'\n AND UPPER(a.\"GAME_NM\") LIKE UPPER('%테스트게임%')\n OR (\n a.\"USE_YN\" = 'Y'\n AND UPPER(a.\"GAME_ALIAS_NM\") LIKE UPPER('%테스트게임%')\n )\n OR (\n a.\"USE_YN\" = 'Y'\n AND UPPER(a.\"GAME_PREFIX\") LIKE UPPER('%테스트게임%')\n )",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 48526
+ }
+ },
+ {
+ "case_id": "STD-08",
+ "title": "로드나인/로드나인 아시아 구분",
+ "category": "STANDARD",
+ "scenario_no": 8,
+ "question": "로드나인 유저 마스터 조회해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 11,
+ "scenario": "로드나인 & 로드나인 아시아 잘 구분할 수 있는지",
+ "natural_question": "로드나인 유저 마스터 조회해줘",
+ "sample_query": "SELECT *\n FROM DW.LORDNINE_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)",
+ "note": ""
+ },
+ "expected_focus": "로드나인 계열 테이블은 현재 object list에 없으므로 거절/실패해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n g.GUID,\n g.AUID,\n g.BASE_DT,\n g.STD_USER_YN,\n g.EXPT_USER_YN,\n g.AU_FLAG,\n g.NRU_FLAG,\n g.RU_FLAG,\n g.USER_TYPE_ITG_CD,\n g.ACM_CONN_DCNT,\n g.CONTINUE_CONN_DCNT,\n g.CONTINUE_NON_CONN_DCNT,\n g.ACM_PAYMT_AMT,\n g.TDAY_PAYMT_AMT,\n g.TDAY_PLAY_TIME,\n g.MAX_LEVEL,\n g.LAST_CONN_DT,\n g.LAST_CONN_COUNTRY_CD,\n g.LAST_CONN_OS_ITG_CD,\n g.LAST_CONN_MARKET_ITG_CD,\n g.FIRST_CONN_DTM,\n g.RAW_NRU_DT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" g\nWHERE\n EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\n WHERE a.USE_YN = 'Y'\n AND a.GAME_PREFIX = 'CZN'\n AND (\n UPPER(a.GAME_NM) LIKE UPPER('%로드나인%')\n OR UPPER(a.GAME_ALIAS_NM) LIKE UPPER('%로드나인%')\n )\n )\nORDER BY\n g.BASE_DT DESC,\n g.GUID",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 56954
+ }
+ },
+ {
+ "case_id": "STD-09",
+ "title": "버블리즈 매출",
+ "category": "STANDARD",
+ "scenario_no": 9,
+ "question": "2026년 7월 15일 버블리즈 매출 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 13,
+ "scenario": "한글 게임명 → game_id 매핑",
+ "natural_question": "버블리즈 어제 매출 알려줘",
+ "sample_query": "SELECT SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID = 'STOVE_BUBBLYZ'\n AND EXPT_USER_YN = 'N'",
+ "note": "게임 조건 기준 프롬프트 추가 필요\n매출 기준 사용시 EXPT_USER_YN = 'N' 조건 사용할 수 있도록 프롬프트 제공"
+ },
+ "expected_focus": "버블리즈 game_id가 현재 샘플 데이터에 없으므로 임의 game_id 생성 없이 거절/0건 처리해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS SALES_AMOUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND s.\"EXPT_USER_YN\" = 'N'\n AND s.\"GAME_ID\" IN (\n SELECT g.\"GAME_ID\"\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE\n g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%버블리즈%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%버블리즈%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%버블리즈%')\n OR UPPER(g.\"GAME_ID\") LIKE UPPER('%버블리즈%')\n )\n )",
+ "answer_text": "SALES_AMOUNT=------------",
+ "execution_output": "SALES_AMOUNT\n------------",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 48233
+ }
+ },
+ {
+ "case_id": "STD-10",
+ "title": "Bubblyz 매출",
+ "category": "STANDARD",
+ "scenario_no": 10,
+ "question": "2026년 7월 15일 Bubblyz 매출 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 14,
+ "scenario": "영문 게임명 → game_id 매핑",
+ "natural_question": "Bubblyz 어제 매출 알려줘",
+ "sample_query": "SELECT SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID = 'STOVE_BUBBLYZ'\n AND EXPT_USER_YN = 'N'",
+ "note": ""
+ },
+ "expected_focus": "Bubblyz game_id가 현재 샘플 데이터에 없으므로 임의 game_id 생성 없이 거절/0건 처리해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS SALES_AMOUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND s.\"EXPT_USER_YN\" = 'N'\n AND s.\"GAME_ID\" = (\n SELECT g.\"GAME_ID\"\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE\n g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%Bubblyz%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%Bubblyz%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%Bubblyz%')\n OR UPPER(g.\"GAME_ID\") LIKE UPPER('%Bubblyz%')\n )\n )",
+ "answer_text": "SALES_AMOUNT=------------",
+ "execution_output": "SALES_AMOUNT\n------------",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 49464
+ }
+ },
+ {
+ "case_id": "STD-11",
+ "title": "Bubblyz와 카제나 매출 비교",
+ "category": "STANDARD",
+ "scenario_no": 11,
+ "question": "2026년 7월 15일 Bubblyz랑 카제나 매출 비교해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 15,
+ "scenario": "한글/영문 혼용 입력",
+ "natural_question": "Bubblyz랑 카제나 매출 비교해줘",
+ "sample_query": "SELECT GAME_ID, SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID IN ('STOVE_BUBBLYZ', 'STOVE_CHAOSZERO')\n AND EXPT_USER_YN = 'N'\n GROUP BY GAME_ID",
+ "note": ""
+ },
+ "expected_focus": "Bubblyz는 미지원, 카제나는 COMN_SALES_TXN/STOVE_CHAOSZERO 집계 가능하다.",
+ "support_level": "PARTIAL",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "COMN_GAME_ALIAS_BAS",
+ "SUM"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n gab1.\"GAME_NM\" AS GAME_NAME_1,\n SUM(CASE WHEN st1.\"BASE_DT\" = TO_DATE('20260715', 'YYYYMMDD') THEN CAST(st1.\"PAYMT_AMT\" AS NUMBER) ELSE 0 END) AS SALES_AMT_1,\n gab2.\"GAME_NM\" AS GAME_NAME_2,\n SUM(CASE WHEN st2.\"BASE_DT\" = TO_DATE('20260715', 'YYYYMMDD') THEN CAST(st2.\"PAYMT_AMT\" AS NUMBER) ELSE 0 END) AS SALES_AMT_2\nFROM\n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" gab1,\n \"SGMP_POC\".\"COMN_SALES_TXN\" st1,\n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" gab2,\n \"SGMP_POC\".\"COMN_SALES_TXN\" st2\nWHERE\n gab1.\"USE_YN\" = 'Y'\n AND (UPPER(gab1.\"GAME_NM\") LIKE UPPER('%Bubblyz%')\n OR UPPER(gab1.\"GAME_ALIAS_NM\") LIKE UPPER('%Bubblyz%')\n OR UPPER(gab1.\"GAME_PREFIX\") LIKE UPPER('%Bubblyz%')\n OR UPPER(gab1.\"GAME_ID\") LIKE UPPER('%Bubblyz%'))\n AND st1.\"GAME_ID\" = gab1.\"GAME_ID\"\n AND st1.\"BASE_DT\" = TO_DATE('20260715', 'YYYYMMDD')\n AND st1.\"EXPT_USER_YN\" = 'N'\n AND gab2.\"USE_YN\" = 'Y'\n AND (UPPER(gab2.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(gab2.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(gab2.\"GAME_PREFIX\") LIKE UPPER('%카제나%')\n OR UPPER(gab2.\"GAME_ID\") LIKE UPPER('%카제나%'))\n AND st2.\"GAME_ID\" = gab2.\"GAME_ID\"\n AND st2.\"BASE_DT\" = TO_DATE('20260715', 'YYYYMMDD')\n AND st2.\"EXPT_USER_YN\" = 'N'\nGROUP BY\n gab1.\"GAME_NM\",\n gab2.\"GAME_NM\"",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "WARN",
+ "judgment_reason": "Required SQL terms are present.\nPartial scenario contains unsupported scope; generated SQL should be reviewed before demo use.",
+ "duration_ms": 51879
+ }
+ },
+ {
+ "case_id": "STD-12",
+ "title": "약어/FULL NAME 매출",
+ "category": "STANDARD",
+ "scenario_no": 12,
+ "question": "2026년 7월 15일 로나 유저 매출과 카오스 제로 나이트메어 매출을 확인해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 16,
+ "scenario": "게임 약어 / FULL NAME 사용",
+ "natural_question": "1. 로나 유저 어제 매출 확인해줘\n2. 카오스 제로 나이트메어 어제 매출 확인해줘",
+ "sample_query": "SELECT 'STOVE_LORD' AS GAME_ID, SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID = 'STOVE_LORD'\n AND EXPT_USER_YN = 'N'\nUNION ALL\nSELECT 'STOVE_CHAOSZERO' AS GAME_ID, SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID = 'STOVE_CHAOSZERO'\n AND EXPT_USER_YN = 'N'",
+ "note": ""
+ },
+ "expected_focus": "로나는 미지원, 카오스 제로 나이트메어는 COMN_SALES_TXN/STOVE_CHAOSZERO 집계 가능하다.",
+ "support_level": "PARTIAL",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "COMN_GAME_ALIAS_BAS",
+ "SUM"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "",
+ "answer_text": "",
+ "execution_output": "Sorry, unfortunately a valid SELECT statement could not be generated for your natural language prompt with the restriction of enforcing the use of object list. Here is some more information to help you further: \n \n```sql \nSELECT \n g1.\"GAME_NM\" AS GAME_NAME_1, \n SUM(s1.\"PAYMT_AMT\") AS SALES_AMOUNT_1, \n g2.\"GAME_NM\" AS GAME_NAME_2, \n s2_subq.\"SALES_AMOUNT_2\" \nFROM \n \"SGMP_POC\".\"COMN_SALES_TXN\" s1 \n INNER JOIN \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g1 \n ON s1.\"GAME_ID\" = g1.\"GAME_ID\" \n AND g1.\"USE_YN\" = 'Y' \n AND (UPPER(g1.\"GAME_NM\") LIKE UPPER('%로나%') \n OR UPPER(g1.\"GAME_ALIAS_NM\") LIKE UPPER('%로나%') \n OR UPPER(g1.\"GAME_PREFIX\") LIKE UPPER('%로나%')) \n CROSS JOIN ( \n SELECT \n g.\"GAME_NM\", \n SUM(s.\"PAYMT_AMT\") AS SALES_AMOUNT_2 \n FROM \n \"SGMP_POC\".\"COMN_SALES_TXN\" s \n INNER JOIN \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g \n ON s.\"GAME_ID\" = g.\"GAME_ID\" \n AND g.\"USE_YN\" = 'Y' \n AND (UPPER(g.\"GAME_NM\") LIKE UPPER('%카오스 제로 나이트메어%') \n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카오스 제로 나이트메어%') \n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%카오스 제로 나이트메어%')) \n WHERE \n TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD') \n AND s.\"EXPT_USER_YN\" = 'N' \n GROUP BY \n g.\"GAME_NM\" \n ) s2_subq \n CROSS JOIN ( \n SELECT \n g.\"GAME_NM\" AS GAME_NAME_2 \n FROM \n \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g \n WHERE \n g.\"USE_YN\" = 'Y' \n AND (UPPER(g.\"GAME_NM\") LIKE UPPER('%카오스 제로 나이트메어%') \n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카오스 제로 나이트메어%') \n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%카오스 제로 나이트메어%')) \n ) g2 \nWHERE \n TRUNC(s1.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD') \n...[historical output truncated]",
+ "execution_status": "FAIL_GENERATE",
+ "judgment_status": "WARN",
+ "judgment_reason": "Execution status is FAIL_GENERATE.\nPartial scenario could not generate SQL; manual split between supported and unsupported game scope is needed.",
+ "duration_ms": 52515
+ }
+ },
+ {
+ "case_id": "STD-13",
+ "title": "전체 매출",
+ "category": "STANDARD",
+ "scenario_no": 13,
+ "question": "2026년 7월 15일 전체 매출 합계를 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 17,
+ "scenario": "게임명 미언급 — 전체 조회",
+ "natural_question": "어제 전체 매출 알려줘",
+ "sample_query": "별도 샘플 쿼리 없음",
+ "note": "L1 게임에 대한 가이드 필요할 듯\n일일 지표 합산과 동일한 값이라고 한다면 로스트아크 제외 와 같은 조건 프롬프트 제공 필요"
+ },
+ "expected_focus": "COMN_SALES_TXN의 PAYMT_AMT 전체 합계.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "PAYMT_AMT",
+ "SUM"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS TOTAL_SALES_AMOUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND s.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "TOTAL_SALES_AMOUNT=227681",
+ "execution_output": "TOTAL_SALES_AMOUNT\n------------------\n 227681",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 55210
+ }
+ },
+ {
+ "case_id": "STD-14",
+ "title": "미등록 게임 매출",
+ "category": "STANDARD",
+ "scenario_no": 14,
+ "question": "2026년 7월 15일 테스트게임 매출 조회해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 18,
+ "scenario": "미등록 게임명 입력",
+ "natural_question": "테스트게임 매출 조회해줘",
+ "sample_query": "별도 샘플 쿼리 없음",
+ "note": ""
+ },
+ "expected_focus": "미등록 게임은 임의 game_id 생성 없이 거절/실패해야 한다.",
+ "support_level": "UNSUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS TOTAL_SALES_AMT,\n COUNT(DISTINCT s.\"GUID\") AS PAYING_USER_COUNT,\n COUNT(*) AS TRANSACTION_COUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n s.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND s.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%테스트게임%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%테스트게임%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%테스트게임%')\n OR UPPER(g.\"GAME_ID\") LIKE UPPER('%테스트게임%')\n )\n AND s.\"GAME_ID\" = g.\"GAME_ID\"\n )",
+ "answer_text": "TOTAL_SALES_AMT PAYING_USER_COUNT TRANSACTION_COUNT\n--------------- ----------------- -----------------\n 0 0",
+ "execution_output": "TOTAL_SALES_AMT PAYING_USER_COUNT TRANSACTION_COUNT\n--------------- ----------------- -----------------\n 0 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.\nUnsupported scenario produced executable SQL constrained by COMN_GAME_ALIAS_BAS lookup; alias mismatch yields empty/NULL result.",
+ "duration_ms": 48992
+ }
+ },
+ {
+ "case_id": "STD-15",
+ "title": "game_id 직접 입력 매출",
+ "category": "STANDARD",
+ "scenario_no": 15,
+ "question": "game_id가 STOVE_CHAOSZERO인 2026년 7월 15일 매출 합계를 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 19,
+ "scenario": "game_id 직접 입력",
+ "natural_question": "game_id가 STOVE_CHAOSZERO 매출 보여줘",
+ "sample_query": "SELECT SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID = 'STOVE_CHAOSZERO'\n AND EXPT_USER_YN = 'N'",
+ "note": "STOVE_CHAOSZERO 매출 보여줘 이렇게만 질문해도 제대로 조건 적용해줄 수 있는지 확인"
+ },
+ "expected_focus": "COMN_SALES_TXN, GAME_ID='STOVE_CHAOSZERO', PAYMT_AMT 합계.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "PAYMT_AMT",
+ "SUM",
+ "STOVE_CHAOSZERO"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS TOTAL_SALES_AMOUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n s.\"GAME_ID\" = 'STOVE_CHAOSZERO'\n AND TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND s.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "TOTAL_SALES_AMOUNT=227681",
+ "execution_output": "TOTAL_SALES_AMOUNT\n------------------\n 227681",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 56149
+ }
+ },
+ {
+ "case_id": "STD-16",
+ "title": "어제 카제나 AU",
+ "category": "STANDARD",
+ "scenario_no": 16,
+ "question": "2026년 7월 15일 카제나 AU 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 21,
+ "scenario": "날짜 상대표현 — 어제",
+ "natural_question": "어제 카제나 AU 알려줘",
+ "sample_query": "SELECT COUNT(1)\n FROM DW.CZN_COMN_USER_MST\nWHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND AU_FLAG = 1\n AND EXPT_USER_YN = 'N'\n AND STD_USER_YN = 'Y' -- 해당 조건 생략 가능",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_USER_MST, AU_FLAG=1, EXPT_USER_YN='N'.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "AU_COUNT=0",
+ "execution_output": "AU_COUNT\n----------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 66228
+ }
+ },
+ {
+ "case_id": "STD-17",
+ "title": "지난달 카제나 매출",
+ "category": "STANDARD",
+ "scenario_no": 17,
+ "question": "2026년 7월 15일 카제나 매출 합계를 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 22,
+ "scenario": "날짜 상대표현 — 지난달",
+ "natural_question": "지난달 카제나 매출",
+ "sample_query": "SELECT SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT BETWEEN TRUNC(ADD_MONTHS(CURRENT_DATE, -1), 'MM') AND DATE_SUB(TRUNC(CURRENT_DATE, 'MM'), 1)\n AND GAME_ID = 'STOVE_CHAOSZERO'\n AND EXPT_USER_YN = 'N'",
+ "note": ""
+ },
+ "expected_focus": "현재 샘플 기준일로 보정한 COMN_SALES_TXN 매출 합계.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "COMN_GAME_ALIAS_BAS",
+ "PAYMT_AMT",
+ "SUM"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS TOTAL_SALES_AMOUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n s.\"EXPT_USER_YN\" = 'N'\n AND TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n AND s.\"GAME_ID\" = g.\"GAME_ID\"\n )",
+ "answer_text": "TOTAL_SALES_AMOUNT=227681",
+ "execution_output": "TOTAL_SALES_AMOUNT\n------------------\n 227681",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 65168
+ }
+ },
+ {
+ "case_id": "STD-18",
+ "title": "1만원 초과 주문",
+ "category": "STANDARD",
+ "scenario_no": 18,
+ "question": "2026년 7월 15일 전체 매출에서 결제금액이 1만원 넘는 주문만 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 23,
+ "scenario": "단위 인식 — 만원",
+ "natural_question": "어제 매출에서 1만원 넘는 주문만",
+ "sample_query": "SELECT *\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND PAYMT_AMT > 10000\n AND EXPT_USER_YN = 'N'",
+ "note": ""
+ },
+ "expected_focus": "COMN_SALES_TXN, PAYMT_AMT > 10000.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "PAYMT_AMT",
+ "10000"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS TOTAL_SALES_AMOUNT,\n COUNT(DISTINCT s.\"GUID\") AS BUYER_COUNT,\n COUNT(*) AS ORDER_COUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND CAST(s.\"PAYMT_AMT\" AS NUMBER) > 10000\n AND s.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "TOTAL_SALES_AMOUNT BUYER_COUNT ORDER_COUNT\n------------------ ----------- -----------\n 204720 6 6",
+ "execution_output": "TOTAL_SALES_AMOUNT BUYER_COUNT ORDER_COUNT\n------------------ ----------- -----------\n 204720 6 6",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 47379
+ }
+ },
+ {
+ "case_id": "STD-19",
+ "title": "매출 발생 유저 수",
+ "category": "STANDARD",
+ "scenario_no": 19,
+ "question": "2026년 7월 15일 카제나 매출 발생 유저 수를 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 24,
+ "scenario": "동의어 매핑",
+ "natural_question": "어제 카제나 매출 발생 유저 수 알려줘\n어제 카제나 구매자 수 알려줘\n어제 카제나 결제 유저 수 알려줘",
+ "sample_query": "SELECT COUNT(DISTINCT GUID)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID = 'STOVE_CHAOSZERO'\n AND EXPT_USER_YN = 'N'",
+ "note": ""
+ },
+ "expected_focus": "COMN_SALES_TXN, COUNT(DISTINCT GUID), GAME_ID='STOVE_CHAOSZERO'.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "COMN_GAME_ALIAS_BAS",
+ "GUID",
+ "COUNT",
+ "DISTINCT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT s.\"GUID\") AS SALES_USER_COUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\n INNER JOIN \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n ON s.\"GAME_ID\" = g.\"GAME_ID\"\nWHERE\n g.\"GAME_NM\" LIKE '%' || UPPER('카제나') || '%'\n AND g.\"USE_YN\" = 'Y'\n AND TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND s.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "SALES_USER_COUNT=10",
+ "execution_output": "SALES_USER_COUNT\n----------------\n 10",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 52163
+ }
+ },
+ {
+ "case_id": "STD-20",
+ "title": "구어체 매출",
+ "category": "STANDARD",
+ "scenario_no": 20,
+ "question": "2026년 7월 15일 카제나 매출 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 25,
+ "scenario": "구어체 컬럼 매핑",
+ "natural_question": "어제 카제나 매출 알려줘",
+ "sample_query": "SELECT SUM(PAYMT_AMT)\n FROM DW.COMN_SALES_TXN\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND GAME_ID = 'STOVE_CHAOSZERO'\n AND EXPT_USER_YN = 'N'",
+ "note": ""
+ },
+ "expected_focus": "COMN_SALES_TXN, PAYMT_AMT 합계, GAME_ID='STOVE_CHAOSZERO'.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "COMN_GAME_ALIAS_BAS",
+ "PAYMT_AMT",
+ "SUM"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(s.\"PAYMT_AMT\" AS NUMBER)) AS SALES_AMOUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\nWHERE\n s.\"EXPT_USER_YN\" = 'N'\n AND TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%CZN%')\n )\n AND s.\"GAME_ID\" = g.\"GAME_ID\"\n )",
+ "answer_text": "SALES_AMOUNT=227681",
+ "execution_output": "SALES_AMOUNT\n------------\n 227681",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 51070
+ }
+ },
+ {
+ "case_id": "STD-21",
+ "title": "복수 조건 AND",
+ "category": "STANDARD",
+ "scenario_no": 21,
+ "question": "2026년 7월 15일 카제나 한국 유저 중 7일 이상 연속 접속한 유저 수를 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 26,
+ "scenario": "복수 조건 AND/OR 구분",
+ "natural_question": "카제나 한국 유저 중 7일 이상 연속 접속한 유저",
+ "sample_query": "SELECT *\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND AU_FLAG = 1\n AND STD_USER_YN = 'Y' -- 해당 조건 생략 가능\n AND EXPT_USER_YN = 'N'\n AND LAST_CONN_COUNTRY_CD = 'KR'\n AND CONTINUE_CONN_DCNT >= 7",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_USER_MST, LAST_CONN_COUNTRY_CD='KR', CONTINUE_CONN_DCNT >= 7.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "LAST_CONN_COUNTRY_CD",
+ "KR",
+ "CONTINUE_CONN_DCNT",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS CONTINUOUS_7DAYS_USER_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"LAST_CONN_COUNTRY_CD\" = 'KR'\n AND u.\"CONTINUE_CONN_DCNT\" >= 7\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "CONTINUOUS_7DAYS_USER_COUNT=0",
+ "execution_output": "CONTINUOUS_7DAYS_USER_COUNT\n---------------------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 52228
+ }
+ },
+ {
+ "case_id": "STD-22",
+ "title": "일별 NRU",
+ "category": "STANDARD",
+ "scenario_no": 22,
+ "question": "2026년 7월 15일 카제나 NRU 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 28,
+ "scenario": "일별 NRU",
+ "natural_question": "카제나 어제 NRU 알려줘",
+ "sample_query": "SELECT COUNT(1) AS NRU\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND EXPT_USER_YN = 'N'\n AND NRU_FLAG = 1\n AND STD_USER_YN = 'Y' -- 해당 조건 생략 가능",
+ "note": "STD_USER_YN = 'Y' 생략 가능 — BASE_DT 단일 시점 기준이므로 해당 날짜의 표준 유저 여부가 이미 확정됨"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, NRU_FLAG=1.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "NRU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS NRU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"NRU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n AND g.\"USE_YN\" = 'Y'\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "NRU_COUNT=0",
+ "execution_output": "NRU_COUNT\n----------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 49058
+ }
+ },
+ {
+ "case_id": "STD-23",
+ "title": "일별 AU",
+ "category": "STANDARD",
+ "scenario_no": 23,
+ "question": "2026년 7월 15일 카제나 AU 구해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 29,
+ "scenario": "일별 AU",
+ "natural_question": "카제나 어제 AU 구해줘",
+ "sample_query": "SELECT COUNT(1) AS AU\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND EXPT_USER_YN = 'N'\n AND AU_FLAG = 1\n AND STD_USER_YN = 'Y' -- 해당 조건 생략 가능",
+ "note": "STD_USER_YN = 'Y' 생략 가능 — BASE_DT 단일 시점 기준이므로 해당 날짜의 표준 유저 여부가 이미 확정됨"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, AU_FLAG=1.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "AU_COUNT=0",
+ "execution_output": "AU_COUNT\n----------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 49017
+ }
+ },
+ {
+ "case_id": "STD-24",
+ "title": "주간 NRU",
+ "category": "STANDARD",
+ "scenario_no": 24,
+ "question": "2026년 7월 15일 기준 최근 7일 카제나 NRU 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 30,
+ "scenario": "주간 NRU",
+ "natural_question": "카제나 지난주 NRU 알려줘",
+ "sample_query": "SELECT COUNT(*) AS WEEKLY_NRU\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = NEXT_DAY(DATE_SUB(CURRENT_DATE, 8), 'SU') -- 지난주 일요일\n AND RAW_NRU_DT BETWEEN\n DATE_SUB(NEXT_DAY(DATE_SUB(CURRENT_DATE, 8), 'SU'), 6) -- 지난주 월요일\n AND NEXT_DAY(DATE_SUB(CURRENT_DATE, 8), 'SU') -- 지난주 일요일\n AND STD_USER_YN = 'Y'\n AND EXPT_USER_YN = 'N'\n;",
+ "note": "STD_USER_YN = 'Y' 필수 — 집계 기간 내 표준 유저 상태가 변동될 수 있음 (예: 월초에는 문제 유저로 분류됐다가 이후 표준 유저로 전환되는 케이스). BASE_DT(마지막날) 기준으로 표준 유저 여부를 최종 필터링해야 정확한 집계 가능"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, RAW_NRU_DT 기간 조건.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "RAW_NRU_DT",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS NRU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"RAW_NRU_DT\" BETWEEN TO_DATE('2026-07-09', 'YYYY-MM-DD') AND TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"STD_USER_YN\" = 'Y'\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "NRU_COUNT=0",
+ "execution_output": "NRU_COUNT\n----------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 47412
+ }
+ },
+ {
+ "case_id": "STD-25",
+ "title": "주간 AU",
+ "category": "STANDARD",
+ "scenario_no": 25,
+ "question": "2026년 7월 15일 기준 최근 7일 카제나 AU 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 31,
+ "scenario": "주간 AU",
+ "natural_question": "카제나 지난주 AU 알려줘",
+ "sample_query": "SELECT COUNT(*) AS WEEKLY_AU\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = NEXT_DAY(DATE_SUB(CURRENT_DATE, 8), 'SU') -- 지난주 일요일\n AND LAST_CONN_DT BETWEEN\n DATE_SUB(NEXT_DAY(DATE_SUB(CURRENT_DATE, 8), 'SU'), 6) -- 지난주 월요일\n AND NEXT_DAY(DATE_SUB(CURRENT_DATE, 8), 'SU') -- 지난주 일요일\n AND STD_USER_YN = 'Y'\n AND EXPT_USER_YN = 'N'\n;",
+ "note": "STD_USER_YN = 'Y' 필수 — 집계 기간 내 표준 유저 상태가 변동될 수 있음 (예: 월초에는 문제 유저로 분류됐다가 이후 표준 유저로 전환되는 케이스). BASE_DT(마지막날) 기준으로 표준 유저 여부를 최종 필터링해야 정확한 집계 가능"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, LAST_CONN_DT 기간 조건.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "LAST_CONN_DT",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS RECENT_7DAY_AU\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = (SELECT MAX(\"BASE_DT\") FROM \"SGMP_POC\".\"CZN_COMN_USER_MST\")\n AND u.\"LAST_CONN_DT\" BETWEEN TRUNC(TO_DATE('20260715', 'YYYYMMDD')) - 6 AND TRUNC(TO_DATE('20260715', 'YYYYMMDD'))\n AND u.\"STD_USER_YN\" = 'Y'\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "RECENT_7DAY_AU=0",
+ "execution_output": "RECENT_7DAY_AU\n--------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 48392
+ }
+ },
+ {
+ "case_id": "STD-26",
+ "title": "월간 NRU",
+ "category": "STANDARD",
+ "scenario_no": 26,
+ "question": "카제나 현재월 NRU 구해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 32,
+ "scenario": "월간 NRU",
+ "natural_question": "카제나 지난달 NRU 구해줘",
+ "sample_query": "SELECT COUNT(*) AS MONTHLY_NRU\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_SUB(TRUNC(CURRENT_DATE, 'MM'), 1)\n AND RAW_NRU_DT BETWEEN TRUNC(ADD_MONTHS(CURRENT_DATE, -1), 'MM')\n AND DATE_SUB(TRUNC(CURRENT_DATE, 'MM'), 1)\n AND STD_USER_YN = 'Y'\n AND EXPT_USER_YN = 'N'",
+ "note": "STD_USER_YN = 'Y' 필수 — 집계 기간 내 표준 유저 상태가 변동될 수 있음 (예: 월초에는 문제 유저로 분류됐다가 이후 표준 유저로 전환되는 케이스). BASE_DT(마지막날) 기준으로 표준 유저 여부를 최종 필터링해야 정확한 집계 가능"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, MAX(BASE_DT) 최신 스냅샷, RAW_NRU_DT 월 범위 조건.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "MAX",
+ "BASE_DT",
+ "RAW_NRU_DT",
+ "STD_USER_YN",
+ "EXPT_USER_YN",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS NRU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = (SELECT MAX(\"BASE_DT\") FROM \"SGMP_POC\".\"CZN_COMN_USER_MST\")\n AND u.\"RAW_NRU_DT\" BETWEEN TRUNC(u.\"BASE_DT\", 'MM') AND u.\"BASE_DT\"\n AND u.\"STD_USER_YN\" = 'Y'\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "NRU_COUNT=0",
+ "execution_output": "NRU_COUNT\n----------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 46667
+ }
+ },
+ {
+ "case_id": "STD-27",
+ "title": "월간 AU",
+ "category": "STANDARD",
+ "scenario_no": 27,
+ "question": "카제나 지난달 AU 구해줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 33,
+ "scenario": "월간 AU",
+ "natural_question": "카제나 지난달 AU 구해줘",
+ "sample_query": "SELECT COUNT(*) AS MONTHLY_AU\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_SUB(TRUNC(CURRENT_DATE, 'MM'), 1)\n AND LAST_CONN_DT BETWEEN TRUNC(ADD_MONTHS(CURRENT_DATE, -1), 'MM')\n AND DATE_SUB(TRUNC(CURRENT_DATE, 'MM'), 1)\n AND STD_USER_YN = 'Y'\n AND EXPT_USER_YN = 'N'",
+ "note": "STD_USER_YN = 'Y' 필수 — 집계 기간 내 표준 유저 상태가 변동될 수 있음 (예: 월초에는 문제 유저로 분류됐다가 이후 표준 유저로 전환되는 케이스). BASE_DT(마지막날) 기준으로 표준 유저 여부를 최종 필터링해야 정확한 집계 가능"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, MAX(BASE_DT) 기준 전월 스냅샷, LAST_CONN_DT 전월 범위, STD_USER_YN/EXPT_USER_YN 조건.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "MAX",
+ "BASE_DT",
+ "ADD_MONTHS",
+ "LAST_CONN_DT",
+ "STD_USER_YN",
+ "EXPT_USER_YN",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS LAST_MONTH_AU\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = LAST_DAY(ADD_MONTHS((SELECT MAX(\"BASE_DT\") FROM \"SGMP_POC\".\"CZN_COMN_USER_MST\"), -1))\n AND u.\"LAST_CONN_DT\" BETWEEN TRUNC(ADD_MONTHS((SELECT MAX(\"BASE_DT\") FROM \"SGMP_POC\".\"CZN_COMN_USER_MST\"), -1), 'MM') AND LAST_DAY(ADD_MONTHS((SELECT MAX(\"BASE_DT\") FROM \"SGMP_POC\".\"CZN_COMN_USER_MST\"), -1))\n AND u.\"STD_USER_YN\" = 'Y'\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "LAST_MONTH_AU=0",
+ "execution_output": "LAST_MONTH_AU\n-------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 51920
+ }
+ },
+ {
+ "case_id": "STD-28",
+ "title": "RU 전체 유저",
+ "category": "STANDARD",
+ "scenario_no": 28,
+ "question": "2026년 7월 15일 카제나 기준 전체 유저 수를 알려줘.",
+ "source": {
+ "workbook": "표준 DW 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "POC 시나리오",
+ "excel_row": 34,
+ "scenario": "RU (전체 유저)",
+ "natural_question": "카제나 어제 기준 전체 유저 수",
+ "sample_query": "SELECT COUNT(1) AS RU\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = DATE_ADD(`CURRENT_DATE`(), -1)\n AND EXPT_USER_YN = 'N'\n AND STD_USER_YN = 'Y'",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_USER_MST, STD_USER_YN='Y', EXPT_USER_YN='N'.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "STD_USER_YN",
+ "COUNT"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS TOTAL_USER_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"STD_USER_YN\" = 'Y'\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "TOTAL_USER_COUNT=10",
+ "execution_output": "TOTAL_USER_COUNT\n----------------\n 10",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 46640
+ }
+ },
+ {
+ "case_id": "CZN-01",
+ "title": "사업 AU",
+ "category": "CZN",
+ "scenario_no": 1,
+ "question": "카제나에서 2026년 7월 15일 기준 사업 AU 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 2,
+ "scenario": "EBIR-6034",
+ "natural_question": "카제나에서 2026년 5월 25일부터 5월 31일까지 일자별 사업 AU 를 알려줘.",
+ "sample_query": "SELECT BASE_DT\n , COUNT(1) AS AU_CNT -- COUNT(DISTINCT GUID) 사용 가능\n FROM DW.CZN_CUSTOM_BIZ_USER_TXN\n WHERE BASE_DT BETWEEN '2026-05-25' AND '2026-05-31'\n AND BIZ_AU_FLAG = 1\n AND EXPT_USER_YN = 'N'\n GROUP BY BASE_DT\n ORDER BY 1\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_CUSTOM_BIZ_USER_TXN, BIZ_AU_FLAG=1.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_CUSTOM_BIZ_USER_TXN",
+ "BIZ_AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT b.\"GUID\") AS BIZ_AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_CUSTOM_BIZ_USER_TXN\" b\nWHERE\n b.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND b.\"BIZ_AU_FLAG\" = 1\n AND b.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "BIZ_AU_COUNT=1",
+ "execution_output": "BIZ_AU_COUNT\n------------\n 1",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 48171
+ }
+ },
+ {
+ "case_id": "CZN-02",
+ "title": "표준 AU",
+ "category": "CZN",
+ "scenario_no": 2,
+ "question": "카제나에서 2026년 7월 15일 기준 표준 AU 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 3,
+ "scenario": "EBIR-6034",
+ "natural_question": "카제나에서 2026년 5월 25일부터 5월 31일까지 일자별 표준 AU 를 알려줘.",
+ "sample_query": "SELECT BASE_DT\n , COUNT(1) AS AU_CNT -- COUNT(DISTINCT GUID) 사용 가능\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT BETWEEN '2026-05-25' AND '2026-05-31'\n AND AU_FLAG = 1\n AND EXPT_USER_YN = 'N'\n GROUP BY BASE_DT\n ORDER BY 1\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_USER_MST, AU_FLAG=1.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS STD_AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "STD_AU_COUNT=0",
+ "execution_output": "STD_AU_COUNT\n------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 45976
+ }
+ },
+ {
+ "case_id": "CZN-03",
+ "title": "표준 AU와 사업 AU 비교",
+ "category": "CZN",
+ "scenario_no": 3,
+ "question": "카제나에서 2026년 7월 15일 기준 표준 AU와 사업 AU를 비교해서 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 4,
+ "scenario": "EBIR-6034",
+ "natural_question": "카제나에서 2026년 5월 25일부터 5월 31일까지 일자별 표준 AU와 사업 AU 를 비교해서 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , SUM(CASE WHEN T1.AU_FLAG = 1 THEN 1 END) AS STD_AU_CNT\n , SUM(CASE WHEN T2.BIZ_AU_FLAG = 1 THEN 1 END) AS BIZ_AU_CNT\n FROM DW.CZN_COMN_USER_MST T1\n LEFT JOIN DW.CZN_CUSTOM_BIZ_USER_TXN T2\n ON T1.GUID = T2.GUID\n AND T1.BASE_DT = T2.BASE_DT\n WHERE T1.BASE_DT BETWEEN '2026-05-25' AND '2026-05-31'\n AND T1.EXPT_USER_YN = 'N'\n GROUP BY T1.BASE_DT\n ORDER BY 1\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_USER_MST와 CZN_CUSTOM_BIZ_USER_TXN의 AU 지표 비교.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "CZN_CUSTOM_BIZ_USER_TXN",
+ "AU_FLAG",
+ "BIZ_AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n STD_AU.STD_AU_COUNT,\n BIZ_AU.BIZ_AU_COUNT\nFROM\n (\n SELECT\n COUNT(DISTINCT u.\"GUID\") AS STD_AU_COUNT\n FROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n WHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%CZN%')\n )\n AND g.\"GAME_PREFIX\" = 'CZN'\n )\n ) STD_AU,\n (\n SELECT\n COUNT(DISTINCT b.\"GUID\") AS BIZ_AU_COUNT\n FROM\n \"SGMP_POC\".\"CZN_CUSTOM_BIZ_USER_TXN\" b\n WHERE\n b.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND b.\"BIZ_AU_FLAG\" = 1\n AND b.\"EXPT_USER_YN\" = 'N'\n ) BIZ_AU",
+ "answer_text": "STD_AU_COUNT BIZ_AU_COUNT\n------------ ------------\n 0 1",
+ "execution_output": "STD_AU_COUNT BIZ_AU_COUNT\n------------ ------------\n 0 1",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 52090
+ }
+ },
+ {
+ "case_id": "CZN-04",
+ "title": "캐릭터 AU",
+ "category": "CZN",
+ "scenario_no": 4,
+ "question": "카제나에서 2026년 7월 15일 기준 캐릭터 AU 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 5,
+ "scenario": "",
+ "natural_question": "카제나에서 2026년 5월 25일부터 5월 31일까지 일자별 캐릭터 AU 를 알려줘.",
+ "sample_query": "SELECT BASE_DT\n , COUNT(1) AS AU_CNT -- COUNT(DISTINCT CONCAT(SERVER_CD, '_', CUID)) 사용 가능\n FROM DW.CZN_COMN_CHARACTER_MST\n WHERE BASE_DT BETWEEN '2026-05-25' AND '2026-05-31'\n AND AU_FLAG = 1\n AND EXPT_USER_YN = 'N'\n GROUP BY BASE_DT\n ORDER BY 1\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_CHARACTER_MST, AU_FLAG=1.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_CHARACTER_MST",
+ "AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT c.\"CUID\") AS CHARACTER_AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_CHARACTER_MST\" c\nWHERE\n c.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND c.\"AU_FLAG\" = 1\n AND c.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "CHARACTER_AU_COUNT=0",
+ "execution_output": "CHARACTER_AU_COUNT\n------------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 49640
+ }
+ },
+ {
+ "case_id": "CZN-05",
+ "title": "주요 국가별 사업 AU",
+ "category": "CZN",
+ "scenario_no": 5,
+ "question": "카제나에서 2026년 7월 15일 기준 주요 국가별 사업 AU 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 6,
+ "scenario": "EBIR-6034",
+ "natural_question": "카제나에서 2026년 5월 25일부터 5월 31일까지 일자별, 주요 국가별 사업 AU 를 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , CASE WHEN T2.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END AS COUNTRY_NM\n , COUNT(1) AS BIZ_AU_CNT\n FROM DW.CZN_CUSTOM_BIZ_USER_TXN T1\n INNER JOIN DW.CZN_COMN_USER_MST T2\n ON T1.GUID = T2.GUID\n AND T1.BASE_DT = T2.BASE_DT\n LEFT JOIN META.COMN_ITG_CD_BAS D1\n ON T2.LAST_CONN_COUNTRY_CD = D1.ITG_CD\n AND D1.ITG_CD_GROUP_ID = 'G20005' -- 주요 국가\n AND D1.SVC_DIV_CD = 'STOVE_CHAOSZERO'\n WHERE T1.BASE_DT BETWEEN '2026-05-25' AND '2026-05-31'\n AND T1.BIZ_AU_FLAG = 1\n AND T1.EXPT_USER_YN = 'N'\n GROUP BY T1.BASE_DT\n , CASE WHEN T2.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END\n ORDER BY 1, 2\n;",
+ "note": "-- 주요 국가 확인 방법\nSELECT *\n FROM META.COMN_ITG_CD_BAS\n WHERE ITG_CD_GROUP_ID = 'G20005' -- 주요국가\n AND SVC_DIV_CD = 'STOVE_CHAOSZERO'"
+ },
+ "expected_focus": "CZN_CUSTOM_BIZ_USER_TXN와 CZN_COMN_USER_MST 국가 컬럼 조인.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_CUSTOM_BIZ_USER_TXN",
+ "CZN_COMN_USER_MST",
+ "BIZ_AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n u.\"LAST_CONN_COUNTRY_CD\" AS COUNTRY_CD,\n COUNT(DISTINCT b.\"GUID\") AS BIZ_AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_CUSTOM_BIZ_USER_TXN\" b\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n ON b.\"GUID\" = u.\"GUID\"\n AND b.\"BASE_DT\" = u.\"BASE_DT\"\nWHERE\n b.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND b.\"BIZ_AU_FLAG\" = 1\n AND b.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND g.\"GAME_PREFIX\" = 'CZN'\n )\nGROUP BY\n u.\"LAST_CONN_COUNTRY_CD\"\nORDER BY\n BIZ_AU_COUNT DESC",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 50201
+ }
+ },
+ {
+ "case_id": "CZN-06",
+ "title": "주요 국가별 표준 AU",
+ "category": "CZN",
+ "scenario_no": 6,
+ "question": "카제나에서 2026년 7월 15일 기준 주요 국가별 표준 AU 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 7,
+ "scenario": "EBIR-6034",
+ "natural_question": "카제나에서 2026년 5월 25일부터 5월 31일까지 일자별, 주요 국가별 표준 AU 를 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END AS COUNTRY_NM\n , COUNT(1) AS STD_AU_CNT\n FROM DW.CZN_COMN_USER_MST T1\n LEFT JOIN META.COMN_ITG_CD_BAS D1\n ON T1.LAST_CONN_COUNTRY_CD = D1.ITG_CD\n AND D1.ITG_CD_GROUP_ID = 'G20005' -- 주요 국가\n AND D1.SVC_DIV_CD = 'STOVE_CHAOSZERO'\n WHERE T1.BASE_DT BETWEEN '2026-05-25' AND '2026-05-31'\n AND T1.AU_FLAG = 1\n AND T1.EXPT_USER_YN = 'N'\n GROUP BY T1.BASE_DT\n , CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END\n ORDER BY 1, 2\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_USER_MST 국가 컬럼과 AU_FLAG 집계.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n u.\"LAST_CONN_COUNTRY_CD\" AS COUNTRY_CD,\n c.\"COUNTRY_KR_NM\" AS COUNTRY_NM,\n COUNT(DISTINCT u.\"GUID\") AS STD_AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nLEFT JOIN\n \"SGMP_POC\".\"COMN_COUNTRY_BAS\" c\n ON u.\"LAST_CONN_COUNTRY_CD\" = c.\"COUNTRY_2CHAR_CD\"\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n AND g.\"USE_YN\" = 'Y'\n AND g.\"GAME_PREFIX\" = 'CZN'\n )\nGROUP BY\n u.\"LAST_CONN_COUNTRY_CD\",\n c.\"COUNTRY_KR_NM\"\nORDER BY\n STD_AU_COUNT DESC",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 51693
+ }
+ },
+ {
+ "case_id": "CZN-07",
+ "title": "크리스탈 보유 유저/보유량",
+ "category": "CZN",
+ "scenario_no": 7,
+ "question": "카제나에서 2026년 4월 23일부터 4월 30일까지 일자별 크리스탈 보유 유저 수와 총 크리스탈 보유량을 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 8,
+ "scenario": "EBIR-5903",
+ "natural_question": "카제나에서 2026년 4월 23일부터 4월 30일까지 일자별 크리스탈 보유 유저 수와 총 크리스탈 보유량을 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , COUNT(DISTINCT T1.GUID) AS HAVE_USER_CNT\n , SUM(T1.HAVE_CNT) AS HAVE_CNT_SUM\n FROM DW.CZN_CUSTOM_GOODS_HAVE_TXN T1\n INNER JOIN META.CZN_COMN_SVC_DIM_BAS D1\n ON T1.GOODS_CD = D1.DIM_CD\n AND T1.BASE_DT = D1.BASE_DT\n AND D1.DIM_TYPE_DIV_CD = 'GOODS_AGG'\n AND D1.DIM_CTG_2_KR_NM = '크리스탈' -- 유료/무료 크리스탈을 모두 확인하기 위해 카테고리 명으로 확인\n INNER JOIN DW.CZN_COMN_USER_MST T2\n ON T1.GUID = T2.GUID\n AND T1.BASE_DT = T2.BASE_DT\n AND T2.RU_FLAG = 1\n AND T2.EXPT_USER_YN = 'N'\n WHERE T1.BASE_DT BETWEEN '2026-04-23' AND '2026-04-30'\n AND T1.HAVE_CNT <> 0\n GROUP BY T1.BASE_DT\n ORDER BY 1\n;",
+ "note": "-- 재화 코드 확인 방법\nSELECT *\n FROM META.CZN_COMN_SVC_DIM_BAS\n WHERE BASE_DT = '2026-06-24' -- 조회 필요한 날짜 기준\n AND DIM_TYPE_DIV_CD = 'GOODS_AGG'\n\n\n- 유저에 대한 기준 명시가 없는 경우 전체 유저(RU) 대상으로 집계"
+ },
+ "expected_focus": "재화 보유 테이블 집계, 재화 차원 lookup, 유저 모집단 조인, HAVE_CNT<>0, BASE_DT 일자별 GROUP BY.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_CUSTOM_GOODS_HAVE_TXN",
+ "CZN_COMN_SVC_DIM_BAS",
+ "CZN_COMN_USER_MST",
+ "GOODS_CD",
+ "DIM_CD",
+ "BASE_DT",
+ "HAVE_CNT",
+ "COUNT",
+ "SUM",
+ "GROUP"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n g.\"BASE_DT\",\n COUNT(DISTINCT u.\"GUID\") AS USER_COUNT,\n SUM(CAST(g.\"HAVE_CNT\" AS NUMBER)) AS TOTAL_CRYSTAL_AMOUNT\nFROM\n \"SGMP_POC\".\"CZN_CUSTOM_GOODS_HAVE_TXN\" g\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n ON g.\"GUID\" = u.\"GUID\"\n AND g.\"BASE_DT\" = u.\"BASE_DT\"\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_SVC_DIM_BAS\" d\n ON g.\"GOODS_CD\" = d.\"DIM_CD\"\n AND g.\"BASE_DT\" = d.\"BASE_DT\"\nWHERE\n EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\n WHERE a.\"USE_YN\" = 'Y'\n AND (\n UPPER(a.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_PREFIX\") LIKE UPPER('%CZN%')\n )\n )\n AND d.\"DIM_TYPE_DIV_CD\" = 'GOODS_AGG'\n AND d.\"DIM_CTG_2_KR_NM\" = '크리스탈'\n AND g.\"HAVE_CNT\" <> 0\n AND u.\"RU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND g.\"BASE_DT\" BETWEEN TO_DATE('2026-04-23', 'YYYY-MM-DD') AND TO_DATE('2026-04-30', 'YYYY-MM-DD')\nGROUP BY\n g.\"BASE_DT\"\nORDER BY\n g.\"BASE_DT\" ASC",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 52287
+ }
+ },
+ {
+ "case_id": "CZN-08",
+ "title": "표준 AU 크리스탈 보유량",
+ "category": "CZN",
+ "scenario_no": 8,
+ "question": "카제나에서 2026년 7월 15일 기준 표준 AU의 크리스탈 총 보유량과 1인당 평균 보유량을 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 9,
+ "scenario": "EBIR-5903",
+ "natural_question": "카제나에서 2026년 4월 23일부터 4월 30일까지 일자별 표준 AU 의 크리스탈 총 보유량과 1인당 평균 보유량을 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , COUNT(DISTINCT T1.GUID) AS HAVE_USER_CNT\n , SUM(T1.HAVE_CNT) AS HAVE_CNT_SUM\n , SUM(T1.HAVE_CNT) / COUNT(DISTINCT T1.GUID) AS AVG_HAVE_CNT_PER_AU\n FROM DW.CZN_CUSTOM_GOODS_HAVE_TXN T1\n INNER JOIN META.CZN_COMN_SVC_DIM_BAS D1\n ON T1.GOODS_CD = D1.DIM_CD\n AND T1.BASE_DT = D1.BASE_DT\n AND D1.DIM_TYPE_DIV_CD = 'GOODS_AGG'\n AND D1.DIM_CTG_2_KR_NM = '크리스탈' -- 유료/무료 크리스탈을 모두 확인하기 위해 카테고리 명으로 확인\n INNER JOIN DW.CZN_COMN_USER_MST T2\n ON T1.GUID = T2.GUID\n AND T1.BASE_DT = T2.BASE_DT\n AND T2.AU_FLAG = 1\n AND T2.EXPT_USER_YN = 'N'\n WHERE T1.BASE_DT BETWEEN '2026-04-23' AND '2026-04-30'\n AND T1.HAVE_CNT <> 0\n GROUP BY T1.BASE_DT\n ORDER BY 1\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_CUSTOM_GOODS_HAVE_TXN, CZN_COMN_USER_MST, CZN_COMN_SVC_DIM_BAS, AU_FLAG=1, EXPT_USER_YN='N', SUM(HAVE_CNT)/COUNT(DISTINCT GUID).",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_CUSTOM_GOODS_HAVE_TXN",
+ "CZN_COMN_USER_MST",
+ "CZN_COMN_SVC_DIM_BAS",
+ "AU_FLAG",
+ "EXPT_USER_YN",
+ "HAVE_CNT",
+ "SUM",
+ "COUNT",
+ "DISTINCT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(g.\"HAVE_CNT\" AS NUMBER)) AS TOTAL_CRYSTAL_HAVE,\n SUM(CAST(g.\"HAVE_CNT\" AS NUMBER)) / NULLIF(COUNT(DISTINCT u.\"GUID\"), 0) AS AVG_CRYSTAL_PER_USER\nFROM\n \"SGMP_POC\".\"CZN_CUSTOM_GOODS_HAVE_TXN\" g\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n ON g.\"GUID\" = u.\"GUID\"\n AND g.\"BASE_DT\" = u.\"BASE_DT\"\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_SVC_DIM_BAS\" d\n ON g.\"GOODS_CD\" = d.\"DIM_CD\"\n AND g.\"BASE_DT\" = d.\"BASE_DT\"\n INNER JOIN \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\n ON a.\"USE_YN\" = 'Y'\n AND (\n UPPER(a.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_PREFIX\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_ID\") LIKE UPPER('%카제나%')\n )\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND d.\"DIM_TYPE_DIV_CD\" = 'GOODS_AGG'\n AND d.\"DIM_CTG_2_KR_NM\" = '크리스탈'\n AND g.\"HAVE_CNT\" <> 0\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" alias_check\n WHERE alias_check.\"USE_YN\" = 'Y'\n AND alias_check.\"GAME_PREFIX\" = 'CZN'\n AND (\n UPPER(alias_check.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(alias_check.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n )\n )",
+ "answer_text": "TOTAL_CRYSTAL_HAVE AVG_CRYSTAL_PER_USER=------------------ --------------------",
+ "execution_output": "TOTAL_CRYSTAL_HAVE AVG_CRYSTAL_PER_USER\n------------------ --------------------",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 52736
+ }
+ },
+ {
+ "case_id": "CZN-09",
+ "title": "국가별 평균 크리스탈",
+ "category": "CZN",
+ "scenario_no": 9,
+ "question": "카제나에서 2026년 7월 15일 기준 주요 국가별 표준 AU의 1인당 평균 크리스탈 보유량을 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 10,
+ "scenario": "EBIR-5903",
+ "natural_question": "카제나에서 2026년 4월 23일부터 4월 30일까지 일자별, 주요 국가별 표준 AU 의 1인당 평균 크리스탈 보유량을 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , CASE WHEN T2.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D2.ITG_CD_KR_NM, '기타') END AS COUNTRY_NM\n , COUNT(DISTINCT T1.GUID) AS AU_HOLDER_CNT\n , SUM(T1.HAVE_CNT) / COUNT(DISTINCT T1.GUID) AS AVG_HAVE_CNT_PER_AU\n FROM DW.CZN_CUSTOM_GOODS_HAVE_TXN T1\n INNER JOIN META.CZN_COMN_SVC_DIM_BAS D1\n ON T1.GOODS_CD = D1.DIM_CD\n AND T1.BASE_DT = D1.BASE_DT\n AND D1.DIM_TYPE_DIV_CD = 'GOODS_AGG'\n AND D1.DIM_CTG_2_KR_NM = '크리스탈'\n INNER JOIN DW.CZN_COMN_USER_MST T2\n ON T1.GUID = T2.GUID\n AND T1.BASE_DT = T2.BASE_DT\n AND T2.AU_FLAG = 1\n AND T2.EXPT_USER_YN = 'N'\n LEFT JOIN META.COMN_ITG_CD_BAS D2\n ON T2.LAST_CONN_COUNTRY_CD = D2.ITG_CD\n AND D2.ITG_CD_GROUP_ID = 'G20005'\n AND D2.SVC_DIV_CD = 'STOVE_CHAOSZERO'\n WHERE T1.BASE_DT BETWEEN '2026-04-23' AND '2026-04-30'\n AND T1.HAVE_CNT <> 0\n GROUP BY T1.BASE_DT\n , CASE WHEN T2.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D2.ITG_CD_KR_NM, '기타') END\n ORDER BY 1, 2\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_CUSTOM_GOODS_HAVE_TXN, CZN_COMN_USER_MST, 국가 컬럼, AVG.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_CUSTOM_GOODS_HAVE_TXN",
+ "CZN_COMN_USER_MST",
+ "AVG"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n u.\"LAST_CONN_COUNTRY_CD\" AS COUNTRY_CD,\n c.\"COUNTRY_KR_NM\" AS COUNTRY_NM,\n COUNT(DISTINCT u.\"GUID\") AS AU_COUNT,\n SUM(CAST(g.\"HAVE_CNT\" AS NUMBER)) AS TOTAL_CRYSTAL_HAVE,\n ROUND(SUM(CAST(g.\"HAVE_CNT\" AS NUMBER)) / NULLIF(COUNT(DISTINCT u.\"GUID\"), 0), 2) AS AVG_CRYSTAL_PER_USER\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n INNER JOIN \"SGMP_POC\".\"CZN_CUSTOM_GOODS_HAVE_TXN\" g\n ON u.\"GUID\" = g.\"GUID\"\n AND u.\"BASE_DT\" = g.\"BASE_DT\"\n LEFT JOIN \"SGMP_POC\".\"COMN_COUNTRY_BAS\" c\n ON u.\"LAST_CONN_COUNTRY_CD\" = c.\"COUNTRY_2CHAR_CD\"\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_SVC_DIM_BAS\" d\n ON g.\"GOODS_CD\" = d.\"DIM_CD\"\n AND g.\"BASE_DT\" = d.\"BASE_DT\"\nWHERE\n u.\"BASE_DT\" = TO_DATE('20260715', 'YYYYMMDD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND g.\"HAVE_CNT\" <> 0\n AND d.\"DIM_TYPE_DIV_CD\" = 'GOODS_AGG'\n AND d.\"DIM_CTG_2_KR_NM\" = '크리스탈'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" alias_tbl\n WHERE alias_tbl.\"USE_YN\" = 'Y'\n AND (\n UPPER(alias_tbl.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(alias_tbl.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(alias_tbl.\"GAME_PREFIX\") LIKE UPPER('%CZN%')\n )\n )\nGROUP BY\n u.\"LAST_CONN_COUNTRY_CD\",\n c.\"COUNTRY_KR_NM\"\nORDER BY\n AU_COUNT DESC",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 53921
+ }
+ },
+ {
+ "case_id": "CZN-10",
+ "title": "국가별 표준 AU/플레이타임",
+ "category": "CZN",
+ "scenario_no": 10,
+ "question": "카제나에서 2026년 7월 15일 기준 주요 국가별 표준 AU 수와 평균 플레이타임(초)을 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 11,
+ "scenario": "EBIR-5766",
+ "natural_question": "카제나에서 2026년 5월 8일에 주요 국가별 표준 AU 수와 평균 플레이타임(초)을 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END AS COUNTRY_NM\n , COUNT(DISTINCT T1.GUID) AS AU_CNT\n , AVG(T1.TDAY_PLAY_TIME) AS AVG_PLAY_TIME_SEC\n FROM DW.CZN_COMN_USER_MST T1\n LEFT JOIN META.COMN_ITG_CD_BAS D1\n ON T1.LAST_CONN_COUNTRY_CD = D1.ITG_CD\n AND D1.ITG_CD_GROUP_ID = 'G20005'\n AND D1.SVC_DIV_CD = 'STOVE_CHAOSZERO'\n WHERE T1.BASE_DT = '2026-05-08'\n AND T1.LAST_CONN_DT = '2026-05-08'\n AND T1.AU_FLAG = 1\n AND T1.EXPT_USER_YN = 'N'\n GROUP BY T1.BASE_DT\n , CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END\n ORDER BY 1, 2\n;",
+ "note": "- 국가에 대한 기준이 작성되지 않은 경우에는 최종 접속 국가 기준(표준)으로 집계\n - 그 외 국가 기준이 명시되어 있는 경우에는 해당 컬럼으로 집계\n - ex) 최초 접속 국가"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, COUNT, AVG(TDAY_PLAY_TIME).",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "COUNT",
+ "AVG",
+ "TDAY_PLAY_TIME",
+ "AU_FLAG"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n u.\"LAST_CONN_COUNTRY_CD\" AS COUNTRY_CD,\n COUNT(DISTINCT u.\"GUID\") AS STD_AU_COUNT,\n ROUND(AVG(CAST(u.\"TDAY_PLAY_TIME\" AS NUMBER)), 2) AS AVG_PLAY_TIME_SEC\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND u.\"LAST_CONN_COUNTRY_CD\" IS NOT NULL\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%CZN%')\n )\n )\nGROUP BY\n u.\"LAST_CONN_COUNTRY_CD\"\nORDER BY\n STD_AU_COUNT DESC",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 50281
+ }
+ },
+ {
+ "case_id": "CZN-11",
+ "title": "국가별 누적 접속 14일 이상 표준 AU",
+ "category": "CZN",
+ "scenario_no": 11,
+ "question": "카제나에서 2026년 7월 15일 기준 주요 국가별 누적 접속 14일 이상인 표준 AU 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 12,
+ "scenario": "EBIR-5763",
+ "natural_question": "카제나에서 2026년 5월 8일 기준 주요 국가별·누적 접속 14일 이상인 표준 AU 수를 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END AS COUNTRY_NM\n , COUNT(DISTINCT T1.GUID) AS LOYAL_AU_CNT\n FROM DW.CZN_COMN_USER_MST T1\n LEFT JOIN META.COMN_ITG_CD_BAS D1\n ON T1.LAST_CONN_COUNTRY_CD = D1.ITG_CD\n AND D1.ITG_CD_GROUP_ID = 'G20005'\n AND D1.SVC_DIV_CD = 'STOVE_CHAOSZERO'\n WHERE T1.BASE_DT = '2026-05-08'\n AND T1.AU_FLAG = 1\n AND T1.EXPT_USER_YN = 'N'\n AND T1.ACM_CONN_DCNT >= 14\n GROUP BY T1.BASE_DT\n , CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END\n ORDER BY 1, 2\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_USER_MST, ACM_CONN_DCNT >= 14.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "ACM_CONN_DCNT",
+ "14",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n u.\"LAST_CONN_COUNTRY_CD\" AS COUNTRY_CD,\n COUNT(DISTINCT u.\"GUID\") AS STD_AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND u.\"ACM_CONN_DCNT\" >= 14\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%CZN%')\n )\n AND g.\"GAME_PREFIX\" = 'CZN'\n )\nGROUP BY\n u.\"LAST_CONN_COUNTRY_CD\"\nORDER BY\n u.\"LAST_CONN_COUNTRY_CD\"",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 51846
+ }
+ },
+ {
+ "case_id": "CZN-12",
+ "title": "주요 국가별 사업 AU 재확인",
+ "category": "CZN",
+ "scenario_no": 12,
+ "question": "카제나에서 2026년 7월 15일 기준 주요 국가별 사업 AU를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 13,
+ "scenario": "EBIR-5742",
+ "natural_question": "카제나에서 2026년 4월 2일부터 4월 8일까지 일자별· 주요 국가별 사업 AU 를 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , CASE WHEN T2.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END AS COUNTRY_NM\n , COUNT(1) AS BIZ_AU_CNT\n FROM DW.CZN_CUSTOM_BIZ_USER_TXN T1\n INNER JOIN DW.CZN_COMN_USER_MST T2\n ON T1.GUID = T2.GUID\n AND T1.BASE_DT = T2.BASE_DT\n LEFT JOIN META.COMN_ITG_CD_BAS D1\n ON T2.LAST_CONN_COUNTRY_CD = D1.ITG_CD\n AND D1.ITG_CD_GROUP_ID = 'G20005'\n AND D1.SVC_DIV_CD = 'STOVE_CHAOSZERO'\n WHERE T1.BASE_DT BETWEEN '2026-04-02' AND '2026-04-08'\n AND T1.BIZ_AU_FLAG = 1\n AND T1.EXPT_USER_YN = 'N'\n GROUP BY T1.BASE_DT\n , CASE WHEN T2.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END\n ORDER BY 1, 2\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_CUSTOM_BIZ_USER_TXN와 CZN_COMN_USER_MST 국가 컬럼 조인.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_CUSTOM_BIZ_USER_TXN",
+ "CZN_COMN_USER_MST",
+ "BIZ_AU_FLAG"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n u.\"LAST_CONN_COUNTRY_CD\" AS COUNTRY_CD,\n c.\"COUNTRY_KR_NM\" AS COUNTRY_NM,\n COUNT(DISTINCT b.\"GUID\") AS BIZ_AU_CNT\nFROM\n \"SGMP_POC\".\"CZN_CUSTOM_BIZ_USER_TXN\" b\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n ON b.\"GUID\" = u.\"GUID\"\n AND u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n LEFT JOIN \"SGMP_POC\".\"COMN_COUNTRY_BAS\" c\n ON u.\"LAST_CONN_COUNTRY_CD\" = c.\"COUNTRY_2CHAR_CD\"\nWHERE\n b.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND b.\"BIZ_AU_FLAG\" = 1\n AND b.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" ga\n WHERE ga.\"USE_YN\" = 'Y'\n AND UPPER(ga.\"GAME_NM\") LIKE UPPER('%카제나%')\n AND ga.\"GAME_PREFIX\" = 'CZN'\n )\nGROUP BY\n u.\"LAST_CONN_COUNTRY_CD\",\n c.\"COUNTRY_KR_NM\"\nORDER BY\n BIZ_AU_CNT DESC",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 51319
+ }
+ },
+ {
+ "case_id": "CZN-13",
+ "title": "에테르 사용량/사용 유저",
+ "category": "CZN",
+ "scenario_no": 13,
+ "question": "카제나에서 2026년 7월 15일 기준 에테르 총 사용량과 사용 유저 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 14,
+ "scenario": "EBIR-5742",
+ "natural_question": "카제나에서 2026년 4월 2일부터 4월 8일까지 일자별 에테르 총 사용량과 사용 유저 수를 알려줘.",
+ "sample_query": "SELECT T1.BASE_DT\n , COUNT(DISTINCT T1.GUID) AS USER_CNT\n , SUM(T1.GOODS_CHANGE_CNT) AS AETHER_USE_SUM\n FROM DW.CZN_CUSTOM_GOODS_CHANGE_TXN T1\n INNER JOIN META.CZN_COMN_SVC_DIM_BAS D1\n ON T1.GOODS_CD = D1.DIM_CD\n AND T1.BASE_DT = D1.BASE_DT\n AND D1.DIM_TYPE_DIV_CD = 'GOODS_AGG'\n AND D1.DIM_CTG_2_KR_NM = '에테르'\n AND D1.USE_YN = 'Y'\n INNER JOIN DW.CZN_COMN_USER_MST T2\n ON T1.GUID = T2.GUID\n AND T1.BASE_DT = T2.BASE_DT\n AND T2.EXPT_USER_YN = 'N'\n WHERE T1.BASE_DT BETWEEN '2026-04-02' AND '2026-04-08'\n AND T1.CHANGE_TYPE_CD = 'USE'\n GROUP BY T1.BASE_DT\n ORDER BY 1\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_CUSTOM_GOODS_CHANGE_TXN와 CZN_COMN_SVC_DIM_BAS GOODS_CD/DIM_CD 조인, CHANGE_TYPE_CD='USE', GOODS_CHANGE_CNT, EXPT_USER_YN='N'.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_CUSTOM_GOODS_CHANGE_TXN",
+ "CZN_COMN_SVC_DIM_BAS",
+ "CZN_COMN_USER_MST",
+ "GOODS_CD",
+ "GOODS_CHANGE_CNT",
+ "CHANGE_TYPE_CD",
+ "USE",
+ "EXPT_USER_YN",
+ "COUNT",
+ "DISTINCT",
+ "SUM"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(g.\"GOODS_CHANGE_CNT\" AS NUMBER)) AS TOTAL_ETHER_USAGE,\n COUNT(DISTINCT u.\"GUID\") AS USAGE_USER_COUNT\nFROM\n \"SGMP_POC\".\"CZN_CUSTOM_GOODS_CHANGE_TXN\" g\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n ON g.\"GUID\" = u.\"GUID\"\n AND g.\"BASE_DT\" = u.\"BASE_DT\"\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_SVC_DIM_BAS\" d\n ON g.\"GOODS_CD\" = d.\"DIM_CD\"\n AND g.\"BASE_DT\" = d.\"BASE_DT\"\nWHERE\n g.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND g.\"CHANGE_TYPE_CD\" = 'USE'\n AND d.\"DIM_TYPE_DIV_CD\" = 'GOODS_AGG'\n AND UPPER(d.\"DIM_CTG_2_KR_NM\") = UPPER('에테르')\n AND d.\"USE_YN\" = 'Y'\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\n WHERE a.\"USE_YN\" = 'Y'\n AND a.\"GAME_PREFIX\" = 'CZN'\n AND UPPER(a.\"GAME_NM\") LIKE UPPER('%카제나%')\n )",
+ "answer_text": "TOTAL_ETHER_USAGE USAGE_USER_COUNT\n----------------- ----------------\n 0",
+ "execution_output": "TOTAL_ETHER_USAGE USAGE_USER_COUNT\n----------------- ----------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 44089
+ }
+ },
+ {
+ "case_id": "CZN-14",
+ "title": "이탈 유저",
+ "category": "CZN",
+ "scenario_no": 14,
+ "question": "카제나에서 2026년 7월 15일 기준 이탈 유저 수 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 15,
+ "scenario": "",
+ "natural_question": "카제나에서 2026년 5월 17일 이탈 유저 수 알려줘",
+ "sample_query": "SELECT COUNT(*)\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = '2026-05-17'\n AND USER_TYPE_ITG_CD = 'CHURN'\n AND EXPT_USER_YN = 'N'\n AND STD_USER_YN = 'Y'",
+ "note": "이탈 유저 기준: 마지막 접속 후 7일이 초과한 유저(GUID)"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, USER_TYPE_ITG_CD='CHURN'.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "USER_TYPE_ITG_CD",
+ "CHURN",
+ "COUNT"
+ ],
+ "recommended_sql_terms": [
+ "EXPT_USER_YN"
+ ]
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS CHURN_USER_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"USER_TYPE_ITG_CD\" = 'CHURN'\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "CHURN_USER_COUNT=10",
+ "execution_output": "CHURN_USER_COUNT\n----------------\n 10",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 49275
+ }
+ },
+ {
+ "case_id": "CZN-15",
+ "title": "복귀 유저",
+ "category": "CZN",
+ "scenario_no": 15,
+ "question": "카제나에서 2026년 7월 15일 기준 복귀 유저 수 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 16,
+ "scenario": "",
+ "natural_question": "카제나에서 2026년 5월 17일 복귀 유저 수 알려줘",
+ "sample_query": "SELECT COUNT(*)\n FROM DW.CZN_COMN_USER_MST\n WHERE BASE_DT = '2026-05-17'\n AND USER_TYPE_ITG_CD = 'RETURN'\n AND EXPT_USER_YN = 'N'\n AND STD_USER_YN = 'Y'",
+ "note": "복귀 유저 기준: 이탈 후 복귀(AU로 잡힘)한 유저"
+ },
+ "expected_focus": "CZN_COMN_USER_MST, USER_TYPE_ITG_CD 기반 복귀 유저.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_USER_MST",
+ "USER_TYPE_ITG_CD",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT u.\"GUID\") AS RETURN_USER_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND u.\"USER_TYPE_ITG_CD\" = 'RETURN'\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND g.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "RETURN_USER_COUNT=0",
+ "execution_output": "RETURN_USER_COUNT\n-----------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 48326
+ }
+ },
+ {
+ "case_id": "CZN-16",
+ "title": "패키지 구매 유저",
+ "category": "CZN",
+ "scenario_no": 16,
+ "question": "카제나에서 2026년 7월 15일 기준 '신임 함장 지원 키트 Ⅰ' 패키지를 구매한 유저 수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 17,
+ "scenario": "EBIR-5995",
+ "natural_question": "카제나에서 2026년 5월 8일부터 5월 12일까지 '신임 함장 지원 키트 Ⅰ' 패키지를 구매한 유저 수를 알려줘.",
+ "sample_query": "SELECT COUNT(DISTINCT T1.GUID) AS USER_CNT\n FROM DW.COMN_SALES_TXN T1\n INNER JOIN META.COMN_SALES_PRODUCT_DISP_BAS D1\n ON T1.PRODUCT_ID = D1.PRODUCT_ID\n AND D1.GAME_ID = 'STOVE_CHAOSZERO'\n AND D1.PRODUCT_KR_NM = '신임 함장 지원 키트 Ⅰ'\n WHERE T1.GAME_ID = 'STOVE_CHAOSZERO'\n AND T1.BASE_DT BETWEEN '2026-05-08' AND '2026-05-12'\n AND T1.EXPT_USER_YN = 'N'\n;",
+ "note": "-- 상품명 확인 방법\nSELECT *\n FROM META.COMN_SALES_PRODUCT_DISP_BAS\n WHERE GAME_ID = 'STOVE_CHAOSZERO'"
+ },
+ "expected_focus": "COMN_SALES_TXN와 COMN_SALES_PRODUCT_DISP_BAS 상품명 조인, COUNT(DISTINCT GUID).",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "COMN_SALES_TXN",
+ "COMN_SALES_PRODUCT_DISP_BAS",
+ "COUNT",
+ "DISTINCT",
+ "GUID"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT s.\"GUID\") AS PURCHASER_COUNT\nFROM\n \"SGMP_POC\".\"COMN_SALES_TXN\" s\n INNER JOIN \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n ON s.\"GAME_ID\" = g.\"GAME_ID\"\n INNER JOIN \"SGMP_POC\".\"COMN_SALES_PRODUCT_DISP_BAS\" p\n ON s.\"GAME_ID\" = p.\"GAME_ID\"\n AND s.\"PRODUCT_ID\" = p.\"PRODUCT_ID\"\nWHERE\n g.\"USE_YN\" = 'Y'\n AND UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n AND UPPER(p.\"PRODUCT_KR_NM\") = UPPER('신임 함장 지원 키트 Ⅰ')\n AND TRUNC(s.\"PAYMT_DTM\") = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND s.\"EXPT_USER_YN\" = 'N'",
+ "answer_text": "PURCHASER_COUNT=0",
+ "execution_output": "PURCHASER_COUNT\n---------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 48855
+ }
+ },
+ {
+ "case_id": "CZN-17",
+ "title": "캐릭터 AU 크리스탈",
+ "category": "CZN",
+ "scenario_no": 17,
+ "question": "카제나에서 2026년 7월 15일 기준 캐릭터 AU의 크리스탈 총 보유량, 보유 캐릭터 수, 1인당 평균 보유량을 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 18,
+ "scenario": "EBIR-5916",
+ "natural_question": "카제나에서 2026년 5월 21일 기준 캐릭터 AU의 크리스탈 총 보유량, 보유 캐릭터 수, 1인당 평균 보유량을 알려줘.",
+ "sample_query": "SELECT SUM(T1.HAVE_CNT) AS HAVE_SUM\n , COUNT(DISTINCT T1.CUID) AS CUID_CNT\n , SUM(T1.HAVE_CNT) / COUNT(DISTINCT T1.CUID) AS AVG_HAVE_CNT\n FROM DW.CZN_CUSTOM_GOODS_HAVE_TXN T1\n INNER JOIN DW.CZN_COMN_CHARACTER_MST T2\n ON T1.BASE_DT = T2.BASE_DT\n AND T1.CUID = T2.CUID\n INNER JOIN META.CZN_COMN_SVC_DIM_BAS D1\n ON T1.GOODS_CD = D1.DIM_CD\n AND T1.BASE_DT = D1.BASE_DT\n AND D1.DIM_TYPE_DIV_CD = 'GOODS_AGG'\n AND D1.DIM_CTG_2_KR_NM = '크리스탈'\n WHERE T1.BASE_DT = '2026-05-21'\n AND T2.AU_FLAG = 1\n AND T2.EXPT_USER_YN = 'N'\n AND T1.HAVE_CNT <> 0\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_CHARACTER_MST, CZN_CUSTOM_GOODS_HAVE_TXN, CZN_COMN_SVC_DIM_BAS, AU_FLAG=1, EXPT_USER_YN='N', SUM(HAVE_CNT)/COUNT(DISTINCT CUID).",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_CHARACTER_MST",
+ "CZN_CUSTOM_GOODS_HAVE_TXN",
+ "CZN_COMN_SVC_DIM_BAS",
+ "CUID",
+ "HAVE_CNT",
+ "AU_FLAG",
+ "EXPT_USER_YN",
+ "SUM",
+ "COUNT",
+ "DISTINCT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n SUM(CAST(g.\"HAVE_CNT\" AS NUMBER)) AS TOTAL_CRYSTAL_HAVE_CNT,\n COUNT(DISTINCT g.\"CUID\") AS HOLDING_CHARACTER_CNT,\n SUM(CAST(g.\"HAVE_CNT\" AS NUMBER)) / NULLIF(COUNT(DISTINCT g.\"CUID\"), 0) AS AVG_HAVE_CNT_PER_CHARACTER\nFROM\n \"SGMP_POC\".\"CZN_CUSTOM_GOODS_HAVE_TXN\" g\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_CHARACTER_MST\" c\n ON g.\"CUID\" = c.\"CUID\"\n AND g.\"BASE_DT\" = c.\"BASE_DT\"\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_SVC_DIM_BAS\" d\n ON g.\"GOODS_CD\" = d.\"DIM_CD\"\n AND g.\"BASE_DT\" = d.\"BASE_DT\"\nWHERE\n g.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND c.\"AU_FLAG\" = 1\n AND c.\"EXPT_USER_YN\" = 'N'\n AND g.\"HAVE_CNT\" <> 0\n AND d.\"DIM_TYPE_DIV_CD\" = 'GOODS_AGG'\n AND d.\"DIM_CTG_2_KR_NM\" = '크리스탈'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" a\n WHERE a.\"USE_YN\" = 'Y'\n AND (\n UPPER(a.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_PREFIX\") LIKE UPPER('%카제나%')\n OR UPPER(a.\"GAME_ID\") LIKE UPPER('%카제나%')\n )\n AND a.\"GAME_PREFIX\" = 'CZN'\n )",
+ "answer_text": "TOTAL_CRYSTAL_HAVE_CNT HOLDING_CHARACTER_CNT AVG_HAVE_CNT_PER_CHARACTER\n---------------------- --------------------- --------------------------\n 0",
+ "execution_output": "TOTAL_CRYSTAL_HAVE_CNT HOLDING_CHARACTER_CNT AVG_HAVE_CNT_PER_CHARACTER\n---------------------- --------------------- --------------------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 43233
+ }
+ },
+ {
+ "case_id": "CZN-18",
+ "title": "30레벨 이상 캐릭터 AU 국가별",
+ "category": "CZN",
+ "scenario_no": 18,
+ "question": "카제나에서 2026년 7월 15일 기준 30레벨 이상 캐릭터 AU를 주요 국가별로 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 19,
+ "scenario": "EBIR-5608",
+ "natural_question": "카제나에서 2026년 4월 24일 기준 30레벨 이상 캐릭터 AU를 주요 국가별로 알려줘.",
+ "sample_query": "SELECT CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END AS COUNTRY_NM\n , COUNT(DISTINCT CUID) AS CHAR_CNT\n FROM DW.CZN_COMN_CHARACTER_MST T1\n LEFT JOIN META.COMN_ITG_CD_BAS D1\n ON T1.LAST_CONN_COUNTRY_CD = D1.ITG_CD\n AND D1.ITG_CD_GROUP_ID = 'G20005'\n AND D1.SVC_DIV_CD = 'STOVE_CHAOSZERO'\n WHERE T1.BASE_DT = '2026-04-24'\n AND T1.AU_FLAG = 1\n AND T1.LEVEL >= 30\n AND T1.EXPT_USER_YN = 'N'\n GROUP BY CASE WHEN T1.LAST_CONN_COUNTRY_CD = '_' THEN '(정보없음)'\n ELSE COALESCE(D1.ITG_CD_KR_NM, '기타') END\n ORDER BY 2 DESC\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_CHARACTER_MST, LEVEL_COL >= 30, AU_FLAG, 국가 컬럼.",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_CHARACTER_MST",
+ "LEVEL_COL",
+ "30",
+ "AU_FLAG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n u.\"LAST_CONN_COUNTRY_CD\" AS COUNTRY_CD,\n COUNT(DISTINCT c.\"CUID\") AS CHARACTER_AU_COUNT\nFROM\n \"SGMP_POC\".\"CZN_COMN_CHARACTER_MST\" c\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n ON c.\"GUID\" = u.\"GUID\"\n AND c.\"BASE_DT\" = u.\"BASE_DT\"\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND c.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND c.\"LEVEL_COL\" >= 30\n AND c.\"AU_FLAG\" = 1\n AND c.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" ga\n WHERE ga.\"GAME_PREFIX\" = 'CZN'\n AND ga.\"USE_YN\" = 'Y'\n )\nGROUP BY\n u.\"LAST_CONN_COUNTRY_CD\"\nORDER BY\n CHARACTER_AU_COUNT DESC",
+ "answer_text": "결과 행 없음",
+ "execution_output": "",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 50324
+ }
+ },
+ {
+ "case_id": "CZN-19",
+ "title": "레벨 12 이상 캐릭터/평균 접속일수",
+ "category": "CZN",
+ "scenario_no": 19,
+ "question": "카제나에서 2026년 7월 15일 AU 중 레벨 12 이상 캐릭터의 캐릭터 수와 평균 누적 접속 일수를 알려줘.",
+ "source": {
+ "workbook": "카제나 샘플 쿼리_20260707 (1).xlsx",
+ "sheet": "text2sql 시나리오 (심플)",
+ "excel_row": 20,
+ "scenario": "EBIR-5186",
+ "natural_question": "카제나에서 2026년 4월 2일 AU 중 레벨 12 이상 캐릭터의 캐릭터 수와 평균 누적 접속 일수를 알려줘.",
+ "sample_query": "SELECT COUNT(DISTINCT CUID) AS CHAR_CNT\n , AVG(ACM_CONN_DCNT) AS AVG_ACM_CONN_DCNT\n FROM DW.CZN_COMN_CHARACTER_MST\n WHERE BASE_DT = '2026-04-02'\n AND AU_FLAG = 1\n AND LEVEL >= 12\n AND EXPT_USER_YN = 'N'\n;",
+ "note": ""
+ },
+ "expected_focus": "CZN_COMN_CHARACTER_MST와 CZN_COMN_USER_MST 조인, LEVEL_COL >= 12, AVG(ACM_CONN_DCNT).",
+ "support_level": "SUPPORTED",
+ "evaluation_rule": {
+ "required_sql_terms": [
+ "CZN_COMN_CHARACTER_MST",
+ "CZN_COMN_USER_MST",
+ "LEVEL_COL",
+ "12",
+ "AVG",
+ "COUNT"
+ ],
+ "recommended_sql_terms": []
+ },
+ "historical_answer": {
+ "generated_at": "2026-07-21T14:44:50+09:00",
+ "profile": "SGMP_POC_HAIKU45",
+ "model": "anthropic/claude-haiku-4.5 via OpenRouter",
+ "generated_sql": "SELECT\n COUNT(DISTINCT c.\"CUID\") AS CHARACTER_COUNT,\n AVG(u.\"ACM_CONN_DCNT\") AS AVG_CUMULATIVE_CONN_DAYS\nFROM\n \"SGMP_POC\".\"CZN_COMN_CHARACTER_MST\" c\n INNER JOIN \"SGMP_POC\".\"CZN_COMN_USER_MST\" u\n ON c.\"GUID\" = u.\"GUID\"\n AND c.\"BASE_DT\" = u.\"BASE_DT\"\nWHERE\n u.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND c.\"BASE_DT\" = TO_DATE('2026-07-15', 'YYYY-MM-DD')\n AND c.\"LEVEL_COL\" >= 12\n AND c.\"AU_FLAG\" = 1\n AND c.\"EXPT_USER_YN\" = 'N'\n AND u.\"AU_FLAG\" = 1\n AND u.\"EXPT_USER_YN\" = 'N'\n AND EXISTS (\n SELECT 1\n FROM \"SGMP_POC\".\"COMN_GAME_ALIAS_BAS\" g\n WHERE g.\"USE_YN\" = 'Y'\n AND (\n UPPER(g.\"GAME_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_ALIAS_NM\") LIKE UPPER('%카제나%')\n OR UPPER(g.\"GAME_PREFIX\") LIKE UPPER('%CZN%')\n )\n )",
+ "answer_text": "CHARACTER_COUNT AVG_CUMULATIVE_CONN_DAYS\n--------------- ------------------------\n 0",
+ "execution_output": "CHARACTER_COUNT AVG_CUMULATIVE_CONN_DAYS\n--------------- ------------------------\n 0",
+ "execution_status": "PASS",
+ "judgment_status": "PASS",
+ "judgment_reason": "Required SQL terms are present.",
+ "duration_ms": 50512
+ }
+ }
+ ]
+}
diff --git a/poc4_active_source_20260714/scripts/sync_smilegate_qa_history.py b/poc4_active_source_20260714/scripts/sync_smilegate_qa_history.py
new file mode 100644
index 0000000..5109826
--- /dev/null
+++ b/poc4_active_source_20260714/scripts/sync_smilegate_qa_history.py
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+"""Create and seed the Smilegate customer QA benchmark history tables."""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+import sys
+
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from src.poc4.qa_history_store import QaHistoryStore, ensure_schema
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--benchmark",
+ type=Path,
+ default=ROOT / "config" / "smilegate_qa_benchmark.json",
+ help="Customer Excel benchmark JSON generated from the approved QA report.",
+ )
+ parser.add_argument(
+ "--env-file",
+ type=Path,
+ default=None,
+ help="Optional environment file containing the QA DB connection settings.",
+ )
+ args = parser.parse_args()
+ if not args.benchmark.is_file():
+ raise SystemExit(f"Benchmark file not found: {args.benchmark}")
+ store = QaHistoryStore(env_file=args.env_file)
+ ensure_schema(store)
+ question_count, historical_insert_count = store.seed_benchmark(args.benchmark)
+ print(
+ "qa_history_sync"
+ f" questions={question_count}"
+ f" historical_answers_inserted={historical_insert_count}"
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/poc4_active_source_20260714/src/poc4/qa_history.py b/poc4_active_source_20260714/src/poc4/qa_history.py
new file mode 100644
index 0000000..7d02892
--- /dev/null
+++ b/poc4_active_source_20260714/src/poc4/qa_history.py
@@ -0,0 +1,289 @@
+"""Customer QA benchmark parsing and deterministic SQL evaluation."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import hashlib
+import json
+from pathlib import Path
+import re
+from typing import Any, Mapping
+
+
+class QaBenchmarkError(RuntimeError):
+ """Raised when the QA benchmark source cannot be used safely."""
+
+
+@dataclass(frozen=True)
+class QaQuestion:
+ question_id: int | None
+ question_code: str
+ category: str
+ title: str
+ question_text: str
+ source_document: str
+ source_sheet: str
+ source_row: int | None
+ source_scenario: str
+ sample_sql: str
+ expected_focus: str
+ baseline_sql: str
+ baseline_answer: str
+ support_level: str
+ evaluation_rule: Mapping[str, Any]
+ last_judgment_status: str = ""
+ last_evaluated_at: str = ""
+
+
+@dataclass(frozen=True)
+class QaJudgment:
+ status: str
+ reason: str
+
+
+def question_fingerprint(question_text: str) -> str:
+ normalized = " ".join(str(question_text or "").split()).casefold()
+ return hashlib.sha256(normalized.encode("utf-8")).hexdigest()
+
+
+def _compact_text(value: object) -> str:
+ return str(value or "").strip()
+
+
+def _string_list(value: object) -> tuple[str, ...]:
+ if not isinstance(value, list):
+ return ()
+ return tuple(_compact_text(item) for item in value if _compact_text(item))
+
+
+def load_benchmark_questions(path: Path) -> tuple[QaQuestion, ...]:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, UnicodeError, ValueError) as exc:
+ raise QaBenchmarkError(f"질답 기준 파일을 읽지 못했습니다: {path}") from exc
+ rows = payload.get("scenarios") if isinstance(payload, Mapping) else None
+ if not isinstance(rows, list):
+ raise QaBenchmarkError("질답 기준 파일에 scenarios 배열이 필요합니다.")
+
+ questions: list[QaQuestion] = []
+ seen_codes: set[str] = set()
+ for row in rows:
+ if not isinstance(row, Mapping):
+ raise QaBenchmarkError("질답 기준의 각 시나리오는 객체여야 합니다.")
+ source = row.get("source") if isinstance(row.get("source"), Mapping) else {}
+ history = (
+ row.get("historical_answer")
+ if isinstance(row.get("historical_answer"), Mapping)
+ else {}
+ )
+ code = _compact_text(row.get("case_id")).upper()
+ question_text = _compact_text(row.get("question"))
+ if not code or not question_text:
+ raise QaBenchmarkError("각 질답 기준에는 case_id와 question이 필요합니다.")
+ if code in seen_codes:
+ raise QaBenchmarkError(f"중복된 질답 case_id입니다: {code}")
+ evaluation_rule = row.get("evaluation_rule")
+ if not isinstance(evaluation_rule, Mapping):
+ evaluation_rule = {}
+ questions.append(
+ QaQuestion(
+ question_id=None,
+ question_code=code,
+ category=_compact_text(row.get("category")) or "GENERAL",
+ title=_compact_text(row.get("title")) or code,
+ question_text=question_text,
+ source_document=_compact_text(source.get("workbook")),
+ source_sheet=_compact_text(source.get("sheet")),
+ source_row=_number_or_none(source.get("excel_row")),
+ source_scenario=_compact_text(source.get("scenario")),
+ sample_sql=_compact_text(source.get("sample_query")),
+ expected_focus=_compact_text(row.get("expected_focus")),
+ baseline_sql=_compact_text(history.get("generated_sql")),
+ baseline_answer=_compact_text(history.get("answer_text")),
+ support_level=_compact_text(row.get("support_level")).upper() or "UNKNOWN",
+ evaluation_rule={
+ "required_sql_terms": list(
+ _string_list(evaluation_rule.get("required_sql_terms"))
+ ),
+ "recommended_sql_terms": list(
+ _string_list(evaluation_rule.get("recommended_sql_terms"))
+ ),
+ },
+ )
+ )
+ seen_codes.add(code)
+ return tuple(questions)
+
+
+def question_from_record(record: Mapping[str, Any]) -> QaQuestion:
+ rule = record.get("evaluation_rule")
+ if isinstance(rule, str):
+ try:
+ rule = json.loads(rule)
+ except ValueError:
+ rule = {}
+ if not isinstance(rule, Mapping):
+ rule = {}
+ return QaQuestion(
+ question_id=_number_or_none(record.get("question_id")),
+ question_code=_compact_text(record.get("question_code")),
+ category=_compact_text(record.get("category")) or "GENERAL",
+ title=_compact_text(record.get("title")) or _compact_text(record.get("question_code")),
+ question_text=_compact_text(record.get("question_text")),
+ source_document=_compact_text(record.get("source_document")),
+ source_sheet=_compact_text(record.get("source_sheet")),
+ source_row=_number_or_none(record.get("source_row")),
+ source_scenario=_compact_text(record.get("source_scenario")),
+ sample_sql=_compact_text(record.get("sample_sql")),
+ expected_focus=_compact_text(record.get("expected_focus")),
+ baseline_sql=_compact_text(record.get("baseline_sql")),
+ baseline_answer=_compact_text(record.get("baseline_answer")),
+ support_level=_compact_text(record.get("support_level")).upper() or "UNKNOWN",
+ evaluation_rule={
+ "required_sql_terms": list(
+ _string_list(rule.get("required_sql_terms"))
+ ),
+ "recommended_sql_terms": list(
+ _string_list(rule.get("recommended_sql_terms"))
+ ),
+ },
+ last_judgment_status=_compact_text(record.get("last_judgment_status")),
+ last_evaluated_at=_compact_text(record.get("last_evaluated_at")),
+ )
+
+
+def _number_or_none(value: object) -> int | None:
+ if value is None or value == "":
+ return None
+ try:
+ return int(value)
+ except (TypeError, ValueError):
+ return None
+
+
+def _append_issue(issues: list[str], condition: bool, message: str) -> None:
+ if condition:
+ issues.append(message)
+
+
+def evaluate_sql(
+ question: QaQuestion | None,
+ generated_sql: str,
+ *,
+ execution_succeeded: bool,
+ error_text: str = "",
+) -> QaJudgment:
+ """Evaluate the generated SQL against the customer-approved benchmark rule."""
+ if question is None or not question.question_code:
+ return QaJudgment(
+ status="REVIEW",
+ reason="자유 질의입니다. 고객 기준 정답 시나리오와 연결되지 않아 수동 검토가 필요합니다.",
+ )
+
+ sql = _compact_text(generated_sql)
+ upper_sql = sql.upper()
+ lower_sql = sql.lower()
+ execution_status = "PASS" if execution_succeeded else "FAIL_EXECUTION"
+ issues: list[str] = []
+ failure_markers = ("could not be generated", "exception encountered", "invalid identifier", "ora-")
+ has_failure_text = any(marker in lower_sql for marker in failure_markers)
+ required = _string_list(question.evaluation_rule.get("required_sql_terms"))
+ recommended = _string_list(question.evaluation_rule.get("recommended_sql_terms"))
+ missing_required = [term for term in required if term.upper() not in upper_sql]
+ missing_recommended = [term for term in recommended if term.upper() not in upper_sql]
+
+ if not execution_succeeded:
+ issues.append(f"실행 상태가 {execution_status}입니다.")
+ if not sql:
+ issues.append("생성 SQL이 없습니다.")
+ if has_failure_text:
+ issues.append("생성 SQL에 오류 또는 생성 실패 문구가 포함되어 있습니다.")
+ if missing_required:
+ issues.append("필수 SQL 요소 누락: " + ", ".join(missing_required))
+ if missing_recommended:
+ issues.append("권장 SQL 요소 누락: " + ", ".join(missing_recommended))
+ _append_issue(
+ issues,
+ bool(re.search(r'_[A-Z0-9]*YN"\s*=\s*\'1\'', sql, flags=re.IGNORECASE)),
+ "*_YN 컬럼은 샘플 메타데이터의 Y/N 값으로 비교해야 합니다.",
+ )
+ _append_issue(
+ issues,
+ bool(re.search(r'_[A-Z0-9]*FLAG"\s*=\s*\'Y\'', sql, flags=re.IGNORECASE)),
+ "*_FLAG 컬럼은 샘플 메타데이터의 0/1 값으로 비교해야 합니다.",
+ )
+
+ support = question.support_level
+ if support == "UNSUPPORTED":
+ uses_alias_lookup = "COMN_GAME_ALIAS_BAS" in upper_sql
+ substitutes_sample_game = "STOVE_CHAOSZERO" in upper_sql
+ if not sql and any(marker in error_text.lower() for marker in failure_markers):
+ return QaJudgment("PASS", "미지원 게임 질문이 실행 가능한 SQL로 변환되지 않았습니다. 기대한 안전 차단입니다.")
+ if uses_alias_lookup and not substitutes_sample_game and execution_succeeded:
+ return QaJudgment("PASS", "미지원 게임을 별칭 테이블로만 확인했고 샘플 게임 ID를 임의 대입하지 않았습니다.")
+ return QaJudgment("FAIL", "미지원 게임이 안전한 별칭 조회로 제한되지 않았거나 실행에 실패했습니다.")
+
+ if not execution_succeeded or not sql or has_failure_text or missing_required:
+ return QaJudgment("FAIL", "\n".join(issues) or "필수 SQL 또는 실행 검증에 실패했습니다.")
+
+ _apply_case_specific_rules(question.question_code, sql, upper_sql, issues)
+ if any(issue.startswith("필수") or issue.startswith("월간") or issue.startswith("일별") or issue.startswith("주간") or issue.startswith("CZN-") for issue in issues):
+ return QaJudgment("FAIL", "\n".join(issues))
+ if support == "PARTIAL":
+ issues.append("지원 범위가 일부인 질문이므로 결과 범위를 함께 검토해야 합니다.")
+ if issues:
+ return QaJudgment("WARN", "\n".join(issues))
+ return QaJudgment("PASS", "필수 테이블·컬럼·집계 조건과 실행 결과를 확인했습니다.")
+
+
+def _apply_case_specific_rules(code: str, sql: str, upper_sql: str, issues: list[str]) -> None:
+ if code == "STD-26":
+ _append_issue(issues, "TRUNC(DATE" in upper_sql, "월간 NRU는 DATE 리터럴이 아니라 BASE_DT 기준 월 범위를 사용해야 합니다.")
+ _append_issue(issues, "MAX" not in upper_sql or "BASE_DT" not in upper_sql, "월간 NRU는 최신 MAX(BASE_DT) 스냅샷을 기준으로 해야 합니다.")
+ _append_issue(issues, "RAW_NRU_DT" in upper_sql and "TRUNC(" not in upper_sql, "월간 NRU의 RAW_NRU_DT 기간은 BASE_DT 기준으로 계산해야 합니다.")
+ elif code == "STD-27":
+ _append_issue(issues, "AU_FLAG" in upper_sql, "월간 AU는 AU_FLAG가 아니라 LAST_CONN_DT 전월 조건으로 계산해야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("MAX", "BASE_DT", "ADD_MONTHS")), "월간 AU는 최신 BASE_DT의 전월을 기준으로 해야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("STD_USER_YN", "EXPT_USER_YN", "LAST_CONN_DT")), "월간 AU에는 STD_USER_YN, EXPT_USER_YN, LAST_CONN_DT 조건이 필요합니다.")
+ elif code == "STD-22":
+ _append_issue(issues, "NRU_FLAG" not in upper_sql, "일별 NRU는 NRU_FLAG=1을 사용해야 합니다.")
+ elif code == "STD-25":
+ _append_issue(issues, "LAST_CONN_DT" not in upper_sql, "최근 7일 AU는 LAST_CONN_DT 기간 조건을 사용해야 합니다.")
+ _append_issue(issues, "AU_FLAG" in upper_sql, "최근 7일 AU는 AU_FLAG로 제한하면 안 됩니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("STD_USER_YN", "EXPT_USER_YN")), "최근 7일 AU에는 STD_USER_YN과 EXPT_USER_YN 조건이 필요합니다.")
+ _append_issue(issues, "GROUP BY" in upper_sql, "최근 7일 AU는 일자별 목록이 아니라 단일 집계여야 합니다.")
+ elif code == "STD-28":
+ _append_issue(issues, any(term not in upper_sql for term in ("STD_USER_YN", "EXPT_USER_YN")), "전체 유저 수에는 STD_USER_YN과 EXPT_USER_YN 조건이 필요합니다.")
+ _append_issue(issues, "RU_FLAG" in upper_sql, "이 기준의 전체 유저 수는 RU_FLAG가 아니라 STD_USER_YN으로 계산해야 합니다.")
+ elif code in {"CZN-07", "CZN-08", "CZN-13", "CZN-17"}:
+ _apply_goods_rules(code, sql, upper_sql, issues)
+
+
+def _apply_goods_rules(code: str, sql: str, upper_sql: str, issues: list[str]) -> None:
+ dimension_name_terms = ("DIM_KR_NM", "DIM_EN_NM", "DIM_CTG", "CD_DESC", "CD_DTL_DESC")
+ has_dimension_name = any(term in upper_sql for term in dimension_name_terms)
+ if code in {"CZN-07", "CZN-08", "CZN-17"}:
+ _append_issue(issues, "CZN_COMN_SVC_DIM_BAS" not in upper_sql or not has_dimension_name, f"{code}은 CZN_COMN_SVC_DIM_BAS의 이름/분류 컬럼으로 재화명을 해석해야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("DIM_TYPE_DIV_CD", "DIM_CTG")) or "크리스탈" not in sql, f"{code}은 GOODS_AGG와 크리스탈 분류 조건을 사용해야 합니다.")
+ _append_issue(issues, "HAVE_CNT" in upper_sql and "<>" not in upper_sql and "!=" not in upper_sql, f"{code}은 HAVE_CNT <> 0으로 0 보유량을 제외해야 합니다.")
+ if code == "CZN-07":
+ _append_issue(issues, any(term not in upper_sql for term in ("CZN_COMN_USER_MST", "EXPT_USER_YN")), "CZN-07은 유저 마스터를 조인하고 제외 유저를 필터링해야 합니다.")
+ _append_issue(issues, "GROUP BY" not in upper_sql or "BASE_DT" not in upper_sql, "CZN-07은 기간별 일자 집계를 위해 BASE_DT GROUP BY가 필요합니다.")
+ elif code == "CZN-08":
+ _append_issue(issues, any(term not in upper_sql for term in ("AU_FLAG", "EXPT_USER_YN")), "CZN-08은 표준 AU와 제외 유저 조건이 필요합니다.")
+ _append_issue(issues, "STD_USER_YN" in upper_sql, "CZN-08 기준에는 STD_USER_YN을 추가하면 모집단이 과도하게 좁아집니다.")
+ _append_issue(issues, "AVG(" in upper_sql, "CZN-08 1인당 평균은 AVG(HAVE_CNT)가 아니라 합계/고유 유저 수여야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("SUM", "COUNT", "DISTINCT", "/")), "CZN-08 평균은 SUM(HAVE_CNT)/COUNT(DISTINCT GUID)로 계산해야 합니다.")
+ elif code == "CZN-13":
+ _append_issue(issues, "CZN_CUSTOM_GOODS_HAVE_TXN" in upper_sql, "CZN-13은 보유 스냅샷이 아니라 재화 변동 테이블을 사용해야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("CZN_CUSTOM_GOODS_CHANGE_TXN", "GOODS_CHANGE_CNT")), "CZN-13은 GOODS_CHANGE_CNT를 사용해야 합니다.")
+ _append_issue(issues, "CZN_COMN_SVC_DIM_BAS" not in upper_sql or not has_dimension_name, "CZN-13은 차원 테이블로 에테르를 해석해야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("DIM_TYPE_DIV_CD", "DIM_CTG", "CHANGE_TYPE_CD", "'USE'")) or "에테르" not in sql, "CZN-13은 에테르 사용 분류와 CHANGE_TYPE_CD='USE' 조건이 필요합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("CZN_COMN_USER_MST", "EXPT_USER_YN", "COUNT", "DISTINCT", "GUID", "SUM")), "CZN-13은 대상 유저 조인, 고유 사용 유저 수, 사용량 합계가 필요합니다.")
+ elif code == "CZN-17":
+ _append_issue(issues, any(term not in upper_sql for term in ("CZN_CUSTOM_GOODS_HAVE_TXN", "HAVE_CNT")), "CZN-17은 재화 보유 스냅샷을 사용해야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("CZN_COMN_CHARACTER_MST", "AU_FLAG", "EXPT_USER_YN")), "CZN-17은 캐릭터 AU와 제외 유저 조건이 필요합니다.")
+ _append_issue(issues, "AVG(" in upper_sql, "CZN-17 평균은 AVG(HAVE_CNT)가 아니라 합계/고유 캐릭터 수여야 합니다.")
+ _append_issue(issues, any(term not in upper_sql for term in ("SUM", "COUNT", "DISTINCT", "/", "CUID")), "CZN-17 평균은 SUM(HAVE_CNT)/COUNT(DISTINCT CUID)로 계산해야 합니다.")
+ _append_issue(issues, "GROUP BY" in upper_sql, "CZN-17은 기준일 단일 집계여야 하므로 상세 GROUP BY를 사용하면 안 됩니다.")
+ _append_issue(issues, any(term in upper_sql for term in ("LEVEL_COL", "ACM_CONN_DCNT", "TDAY_PLAY_TIME")), "CZN-17에는 레벨·접속일·플레이타임 지표가 포함되면 안 됩니다.")
diff --git a/poc4_active_source_20260714/src/poc4/qa_history_store.py b/poc4_active_source_20260714/src/poc4/qa_history_store.py
new file mode 100644
index 0000000..5d3b39a
--- /dev/null
+++ b/poc4_active_source_20260714/src/poc4/qa_history_store.py
@@ -0,0 +1,566 @@
+"""Oracle ADB persistence for the Smilegate customer QA benchmark."""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from datetime import datetime, timezone
+import json
+import os
+from pathlib import Path
+from typing import Any, Iterator, Mapping
+from urllib.parse import parse_qs
+
+import oracledb
+
+from src.poc4.qa_history import QaQuestion, load_benchmark_questions, question_fingerprint, question_from_record
+
+
+class QaHistoryStoreError(RuntimeError):
+ """A safe user-facing persistence error."""
+
+
+QUESTION_TABLE = "SG_AI_QA_QUESTION"
+ANSWER_TABLE = "SG_AI_QA_ANSWER"
+HISTORICAL_RUN_KEY = "HISTORICAL:2026-07-21:term-dict-final-v2"
+
+
+def _env_value(name: str, env_file: Path | None = None) -> str:
+ value = os.environ.get(name, "").strip()
+ if value or env_file is None or not env_file.is_file():
+ return value
+ try:
+ lines = env_file.read_text(encoding="utf-8").splitlines()
+ except (OSError, UnicodeError):
+ return ""
+ for line in lines:
+ line = line.strip()
+ if not line or line.startswith("#") or "=" not in line:
+ continue
+ if line.startswith("export "):
+ line = line[7:].lstrip()
+ key, raw = line.split("=", 1)
+ if key.strip() != name:
+ continue
+ raw = raw.strip()
+ if len(raw) >= 2 and raw[0] == raw[-1] and raw[0] in {"'", '"'}:
+ raw = raw[1:-1]
+ return raw.strip()
+ return ""
+
+
+def _normalize_oracle_dsn(raw_dsn: str) -> tuple[str, str]:
+ value = str(raw_dsn or "").strip()
+ if value.startswith("jdbc:oracle:thin:@"):
+ value = value[len("jdbc:oracle:thin:@"):]
+ if "?" not in value:
+ return value, ""
+ dsn, query = value.split("?", 1)
+ parsed = parse_qs(query, keep_blank_values=False)
+ wallet_dir = (parsed.get("TNS_ADMIN") or parsed.get("tns_admin") or [""])[0]
+ return dsn.strip(), wallet_dir.strip()
+
+
+def _read_lob(value: Any) -> Any:
+ return value.read() if hasattr(value, "read") else value
+
+
+def _record_from_cursor(cursor: Any, row: Any) -> dict[str, Any]:
+ names = [column[0].lower() for column in cursor.description]
+ return {name: _read_lob(value) for name, value in zip(names, row)}
+
+
+def _to_json(value: Mapping[str, Any] | None) -> str:
+ payload = dict(value or {})
+ text = json.dumps(payload, ensure_ascii=False, default=str)
+ if len(text) <= 120_000:
+ return text
+ return json.dumps(
+ {
+ "truncated": True,
+ "preview": text[:119_800],
+ },
+ ensure_ascii=False,
+ )
+
+
+def _answer_record(row: Mapping[str, Any]) -> dict[str, Any]:
+ result_json = str(row.get("result_json") or "").strip()
+ try:
+ result = json.loads(result_json) if result_json else {}
+ except ValueError:
+ result = {"raw": result_json}
+ return {
+ "answer_seq": row.get("answer_seq"),
+ "question_id": row.get("question_id"),
+ "answer_kind": str(row.get("answer_kind") or ""),
+ "run_key": str(row.get("run_key") or ""),
+ "conversation_id": str(row.get("conversation_id") or ""),
+ "requested_by": str(row.get("requested_by") or ""),
+ "requested_at": str(row.get("requested_at") or ""),
+ "model_profile": str(row.get("model_profile") or ""),
+ "generated_sql": str(row.get("generated_sql") or ""),
+ "answer_text": str(row.get("answer_text") or ""),
+ "result": result,
+ "execution_output": str(row.get("execution_output") or ""),
+ "execution_status": str(row.get("execution_status") or ""),
+ "judgment_status": str(row.get("judgment_status") or ""),
+ "judgment_reason": str(row.get("judgment_reason") or ""),
+ "duration_ms": row.get("duration_ms"),
+ "created_at": str(row.get("created_at") or ""),
+ }
+
+
+class QaHistoryStore:
+ def __init__(self, *, env_file: Path | None = None) -> None:
+ self._env_file = env_file
+ self._pool: Any | None = None
+
+ def _config(self) -> dict[str, str]:
+ username = (
+ _env_value("POC4_QA_DB_USERNAME", self._env_file)
+ or _env_value("BACKOFFICE_SELECT_AI_DB_USERNAME", self._env_file)
+ )
+ password = (
+ _env_value("POC4_QA_DB_PASSWORD", self._env_file)
+ or _env_value("BACKOFFICE_SELECT_AI_DB_PASSWORD", self._env_file)
+ )
+ raw_dsn = (
+ _env_value("POC4_QA_DB_DSN", self._env_file)
+ or _env_value("BACKOFFICE_SELECT_AI_DB_URL", self._env_file)
+ )
+ dsn, wallet_from_dsn = _normalize_oracle_dsn(raw_dsn)
+ wallet_dir = (
+ _env_value("POC4_QA_DB_WALLET_DIR", self._env_file)
+ or wallet_from_dsn
+ or _env_value("ORACLE_WALLET_DIR", self._env_file)
+ )
+ if not username or not password or not dsn:
+ raise QaHistoryStoreError("질답 이력 DB 접속 설정을 확인해 주세요.")
+ return {
+ "username": username,
+ "password": password,
+ "dsn": dsn,
+ "wallet_dir": wallet_dir,
+ }
+
+ def _get_pool(self) -> Any:
+ if self._pool is not None:
+ return self._pool
+ config = self._config()
+ kwargs: dict[str, Any] = {
+ "user": config["username"],
+ "password": config["password"],
+ "dsn": config["dsn"],
+ "min": 1,
+ "max": 3,
+ "increment": 1,
+ "getmode": oracledb.POOL_GETMODE_WAIT,
+ }
+ wallet_dir = Path(config["wallet_dir"]).expanduser()
+ if config["wallet_dir"]:
+ if not wallet_dir.is_dir():
+ raise QaHistoryStoreError("질답 이력 DB Wallet 경로를 확인해 주세요.")
+ kwargs["config_dir"] = str(wallet_dir)
+ kwargs["wallet_location"] = str(wallet_dir)
+ try:
+ self._pool = oracledb.create_pool(**kwargs)
+ return self._pool
+ except (oracledb.Error, OSError, ValueError) as exc:
+ raise QaHistoryStoreError("질답 이력 DB에 연결하지 못했습니다.") from exc
+
+ @contextmanager
+ def _connection(self) -> Iterator[Any]:
+ try:
+ with self._get_pool().acquire() as connection:
+ yield connection
+ except QaHistoryStoreError:
+ raise
+ except (oracledb.Error, OSError, ValueError) as exc:
+ raise QaHistoryStoreError("질답 이력 DB 작업에 실패했습니다.") from exc
+
+ def list_questions(self, *, limit: int = 200) -> list[QaQuestion]:
+ sql = f"""
+ SELECT q.question_id, q.question_code, q.category, q.title,
+ q.question_text, q.source_document, q.source_sheet,
+ q.source_row, q.source_scenario, q.sample_sql,
+ q.expected_focus, q.baseline_sql, q.baseline_answer,
+ q.support_level, q.evaluation_rule_json,
+ latest.judgment_status AS last_judgment_status,
+ TO_CHAR(latest.evaluated_at AT TIME ZONE 'Asia/Seoul',
+ 'YYYY-MM-DD HH24:MI:SS TZH:TZM') AS last_evaluated_at
+ FROM {QUESTION_TABLE} q
+ LEFT JOIN (
+ SELECT question_id, judgment_status, evaluated_at
+ FROM (
+ SELECT question_id, judgment_status, evaluated_at,
+ ROW_NUMBER() OVER (
+ PARTITION BY question_id ORDER BY answer_seq DESC
+ ) AS row_no
+ FROM {ANSWER_TABLE}
+ )
+ WHERE row_no = 1
+ ) latest ON latest.question_id = q.question_id
+ WHERE q.active_yn = 'Y'
+ ORDER BY q.category, q.question_code
+ FETCH FIRST :row_limit ROWS ONLY
+ """
+ with self._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(sql, {"row_limit": int(limit)})
+ rows = [_record_from_cursor(cursor, row) for row in cursor]
+ return [question_from_record(row) for row in rows]
+
+ def get_question(self, question_id: int) -> QaQuestion | None:
+ sql = f"""
+ SELECT question_id, question_code, category, title, question_text,
+ source_document, source_sheet, source_row, source_scenario,
+ sample_sql, expected_focus, baseline_sql, baseline_answer,
+ support_level, evaluation_rule_json
+ FROM {QUESTION_TABLE}
+ WHERE question_id = :question_id AND active_yn = 'Y'
+ """
+ with self._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(sql, {"question_id": int(question_id)})
+ row = cursor.fetchone()
+ return question_from_record(_record_from_cursor(cursor, row)) if row else None
+
+ def list_answers(self, question_id: int, *, limit: int = 30) -> list[dict[str, Any]]:
+ sql = f"""
+ SELECT answer_seq, question_id, answer_kind, run_key, conversation_id,
+ requested_by,
+ TO_CHAR(requested_at AT TIME ZONE 'Asia/Seoul',
+ 'YYYY-MM-DD HH24:MI:SS TZH:TZM') AS requested_at,
+ model_profile, generated_sql, answer_text, result_json,
+ execution_output, execution_status, judgment_status,
+ judgment_reason, duration_ms,
+ TO_CHAR(created_at AT TIME ZONE 'Asia/Seoul',
+ 'YYYY-MM-DD HH24:MI:SS TZH:TZM') AS created_at
+ FROM {ANSWER_TABLE}
+ WHERE question_id = :question_id
+ ORDER BY answer_seq DESC
+ FETCH FIRST :row_limit ROWS ONLY
+ """
+ with self._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(sql, {"question_id": int(question_id), "row_limit": int(limit)})
+ rows = [_record_from_cursor(cursor, row) for row in cursor]
+ return [_answer_record(row) for row in rows]
+
+ def find_or_create_free_text_question(self, question_text: str) -> QaQuestion:
+ normalized = str(question_text or "").strip()
+ if not normalized:
+ raise QaHistoryStoreError("자유 질의가 비어 있습니다.")
+ fingerprint = question_fingerprint(normalized)
+ code = f"ADHOC-{fingerprint[:12].upper()}"
+ merge_sql = f"""
+ MERGE INTO {QUESTION_TABLE} target
+ USING (SELECT :question_hash AS question_hash FROM dual) source
+ ON (target.question_hash = source.question_hash)
+ WHEN NOT MATCHED THEN INSERT (
+ question_code, question_source, question_hash, category, title,
+ question_text, support_level, evaluation_rule_json, active_yn
+ ) VALUES (
+ :question_code, 'FREE_TEXT', :question_hash, 'FREE_TEXT',
+ :title, :question_text, 'REVIEW', '{{}}', 'Y'
+ )
+ """
+ with self._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(
+ merge_sql,
+ {
+ "question_hash": fingerprint,
+ "question_code": code,
+ "title": normalized[:180],
+ "question_text": normalized,
+ },
+ )
+ connection.commit()
+ cursor.execute(
+ f"""SELECT question_id FROM {QUESTION_TABLE}
+ WHERE question_hash = :question_hash""",
+ {"question_hash": fingerprint},
+ )
+ row = cursor.fetchone()
+ if not row:
+ raise QaHistoryStoreError("자유 질의 마스터를 저장하지 못했습니다.")
+ question = self.get_question(int(row[0]))
+ if question is None:
+ raise QaHistoryStoreError("자유 질의 마스터를 다시 읽지 못했습니다.")
+ return question
+
+ def record_answer(
+ self,
+ *,
+ question_id: int,
+ answer_kind: str,
+ conversation_id: str,
+ requested_by: str,
+ model_profile: str,
+ generated_sql: str,
+ answer_text: str,
+ result: Mapping[str, Any] | None,
+ execution_output: str,
+ execution_status: str,
+ judgment_status: str,
+ judgment_reason: str,
+ duration_ms: int | None,
+ run_key: str = "",
+ ) -> None:
+ sql = f"""
+ INSERT INTO {ANSWER_TABLE} (
+ question_id, answer_kind, run_key, conversation_id, requested_by,
+ requested_at, model_profile, generated_sql, answer_text, result_json,
+ execution_output, execution_status, judgment_status,
+ judgment_reason, duration_ms
+ ) VALUES (
+ :question_id, :answer_kind, :run_key, :conversation_id,
+ :requested_by, SYSTIMESTAMP, :model_profile, :generated_sql,
+ :answer_text, :result_json, :execution_output, :execution_status,
+ :judgment_status, :judgment_reason, :duration_ms
+ )
+ """
+ with self._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(
+ sql,
+ {
+ "question_id": int(question_id),
+ "answer_kind": str(answer_kind)[:20],
+ "run_key": str(run_key)[:100] or None,
+ "conversation_id": str(conversation_id)[:100] or None,
+ "requested_by": str(requested_by)[:100] or None,
+ "model_profile": str(model_profile)[:100] or None,
+ "generated_sql": str(generated_sql or ""),
+ "answer_text": str(answer_text or ""),
+ "result_json": _to_json(result),
+ "execution_output": str(execution_output or ""),
+ "execution_status": str(execution_status)[:40] or None,
+ "judgment_status": str(judgment_status)[:20],
+ "judgment_reason": str(judgment_reason or ""),
+ "duration_ms": duration_ms,
+ },
+ )
+ connection.commit()
+
+ def seed_benchmark(self, benchmark_file: Path) -> tuple[int, int]:
+ questions = load_benchmark_questions(benchmark_file)
+ raw = json.loads(benchmark_file.read_text(encoding="utf-8"))
+ raw_by_code = {
+ str(item.get("case_id") or "").upper(): item
+ for item in raw.get("scenarios", [])
+ if isinstance(item, Mapping)
+ }
+ seeded_questions = 0
+ seeded_answers = 0
+ for question in questions:
+ question_id = self._upsert_benchmark_question(question)
+ seeded_questions += 1
+ raw_item = raw_by_code[question.question_code]
+ history = raw_item.get("historical_answer") if isinstance(raw_item.get("historical_answer"), Mapping) else {}
+ inserted = self._seed_historical_answer(question_id, history, raw)
+ seeded_answers += 1 if inserted else 0
+ return seeded_questions, seeded_answers
+
+ def _upsert_benchmark_question(self, question: QaQuestion) -> int:
+ sql = f"""
+ MERGE INTO {QUESTION_TABLE} target
+ USING (SELECT :question_code AS question_code FROM dual) source
+ ON (target.question_code = source.question_code)
+ WHEN MATCHED THEN UPDATE SET
+ question_source = 'CUSTOMER_EXCEL',
+ question_hash = :question_hash,
+ category = :category,
+ title = :title,
+ question_text = :question_text,
+ source_document = :source_document,
+ source_sheet = :source_sheet,
+ source_row = :source_row,
+ source_scenario = :source_scenario,
+ sample_sql = :sample_sql,
+ expected_focus = :expected_focus,
+ baseline_sql = :baseline_sql,
+ baseline_answer = :baseline_answer,
+ support_level = :support_level,
+ evaluation_rule_json = :evaluation_rule_json,
+ active_yn = 'Y',
+ updated_at = SYSTIMESTAMP
+ WHEN NOT MATCHED THEN INSERT (
+ question_code, question_source, question_hash, category, title,
+ question_text, source_document, source_sheet, source_row,
+ source_scenario, sample_sql, expected_focus, baseline_sql,
+ baseline_answer, support_level, evaluation_rule_json, active_yn
+ ) VALUES (
+ :question_code, 'CUSTOMER_EXCEL', :question_hash, :category,
+ :title, :question_text, :source_document, :source_sheet,
+ :source_row, :source_scenario, :sample_sql, :expected_focus,
+ :baseline_sql, :baseline_answer, :support_level,
+ :evaluation_rule_json, 'Y'
+ )
+ """
+ binds = {
+ "question_code": question.question_code,
+ "question_hash": question_fingerprint(question.question_text),
+ "category": question.category[:30],
+ "title": question.title[:200],
+ "question_text": question.question_text,
+ "source_document": question.source_document[:255] or None,
+ "source_sheet": question.source_sheet[:255] or None,
+ "source_row": question.source_row,
+ "source_scenario": question.source_scenario,
+ "sample_sql": question.sample_sql,
+ "expected_focus": question.expected_focus,
+ "baseline_sql": question.baseline_sql,
+ "baseline_answer": question.baseline_answer,
+ "support_level": question.support_level[:20],
+ "evaluation_rule_json": json.dumps(question.evaluation_rule, ensure_ascii=False),
+ }
+ with self._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(sql, binds)
+ connection.commit()
+ cursor.execute(
+ f"SELECT question_id FROM {QUESTION_TABLE} WHERE question_code = :question_code",
+ {"question_code": question.question_code},
+ )
+ row = cursor.fetchone()
+ if not row:
+ raise QaHistoryStoreError(f"질문 마스터를 적재하지 못했습니다: {question.question_code}")
+ return int(row[0])
+
+ def _seed_historical_answer(
+ self,
+ question_id: int,
+ history: Mapping[str, Any],
+ benchmark: Mapping[str, Any],
+ ) -> bool:
+ exists_sql = f"""
+ SELECT COUNT(*) FROM {ANSWER_TABLE}
+ WHERE question_id = :question_id AND run_key = :run_key
+ """
+ with self._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(exists_sql, {"question_id": question_id, "run_key": HISTORICAL_RUN_KEY})
+ if int(cursor.fetchone()[0]) > 0:
+ return False
+ result = {
+ "source_report": str(benchmark.get("source_report") or ""),
+ "source_redmine": benchmark.get("source_redmine"),
+ "historical_execution_output": str(history.get("execution_output") or ""),
+ }
+ self.record_answer(
+ question_id=question_id,
+ answer_kind="HISTORICAL",
+ run_key=HISTORICAL_RUN_KEY,
+ conversation_id="",
+ requested_by="customer-excel-baseline",
+ model_profile=str(history.get("profile") or ""),
+ generated_sql=str(history.get("generated_sql") or ""),
+ answer_text=str(history.get("answer_text") or ""),
+ result=result,
+ execution_output=str(history.get("execution_output") or ""),
+ execution_status=str(history.get("execution_status") or ""),
+ judgment_status=str(history.get("judgment_status") or "REVIEW"),
+ judgment_reason=str(history.get("judgment_reason") or ""),
+ duration_ms=int(history.get("duration_ms") or 0),
+ )
+ return True
+
+
+def schema_statements() -> tuple[str, ...]:
+ return (
+ f"""
+ CREATE TABLE {QUESTION_TABLE} (
+ question_id NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY PRIMARY KEY,
+ question_code VARCHAR2(30) UNIQUE,
+ question_source VARCHAR2(30) NOT NULL,
+ question_hash VARCHAR2(64) NOT NULL UNIQUE,
+ category VARCHAR2(30) NOT NULL,
+ title VARCHAR2(200) NOT NULL,
+ question_text CLOB NOT NULL,
+ source_document VARCHAR2(255),
+ source_sheet VARCHAR2(255),
+ source_row NUMBER,
+ source_scenario CLOB,
+ sample_sql CLOB,
+ expected_focus CLOB,
+ baseline_sql CLOB,
+ baseline_answer CLOB,
+ support_level VARCHAR2(20) NOT NULL,
+ evaluation_rule_json CLOB CHECK (evaluation_rule_json IS JSON),
+ active_yn CHAR(1) DEFAULT 'Y' NOT NULL CHECK (active_yn IN ('Y', 'N')),
+ created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ CONSTRAINT sg_ai_qa_question_source_ck
+ CHECK (question_source IN ('CUSTOMER_EXCEL', 'FREE_TEXT'))
+ )
+ """,
+ f"""
+ CREATE TABLE {ANSWER_TABLE} (
+ answer_seq NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ question_id NUMBER NOT NULL,
+ answer_kind VARCHAR2(20) NOT NULL,
+ run_key VARCHAR2(100),
+ conversation_id VARCHAR2(100),
+ requested_by VARCHAR2(100),
+ requested_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ model_profile VARCHAR2(100),
+ generated_sql CLOB,
+ answer_text CLOB,
+ result_json CLOB CHECK (result_json IS JSON),
+ execution_output CLOB,
+ execution_status VARCHAR2(40),
+ judgment_status VARCHAR2(20) NOT NULL,
+ judgment_reason CLOB,
+ duration_ms NUMBER,
+ created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ CONSTRAINT sg_ai_qa_answer_question_fk
+ FOREIGN KEY (question_id)
+ REFERENCES {QUESTION_TABLE} (question_id)
+ ON DELETE CASCADE,
+ CONSTRAINT sg_ai_qa_answer_kind_ck
+ CHECK (answer_kind IN ('HISTORICAL', 'LIVE')),
+ CONSTRAINT sg_ai_qa_answer_judgment_ck
+ CHECK (judgment_status IN ('PASS', 'WARN', 'FAIL', 'REVIEW'))
+ )
+ """,
+ f"""
+ CREATE INDEX sg_ai_qa_answer_question_ix
+ ON {ANSWER_TABLE} (question_id, answer_seq DESC)
+ """,
+ f"""
+ CREATE UNIQUE INDEX sg_ai_qa_answer_run_uk
+ ON {ANSWER_TABLE} (question_id, run_key)
+ """,
+ )
+
+
+def ensure_schema(store: QaHistoryStore) -> None:
+ objects = (QUESTION_TABLE, ANSWER_TABLE)
+ with store._connection() as connection:
+ with connection.cursor() as cursor:
+ cursor.execute(
+ "SELECT table_name FROM user_tables WHERE table_name IN (:q, :a)",
+ {"q": objects[0], "a": objects[1]},
+ )
+ existing = {str(row[0]) for row in cursor}
+ statements = schema_statements()
+ if QUESTION_TABLE not in existing:
+ cursor.execute(statements[0])
+ if ANSWER_TABLE not in existing:
+ cursor.execute(statements[1])
+ cursor.execute(
+ "SELECT index_name FROM user_indexes WHERE index_name IN (:ix1, :ix2)",
+ {"ix1": "SG_AI_QA_ANSWER_QUESTION_IX", "ix2": "SG_AI_QA_ANSWER_RUN_UK"},
+ )
+ indexes = {str(row[0]) for row in cursor}
+ if "SG_AI_QA_ANSWER_QUESTION_IX" not in indexes:
+ cursor.execute(statements[2])
+ if "SG_AI_QA_ANSWER_RUN_UK" not in indexes:
+ cursor.execute(statements[3])
+ connection.commit()
+
+
+def timestamp_now() -> str:
+ return datetime.now(timezone.utc).isoformat(timespec="seconds")
diff --git a/poc4_active_source_20260714/tests/test_qa_history.py b/poc4_active_source_20260714/tests/test_qa_history.py
new file mode 100644
index 0000000..a0cc8b8
--- /dev/null
+++ b/poc4_active_source_20260714/tests/test_qa_history.py
@@ -0,0 +1,80 @@
+from __future__ import annotations
+
+from pathlib import Path
+import unittest
+
+from src.poc4.qa_history import evaluate_sql, load_benchmark_questions
+from src.poc4.qa_history_store import _normalize_oracle_dsn, schema_statements
+
+
+class QaHistoryTest(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls) -> None:
+ benchmark = Path(__file__).parents[1] / "config" / "smilegate_qa_benchmark.json"
+ cls.questions = {item.question_code: item for item in load_benchmark_questions(benchmark)}
+
+ def test_customer_excel_benchmark_contains_all_47_cases(self) -> None:
+ self.assertEqual(47, len(self.questions))
+ self.assertIn("STD-01", self.questions)
+ self.assertIn("CZN-19", self.questions)
+
+ def test_supported_query_passes_when_required_terms_are_present(self) -> None:
+ judgment = evaluate_sql(
+ self.questions["STD-13"],
+ "SELECT SUM(PAYMT_AMT) FROM COMN_SALES_TXN",
+ execution_succeeded=True,
+ )
+ self.assertEqual("PASS", judgment.status)
+
+ def test_monthly_au_with_au_flag_fails(self) -> None:
+ judgment = evaluate_sql(
+ self.questions["STD-27"],
+ """
+ SELECT COUNT(*)
+ FROM CZN_COMN_USER_MST
+ WHERE AU_FLAG = 1
+ AND BASE_DT = (SELECT MAX(BASE_DT) FROM CZN_COMN_USER_MST)
+ AND LAST_CONN_DT >= ADD_MONTHS(BASE_DT, -1)
+ AND STD_USER_YN = 'Y'
+ AND EXPT_USER_YN = 'N'
+ """,
+ execution_succeeded=True,
+ )
+ self.assertEqual("FAIL", judgment.status)
+ self.assertIn("AU_FLAG", judgment.reason)
+
+ def test_unsupported_game_requires_safe_alias_lookup(self) -> None:
+ safe = evaluate_sql(
+ self.questions["STD-02"],
+ "SELECT GAME_ID FROM COMN_GAME_ALIAS_BAS WHERE GAME_NM LIKE '%버블리즈%'",
+ execution_succeeded=True,
+ )
+ unsafe = evaluate_sql(
+ self.questions["STD-02"],
+ "SELECT COUNT(*) FROM CZN_COMN_USER_MST WHERE GAME_ID = 'STOVE_CHAOSZERO'",
+ execution_succeeded=True,
+ )
+ self.assertEqual("PASS", safe.status)
+ self.assertEqual("FAIL", unsafe.status)
+
+ def test_free_text_is_review_not_automatic_pass(self) -> None:
+ judgment = evaluate_sql(None, "SELECT 1 FROM DUAL", execution_succeeded=True)
+ self.assertEqual("REVIEW", judgment.status)
+
+ def test_jdbc_url_wallet_is_normalized_for_python_driver(self) -> None:
+ self.assertEqual(
+ ("sgmpaipoc_medium", "/home/opc/wallet/sgmpaipoc"),
+ _normalize_oracle_dsn(
+ "jdbc:oracle:thin:@sgmpaipoc_medium?TNS_ADMIN=/home/opc/wallet/sgmpaipoc"
+ ),
+ )
+
+ def test_schema_defines_two_history_tables_and_indexes(self) -> None:
+ statements = "\n".join(schema_statements())
+ self.assertIn("CREATE TABLE SG_AI_QA_QUESTION", statements)
+ self.assertIn("CREATE TABLE SG_AI_QA_ANSWER", statements)
+ self.assertIn("answer_seq NUMBER GENERATED ALWAYS AS IDENTITY", statements)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/sql/adb/47_smilegate_qa_history.sql b/sql/adb/47_smilegate_qa_history.sql
new file mode 100644
index 0000000..b1788bf
--- /dev/null
+++ b/sql/adb/47_smilegate_qa_history.sql
@@ -0,0 +1,62 @@
+-- Smilegate customer Excel QA benchmark history.
+-- This script is also applied by poc4_active_source_20260714/scripts/
+-- sync_smilegate_qa_history.py with existence checks for repeatable deployment.
+
+CREATE TABLE SG_AI_QA_QUESTION (
+ question_id NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY PRIMARY KEY,
+ question_code VARCHAR2(30) UNIQUE,
+ question_source VARCHAR2(30) NOT NULL,
+ question_hash VARCHAR2(64) NOT NULL UNIQUE,
+ category VARCHAR2(30) NOT NULL,
+ title VARCHAR2(200) NOT NULL,
+ question_text CLOB NOT NULL,
+ source_document VARCHAR2(255),
+ source_sheet VARCHAR2(255),
+ source_row NUMBER,
+ source_scenario CLOB,
+ sample_sql CLOB,
+ expected_focus CLOB,
+ baseline_sql CLOB,
+ baseline_answer CLOB,
+ support_level VARCHAR2(20) NOT NULL,
+ evaluation_rule_json CLOB CHECK (evaluation_rule_json IS JSON),
+ active_yn CHAR(1) DEFAULT 'Y' NOT NULL CHECK (active_yn IN ('Y', 'N')),
+ created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ CONSTRAINT sg_ai_qa_question_source_ck
+ CHECK (question_source IN ('CUSTOMER_EXCEL', 'FREE_TEXT'))
+);
+
+CREATE TABLE SG_AI_QA_ANSWER (
+ answer_seq NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ question_id NUMBER NOT NULL,
+ answer_kind VARCHAR2(20) NOT NULL,
+ run_key VARCHAR2(100),
+ conversation_id VARCHAR2(100),
+ requested_by VARCHAR2(100),
+ requested_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ model_profile VARCHAR2(100),
+ generated_sql CLOB,
+ answer_text CLOB,
+ result_json CLOB CHECK (result_json IS JSON),
+ execution_output CLOB,
+ execution_status VARCHAR2(40),
+ judgment_status VARCHAR2(20) NOT NULL,
+ judgment_reason CLOB,
+ duration_ms NUMBER,
+ created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+ CONSTRAINT sg_ai_qa_answer_question_fk
+ FOREIGN KEY (question_id)
+ REFERENCES SG_AI_QA_QUESTION (question_id)
+ ON DELETE CASCADE,
+ CONSTRAINT sg_ai_qa_answer_kind_ck
+ CHECK (answer_kind IN ('HISTORICAL', 'LIVE')),
+ CONSTRAINT sg_ai_qa_answer_judgment_ck
+ CHECK (judgment_status IN ('PASS', 'WARN', 'FAIL', 'REVIEW'))
+);
+
+CREATE INDEX sg_ai_qa_answer_question_ix
+ ON SG_AI_QA_ANSWER (question_id, answer_seq DESC);
+
+CREATE UNIQUE INDEX sg_ai_qa_answer_run_uk
+ ON SG_AI_QA_ANSWER (question_id, run_key);