20 Commits

Author SHA1 Message Date
b7588fa3b6 memo: 한 입력의 여러 주제를 항목별로 분리 저장
- MemoService.organize() → List<MemoDraft> 반환. 프롬프트를 JSON 배열 출력으로
  변경, 서로 다른 주제는 분리·관련 내용은 묶도록 지시. extractJson 배열 대응.
- POST /api/notes/organize → { items: [...] } (save=true면 전부 저장).
- 프론트: 정리 결과를 다중 카드로 표시, 항목별 수정·삭제 + '모두 저장'(카드마다 /memo).

실측: 두서없는 5주제 입력 → 5개 항목(각 PARA 카테고리+GTD 태그)으로 정확히 분리.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 06:41:58 +00:00
df9c6a6b50 memo: 두서없는 메모 → LLM(PARA+GTD) 자동정리·분류 저장
- MemoService(save_memo tool): organize(정리·분류→구조화 JSON) + saveMemo(카테고리
  findOrCreate + 노트+태그 적재) 분리 설계. GUI/외부API/향후 MCP 공통 사용.
- 분류 체계: PARA(카테고리 경로) + GTD(next-action/waiting/someday/reference 태그) 하이브리드.
- NoteController: POST /api/notes/organize (미리보기, save=true면 원샷 저장),
  POST /api/notes/memo (구조화 필드 저장).
- NoteRepository: insertMemo(project 포함), addTag/findTags.
- DB: notes.project 컬럼 + note_tags 테이블 (V2 마이그레이션).
- 프론트: notes/new 텍스트 모드에 'AI로 정리' → 미리보기(카테고리/제목/태그/
  프로젝트/요약/본문 수정) → 정리본 저장 흐름 추가.
- LLM은 OCI GenAI(현재 gpt-5.6-terra) 구조화 JSON 출력 방식(tool-calling 의존 없음).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 04:09:21 +00:00
e1e71dc954 GenAI: chat 모델 OpenAI(gpt-5.6) 호환 + 노트 하단 사용모델 표기
- OciGenAiService.chat(): 모델 인지 분기. openai.* 모델은 maxTokens/
  temperature(0.3)를 거부하므로 maxCompletionTokens 사용 + temperature 생략.
  gemini/cohere 등 기존 모델은 maxTokens+temperature 유지(하위호환).
- NoteController: 노트 최종 결과 하단에 사용 모델 푸터 표기
  (음성변환 모델 + 교정·요약 모델). 모델명은 설정값에서 조회해 자동 반영.
- STT(음성→텍스트)는 OpenRouter Gemini 유지, 변경 없음.
- OCI_GENAI_MODEL은 .env에서 openai.gpt-5.6-terra로 전환(커밋 제외).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 07:34:55 +00:00
446969b862 note STT: OpenRouter 경로 오버랩 청크 분할 + 퍼지 디덥 병합
대용량 오디오가 OpenRouter(Gemini) 단일 요청 한도(~20MB)를 초과해 502로
실패하고, 깨진 Gemma fallback(ollama runner crash)으로 넘어가 전체 실패하던
문제 수정.

- 공통 청크 분할기 transcribeChunked(전략 패턴)로 OpenRouter/Gemma 통일
- CHUNK_SECONDS=180s, OVERLAP_SECONDS=10s 오버랩 분할(경계 단어 손실 방지)
- mergeWithOverlapDedup: 인접 청크 토큰 위치정렬 80% 일치로 중복 제거,
  미검출 시 줄바꿈 안전 연결(누락 방지)
- 청크 실패는 기록·로그 후 [변환 실패한 구간 번호] 명시, 전부 실패 시 예외
- transcribeAsync의 Gemma fallback도 단발→transcribeWithGemma(청크) 사용

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 05:36:40 +00:00
9569309e49 크롤링 복구: Chrome CDP를 PM2 sundol-chrome로 상시화
Chrome 136+가 기본 프로필 디렉토리에서 원격 디버깅(CDP)을 거부하여
4월 13일 이후 웹크롤링 3차 폴백/유튜브 자막 추출이 전부 실패하던 문제 해결.

- 프로필을 non-default 디렉토리(~/.config/google-chrome-cdp)로 이동해
  로그인 세션 유지한 채 CDP 허용
- start-chrome.sh 신규: 기존 Chrome 정리 + stale lock 제거 후
  --remote-debugging-port=9222 --remote-debugging-address=127.0.0.1 로 기동
- ecosystem.config.cjs: sundol-chrome PM2 앱 추가 (수동 실행 금지, PM2 통일)
  ※ frontend script의 /usr/local/bin/node 변경은 이전 작업분이 함께 포함됨
- PlaywrightBrowserService: CDP_URL을 127.0.0.1로 고정 (IPv6 ::1 해석 함정 제거)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 01:06:58 +00:00
1088b23790 Add Notes, Voice Clone TTS, fix auth persistence and maxTokens
Notes:
- notes table with TEXT/AUDIO types, category support
- Audio upload → OpenRouter Gemini STT → OCI GenAI polish/summary
- Raw STT saved separately in raw_content column
- Polish/summary button for manual re-processing
- Async processing with real-time polling

Voice Clone TTS:
- Qwen3-TTS 1.7B model on A10 GPU via FastAPI server
- Voice profile registration (record/upload → save embedding)
- Profile-based TTS generation API
- TTS web page with recording, profile management, generation

Auth fixes:
- Store both access + refresh tokens in localStorage
- Initialize state from localStorage synchronously (no flash)
- Request interceptor reads token from localStorage every request
- Refresh via body (not just cookie)

Other fixes:
- maxTokens 4096 → 65536 (OCI GenAI Gemini supports up to 65536)
- Fix broken Korean chars in source files
- OpenRouter config for STT
- ffmpeg installed for audio conversion
- Ollama + Gemma 4 E4B installed (STT fallback)
- nginx proxy for TTS server (/api/tts/)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 07:34:18 +00:00
6c2129d42e Add category view, pagination, and persist login across deployments
- Add 2-panel category view: sidebar tree + filtered item list
- Category counts use DISTINCT with descendant inclusion
- Hide empty categories, show category badges on item cards
- Add client-side pagination (10 items/page) for both views
- Persist access token in localStorage to survive page refresh
- Fix token refresh retry on backend restart

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 04:11:08 +00:00
f9f710ec90 Add English level settings, improve content structuring and rendering
- Add english_level column to users table (CEFR with TOEIC mapping)
- Add UserController (GET/PATCH /api/users/me) and Settings page
- Enhance structuring prompts: sequential TOC, no summary sections,
  no content overlap, English expression extraction by CEFR level
- Remove sub-TOC analysis (caused content repetition), use simple
  per-section generation with truncation detection and continuation
- Fix CLOB truncation: explicit Clob-to-String conversion in repository
- Replace regex-based markdown rendering with react-markdown
- Add wallet renewal procedure to troubleshooting docs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 23:48:38 +00:00
4cde775809 Switch to user Chrome CDP for YouTube transcript, fix auth and ads
- Replace Playwright standalone browser with CDP connection to user Chrome
  (bypasses YouTube bot detection by using logged-in Chrome session)
- Add video playback, ad detection/skip, and play confirmation before transcript extraction
- Extract transcript JS to separate resource files (fix SyntaxError in evaluate)
- Add ytInitialPlayerResponse-based transcript extraction as primary method
- Fix token refresh: retry on network error during backend restart
- Fix null userId logout, CLOB type hint for structured_content
- Disable XFCE screen lock/screensaver
- Add troubleshooting entries (#10-12) and YouTube transcript guide

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 21:01:49 +00:00
9abb770e37 Add knowledge structuring feature with incremental LLM processing
- Add structured_content column and STRUCTURING pipeline step
- Split LLM structuring into TOC + per-section calls to avoid token limit
- Save intermediate results to DB for real-time frontend polling (3s)
- Add manual "정리하기" button with async processing
- Fix browser login modal by customizing authentication entry point
- Fix standalone build symlinks for server.js and static files
- Add troubleshooting guide

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 19:43:21 +00:00
afc9cdcde6 Refactor Playwright to singleton browser with tab-based crawling
- Add PlaywrightBrowserService: singleton Chromium browser with auto-recovery
- Refactor WebCrawlerService/YouTubeTranscriptService to use shared browser tabs
- Fix YouTube transcript: extract from DOM panel + fmt=json3 fallback
- Keep browser window alive (about:blank instead of page.close)
- Add docs: X Window setup, operation manual, crawling guide

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 19:18:33 +00:00
db4155c36d Add error logging and improve HTTP handling for YouTube transcript fetching
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 18:38:23 +00:00
56d5752095 Add YouTube cookie support to Playwright fallback for bot bypass
Load cookies.txt (Netscape format) into Playwright browser context
before navigating to YouTube, enabling authenticated access to bypass
bot detection that blocks transcript retrieval.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 07:37:48 +00:00
677a79978f Use youtube-transcript-api library with Playwright fallback for YouTube transcripts
Replace Jsoup-based approach with io.github.thoroldvix:youtube-transcript-api
as primary method (supports manual/generated captions, language priority).
Playwright head mode kept as fallback when API fails.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 07:26:52 +00:00
1bfe55d5a8 Switch YouTube transcript fetching from Jsoup to Playwright head mode
Jsoup was blocked by YouTube bot detection. Now uses Playwright with
headed Chromium via Xvfb virtual display to bypass restrictions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 07:11:52 +00:00
bb5a601433 Add YouTube transcript auto-fetch button on Knowledge add page
- YouTubeTranscriptService: fetches captions from YouTube page (ko > en > first available)
- GET /api/knowledge/youtube-transcript endpoint
- Frontend: "트랜스크립트 자동 가져오기" button appears when valid YouTube URL entered

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 04:20:13 +00:00
f0f7b62e3d Add Playwright headless browser as 3rd crawling fallback
Crawl chain: Jsoup → Jina Reader → Playwright (headless Chromium).
Error page detection (403, Access Denied, etc.) triggers next fallback.
Switch to exploded classpath for Playwright driver-bundle compatibility.
Fix Next.js standalone static file serving with symlink.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 22:36:24 +00:00
0cc84354f5 Add Jina Reader API fallback for web crawling
Jsoup fails on bot-blocked sites (403). Now tries Jsoup first,
then falls back to Jina Reader (r.jina.ai) for better coverage.
Supports optional API key via JINA_READER_API_KEY env var.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 22:03:09 +00:00
9929322de0 Implement all core features: Knowledge pipeline, RAG chat, Todos, Habits, Study Cards, Tags, Dashboard
- Google OAuth authentication with callback flow
- Knowledge ingest pipeline (TEXT/WEB/YOUTUBE → chunking → categorization → embedding)
- OCI GenAI integration (chat, embeddings) with multi-model support
- Semantic search via Oracle VECTOR_DISTANCE
- RAG-based AI chat with source attribution
- Todos with subtasks, filters, and priority levels
- Habits with daily check-in, streak tracking, and color customization
- Study Cards with SM-2 spaced repetition and LLM auto-generation
- Tags system with knowledge item mapping
- Dashboard with live data from all modules

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 21:43:51 +00:00
3d2aa6cf46 Add backend/frontend scaffolding with Oracle ADB wallet config
- Backend: Spring Boot 3 + WebFlux, JWT auth, Oracle ADB wallet,
  8 controllers/services/repositories (Auth~Tag), DTOs, exception handling
- Frontend: Next.js 15, TypeScript, Tailwind CSS, AuthContext,
  7 pages (dashboard, knowledge, chat, study, todos, habits, login)
- DB: V1 migration with 12 tables including VECTOR(1024) + HNSW index
- Ops: PM2 ecosystem config, deploy.sh, start-backend.sh
- CLAUDE.md: DB credentials replaced with env var references

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 06:56:26 +00:00