feat: add scripts for fetching and training educational corpus

Browse Source

- Implemented `fetch_corpus.py` to scrape Chinese educational content from Wikipedia, covering subjects like Chinese, Math, English, Chemistry, Politics, History, Geography, Physics, and Biology.
- Developed `fetch_corpus_incremental.py` for incremental fetching of missing entries, with automatic retries for rate limiting and support for resuming interrupted downloads.
- Created `train_chat.py` for training a dialogue model, including corpus cleaning, vocabulary management, and staged training with adaptive learning rates.

...

This commit is contained in:

XiuchengWu

2026-07-07 17:42:24 +08:00

parent d5affe4458

commit 81f6fd58e0

96 changed files with 53920 additions and 17 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

14343

corpus/数学.txt Normal file

File diff suppressed because it is too large Load Diff