# I am fine tuning llama 3.2 8bit lora using chetgpt, but I am doubtful whether the direction is correct

**URL:** <https://discuss.huggingface.co/t/i-am-fine-tuning-llama-3-2-8bit-lora-using-chetgpt-but-i-am-doubtful-whether-the-direction-is-correct/177482>\
**Category:** Beginners\
**Created:** [July 5, 2026, 10:08pm UTC](https://discuss.huggingface.co/t/i-am-fine-tuning-llama-3-2-8bit-lora-using-chetgpt-but-i-am-doubtful-whether-the-direction-is-correct/177482 "2026-07-05T22:08:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kbuwel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kbuwel/32/46076_2.png) [@kbuwel](https://discuss.huggingface.co/u/kbuwel)\
**Post date:** [July 5, 2026, 10:08pm UTC](https://discuss.huggingface.co/t/i-am-fine-tuning-llama-3-2-8bit-lora-using-chetgpt-but-i-am-doubtful-whether-the-direction-is-correct/177482/1 "2026-07-05T22:08:59Z")

</div>

I’m trying to fine-tune using things like the Harry Potter novels.  
I performed fine tuning several times with llama-3-Korean-Bllossom-8B.  
I wanted to create a creative llama model rather than an instructional one.  
Let me show you an example.  
I made one sample from Chapter 1: The Surviving Child to Chapter 2: In Front of the Vanishing Window.  
The results of doing as gpt said were disastrous.  
In order to divide the scenes into chunks, there were times when scenes were cut randomly and the scenes were messed around, saying they had to be stitched together.  
So I cut one sample like that.  
The jsonl file contains {“text”: “Chapter 1 The Child Who Survived\nThe Dursleys living at 4 Privet Drive were people who were very proud of being normal.  
Everything.”}  
I created a sample like this.  
Here is the code:

# -_- coding: utf-8 -_-

“”"  
train\_auto.py

QLoRA continuation/style learning automation script

Goal:

- train.jsonl {“text”: “…”} Automatic data analysis
- Automatic learning amount determination based on data size/token/block
- Prevents the problem of small data lasting dozens of epochs
- Test creation of the same prompt before and after learning
- Automatically save before\_sample.txt / after\_sample.txt / train\_report.txt
- Automatically generate verification prompts for short/long posts

Note:

- This script is for learning “writing style/sequence.”
- Not for learning chat/RP format.  
“”"

import gc  
import json  
import math  
import random  
import inspection  
import time  
from dataclasses import dataclass, asdict  
from pathlib import Path  
from typing import Any, Dict, List, Optional, Tuple

import torch  
from datasets import Dataset  
from peft import LoraConfig, get\_peft\_model, prepare\_model\_for\_kbit\_training  
from transformers import (  
AutoModelForCausalLM;  
Auto Tokenizer,  
BitsAndBytesConfig,  
DataCollatorForLanguageModeling,  
EarlyStoppingCallback,  
GenerationConfig,  
Trainer,  
TrainerCallback,  
TrainingArguments,  
)

# ===========================================================

# Default settings

# ===========================================================

MODEL\_NAME = “Llama-3.1-8B”  
DATA\_PATH = “train.jsonl”  
OUTPUT\_DIR = “blossom-8b-qlora-story-auto”

MAX\_SEQ\_LENGTH = 1536  
VALID\_RATIO = 0.05  
RANDOM\_SEED = 42  
MIN\_CHARS = 40

# Auto tuning on/off

AUTO\_TUNE = True

# Verification of generation before/after learning on/off

AUTO\_GENERATION\_TEST = True  
GENERATION\_TEST\_DIR = “generation\_check”  
GENERATION\_MAX\_NEW\_TOKENS = 120

# If you want to specify your own verification prompt, put it here.

# If left blank, it is automatically extracted from train.jsonl.

GENERATION\_TEST\_PROMPTS: List[str] =

# Saving/evaluating too small data only takes time, so it is automatically turned off.

SAVE\_TOTAL\_LIMIT = 2

USE\_BF16 = torch.cuda.is\_available() and torch.cuda.is\_bf16\_supported()

# ===========================================================

# data structure

# ===========================================================

@dataclass  
classDataStats:  
file\_size\_bytes: int  
file\_size\_label: str  
samples: int  
total\_chars: int  
total\_tokens: int  
avg\_chars: float  
avg\_tokens: float  
min\_chars: int  
max\_chars: int  
min\_tokens: int  
max\_tokens: int  
estimated\_blocks: int  
skipped\_short: int  
skipped\_duplicate: int  
skipped\_json: int

@dataclass  
class TrainPlan:  
profile: str  
mode: str  
train\_blocks: int  
eval\_blocks: int  
max\_steps: int  
num\_train\_epochs: int  
gradient\_accumulation\_steps: int  
optimize\_updates: int  
learning\_rate: float  
warmup\_ratio: float  
eval\_strategy: str  
save\_strategy: str  
eval\_steps: Optional[int]  
save\_steps: Optional[int]  
load\_best\_model\_at\_end: bool  
early\_stopping: bool

@dataclass  
class PromptItem:  
prompt: str  
position: str  
source\_chars: int  
line\_no: int

# ===========================================================

# utility

# ===========================================================

def set\_seed(seed: int) → None:  
random.seed(seed)  
torch.manual\_seed(seed)  
if torch.cuda.is\_available():  
torch.cuda.manual\_seed\_all(seed)

def file\_size\_label(path: str) → Tuple[int, str]:  
size = Path(path).stat().st\_size  
if size \< 1024:  
return size, f"{size} B"  
if size \< 1024 \* 1024:  
return size, f"{size / 1024:.1f} KB"  
return size, f"{size / (1024 \* 1024):.2f} MB"

def ceil\_div(a: int, b: int) → int:  
return (a + b - 1) // b

def normalize\_text(text: str) → str:  
return " “.join(str(text).replace(”\r\n", “\n”).replace(“\r”, “\n”).split()).strip()

# ===========================================================

# Data load/analysis

# ===========================================================

def load\_jsonl(path: str, tokenizer: AutoTokenizer) → Tuple[Dataset, DataStats, List[Dict[str, Any]]]:  
rows: List[Dict[str, Any]] =   
raw\_items: List[Dict[str, Any]] =   
seen = set()

skipped\_short = 0  
skipped\_duplicate = 0  
skipped\_json = 0

total\_chars = 0  
total\_tokens = 0  
min\_chars: Optional[int] = None  
max\_chars = 0  
min\_tokens: Optional[int] = None  
max\_tokens = 0

with open(path, “r”, encoding=“utf-8-sig”) as f:  
for line\_no, line in enumerate(f, start=1):  
line = line.strip()  
if not line:  
continue

try:  
obj = json.loads(line)  
except json.JSONDecodeError:  
skipped\_json += 1  
print(f"[skip] JSON parsing failed: line {line\_no}")  
continue

text = str(obj.get(“text”, “”)).replace(“\r\n”, “\n”).replace(“\r”, “\n”).strip()  
text = “\n”.join(x.rstrip() for x in text.splitlines()).strip()

if len(text) \< MIN\_CHARS:  
skipped\_short += 1  
continue

```auto
        if text in seen:
            skipped_duplicate += 1
            continue

        token_count = len(tokenizer.encode(text, add_special_tokens=False))
        char_count = len(text)

        seen.add(text)
        rows.append({"text": text, "chars": char_count, "tokens": token_count})
        raw_items.append({"text": text, "chars": char_count, "tokens": token_count, "line_no": line_no})

        total_chars += char_count
        total_tokens += token_count
        min_chars = char_count if min_chars is None else min(min_chars, char_count)
        max_chars = max(max_chars, char_count)
        min_tokens = token_count if min_tokens is None else min(min_tokens, token_count)
        max_tokens = max(max_tokens, token_count)

if not rows:
    raise ValueError(f"학습 가능한 text가 없습니다: {path}")

size_bytes, size_label = file_size_label(path)
stats = DataStats(
    file_size_bytes=size_bytes,
    file_size_label=size_label,
    samples=len(rows),
    total_chars=total_chars,
    total_tokens=total_tokens,
    avg_chars=total_chars / len(rows),
    avg_tokens=total_tokens / len(rows),
    min_chars=min_chars or 0,
    max_chars=max_chars,
    min_tokens=min_tokens or 0,
    max_tokens=max_tokens,
    estimated_blocks=max(1, ceil_div(total_tokens, MAX_SEQ_LENGTH)),
    skipped_short=skipped_short,
    skipped_duplicate=skipped_duplicate,
    skipped_json=skipped_json,
)

print_data_stats(stats)
return Dataset.from_list(rows), stats, raw_items

```

def print\_data\_stats(stats: DataStats) → None:  
print(“=” \* 60)  
print(“[data stats]”)  
print(f"파일 크기 : {stats.file\_size\_label}“)  
print(f"샘플 수 : {stats.samples:,}”)  
print(f"총 문자 수 : {stats.total\_chars:,}“)  
print(f"총 토큰 수 : {stats.total\_tokens:,}”)  
print(f"평균 문자 수 : {stats.avg\_chars:,.1f}“)  
print(f"평균 토큰 수 : {stats.avg\_tokens:,.1f}”)  
print(f"최소/최대 문자 수 : {stats.min\_chars:,} / {stats.max\_chars:,}“)  
print(f"최소/최대 토큰 수 : {stats.min\_tokens:,} / {stats.max\_tokens:,}”)  
print(f"예상 블록 : {stats.estimated\_blocks:,}개 @ MAX\_SEQ\_LENGTH={MAX\_SEQ\_LENGTH}“)  
print(f"제외 : 짧음={stats.skipped\_short:,}, 중복={stats.skipped\_duplicate:,}, JSON오류={stats.skipped\_json:,}”)  
print(“=” \* 60)

# ============================================================

# 토큰화/패킹

# ============================================================

def tokenize\_for\_continuation(examples: Dict[str, List[str]], tokenizer: AutoTokenizer) → Dict[str, Any]:  
texts = [text + tokenizer.eos\_token for text in examples[“text”]]  
return tokenizer(texts, add\_special\_tokens=False)

def pack\_blocks(examples: Dict[str, List[List[int]]]) → Dict[str, List[List[int]]]:  
“”"  
MAX\_SEQ\_LENGTH 단위 packing.  
마지막 remainder를 버리지 않는다.  
KB급 데이터가 1536토큰 미만이라는 이유로 0블록이 되는 문제를 막는다.  
“”"  
result: Dict[str, List[List[int]]] = {}

```auto
for key in ("input_ids", "attention_mask"):
    concatenated: List[int] = []
    for item in examples[key]:
        concatenated.extend(item)

    if not concatenated:
        result[key] = []
        continue

    result[key] = [
        concatenated[i : i + MAX_SEQ_LENGTH]
        for i in range(0, len(concatenated), MAX_SEQ_LENGTH)
    ]

return result

```

# ============================================================

# 자동 학습 계획

# ============================================================

def choose\_profile(stats: DataStats, train\_blocks: int) → Tuple[str, int, int, float, float, int]:  
“”"  
반환:  
profile, target\_optimizer\_updates, grad\_accum, learning\_rate, warmup\_ratio, save\_eval\_steps

```auto
기준:
- 파일 크기 하나만 보지 않는다.
- 총 토큰 수와 packed block 수를 같이 본다.
- KB급 파일이 block 수 때문에 large로 튀지 않게 방지한다.
"""
size_mb = stats.file_size_bytes / (1024 * 1024)
tokens = stats.total_tokens
blocks = train_blocks

rules = [
    # profile, max_size_mb, max_tokens, max_blocks, updates, grad_accum, lr, warmup, save_eval
    ("tiny / KB급", 0.10, 12_000, 8, 8, 1, 1.0e-4, 0.00, 0),
    ("small / 소형", 0.50, 50_000, 32, 16, 1, 9.0e-5, 0.00, 0),
    ("medium / 중형", 2.00, 200_000, 128, 32, 2, 8.0e-5, 0.02, 50),
    ("large / MB급", 10.00, 1_000_000, 768, 64, 4, 7.0e-5, 0.03, 100),
    ("xlarge / 대형", 50.00, 5_000_000, 4_096, 128, 8, 6.0e-5, 0.03, 200),
]

for profile, max_size, max_tokens, max_blocks, updates, grad_accum, lr, warmup, save_eval in rules:
    if size_mb <= max_size and tokens <= max_tokens and blocks <= max_blocks:
        return profile, updates, grad_accum, lr, warmup, save_eval

return "huge / 초대형", 0, 8, 5.0e-5, 0.03, 500

```

def build\_train\_plan(stats: DataStats, train\_blocks: int, eval\_blocks: int) → TrainPlan:  
train\_blocks = max(1, int(train\_blocks))  
eval\_blocks = int(eval\_blocks)

```auto
profile, target_updates, grad_accum, lr, warmup, save_eval = choose_profile(stats, train_blocks)

if target_updates > 0:
    # 작은~대형 대부분은 update 수 기준으로 제한
    mode = "max_steps"
    max_steps = max(1, target_updates * grad_accum)
    num_epochs = 1
    optimizer_updates = target_updates
else:
    # 진짜 초대형만 epoch 기반
    mode = "epoch"
    max_steps = -1
    num_epochs = 2
    optimizer_updates = max(1, (train_blocks * num_epochs) // grad_accum)

# 작은 데이터는 eval/save가 오히려 시간 낭비
if save_eval <= 0 or train_blocks <= 32 or eval_blocks <= 0:
    eval_strategy = "no"
    save_strategy = "no"
    eval_steps = None
    save_steps = None
    load_best = False
    early_stopping = False
else:
    eval_strategy = "steps"
    save_strategy = "steps"
    eval_steps = save_eval
    save_steps = save_eval
    load_best = True
    early_stopping = True

return TrainPlan(
    profile=profile,
    mode=mode,
    train_blocks=train_blocks,
    eval_blocks=eval_blocks,
    max_steps=max_steps,
    num_train_epochs=num_epochs,
    gradient_accumulation_steps=grad_accum,
    optimizer_updates=optimizer_updates,
    learning_rate=lr,
    warmup_ratio=warmup,
    eval_strategy=eval_strategy,
    save_strategy=save_strategy,
    eval_steps=eval_steps,
    save_steps=save_steps,
    load_best_model_at_end=load_best,
    early_stopping=early_stopping,
)

```

def print\_train\_plan(plan: TrainPlan) → None:  
print(“=” \* 60)  
print(“[auto training plan]”)  
print(f"profile : {plan.profile}“)  
print(f"mode : {plan.mode}”)  
print(f"train/eval blocks : {plan.train\_blocks:,} / {plan.eval\_blocks:,}“)  
print(f"epochs : {plan.num\_train\_epochs}”)  
print(f"max steps : {plan.max\_steps}“)  
print(f"grad accumulation : {plan.gradient\_accumulation\_steps}”)  
print(f"optimizer updates : {plan.optimizer\_updates:,}“)  
print(f"learning rate : {plan.learning\_rate}”)  
print(f"warmup ratio : {plan.warmup\_ratio}“)  
print(f"eval/save : {plan.eval\_strategy} / {plan.save\_strategy}”)  
print(“=” \* 60)

# ============================================================

# 검증 프롬프트 자동 생성

# ============================================================

def safe\_cut\_at\_sentence(text: str, min\_len: int = 35, max\_len: int = 100) → str:  
text = normalize\_text(text)  
if not text:  
return “”

```auto
if len(text) <= max_len:
    return text

endings = ["다.", "요.", "까?", "까.", "!", "?", ".", "”", "\""]
best = -1
for ending in endings:
    pos = text.rfind(ending, min_len, max_len)
    if pos >= 0:
        best = max(best, pos + len(ending))

if best >= min_len:
    return text[:best].strip()

cut = text.rfind(" ", min_len, max_len)
if cut < min_len:
    cut = max_len
return text[:cut].strip()

```

def make\_segment\_prompt(text: str, ratio: float, target\_len: int = 95) → str:  
“”"  
긴 글의 앞/중/후반 위치에서 자연스러운 짧은 이어쓰기 프롬프트를 만든다.  
긴 원문 전체를 넣지 않는다.  
“”"  
text = normalize\_text(text)  
if not text:  
return “”

```auto
n = len(text)
start = int(n * ratio)
start = max(0, min(start, max(0, n - target_len - 1)))

window_start = max(0, start - 140)
window_end = min(n, start + target_len + 180)
window = text[window_start:window_end]

local_start = start - window_start

# 문장 중간에서 시작하는 것을 줄인다.
prev_candidates = []
for ending in ["다.", "요.", "!", "?", "."]:
    prev_candidates.append(window.rfind(ending, 0, local_start))
prev = max(prev_candidates)

if prev >= 0 and local_start - prev <= 120:
    local_start = prev + 2

segment = window[local_start : local_start + target_len + 80].strip()
return safe_cut_at_sentence(segment, min_len=30, max_len=target_len)

```

def build\_prompt\_items(raw\_items: List[Dict[str, Any]], max\_prompts: int = 6) → List[PromptItem]:  
if GENERATION\_TEST\_PROMPTS:  
manual: List[PromptItem] =   
for idx, p in enumerate(GENERATION\_TEST\_PROMPTS, start=1):  
prompt = safe\_cut\_at\_sentence(p, min\_len=20, max\_len=100)  
if prompt:  
manual.append(PromptItem(prompt=prompt, position=“manual”, source\_chars=len(p), line\_no=idx))  
return manual[:max\_prompts]

```auto
candidates: List[PromptItem] = []

for item in raw_items:
    text = item["text"]
    chars = int(item["chars"])
    line_no = int(item["line_no"])

    if chars < 600:
        positions = [("beginning", 0.00)]
    elif chars < 2500:
        positions = [("beginning", 0.00), ("middle", 0.45)]
    else:
        positions = [("beginning", 0.00), ("middle", 0.45), ("late", 0.78)]

    for pos_name, ratio in positions:
        prompt = make_segment_prompt(text, ratio=ratio)
        if len(prompt) >= 20:
            candidates.append(PromptItem(prompt=prompt, position=pos_name, source_chars=chars, line_no=line_no))

if not candidates:
    return [
        PromptItem("오늘은 이상하게 마음이 복잡했다.", "fallback", 0, 0),
        PromptItem("나는 조용히 고개를 들었다.", "fallback", 0, 0),
        PromptItem("그때 문득 이상한 생각이 들었다.", "fallback", 0, 0),
    ]

def score(item: PromptItem) -> Tuple[int, int, int]:
    # 긴 글의 middle/late를 우선 확보
    pos_score = {"middle": 0, "late": 1, "beginning": 2}.get(item.position, 3)
    return (pos_score, -item.source_chars, item.line_no)

candidates.sort(key=score)

selected: List[PromptItem] = []
seen_prompt_keys = set()
seen_positions = set()

# 1차: position 다양성 확보
for item in candidates:
    if len(selected) >= max_prompts:
        break
    key = item.prompt[:80]
    if key in seen_prompt_keys:
        continue
    if item.position in seen_positions and len(seen_positions) < 3:
        continue
    selected.append(item)
    seen_prompt_keys.add(key)
    seen_positions.add(item.position)

# 2차: 남은 칸 채우기
for item in candidates:
    if len(selected) >= max_prompts:
        break
    key = item.prompt[:80]
    if key in seen_prompt_keys:
        continue
    selected.append(item)
    seen_prompt_keys.add(key)

return selected[:max_prompts]

```

# ============================================================

# 생성 검증/리포트

# ============================================================

def generate\_samples(  
model: AutoModelForCausalLM,  
tokenizer: AutoTokenizer,  
prompts: List[PromptItem],  
label: str,  
out\_dir: str,  
) → Path:  
out\_path = Path(out\_dir)  
out\_path.mkdir(parents=True, exist\_ok=True)  
file\_path = out\_path / f"{label}\_sample.txt"

```auto
was_training = model.training
model.eval()

gen_config = GenerationConfig(
    max_new_tokens=GENERATION_MAX_NEW_TOKENS,
    do_sample=True,
    temperature=0.72,
    top_p=0.88,
    repetition_penalty=1.18,
    no_repeat_ngram_size=4,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

lines: List[str] = []
lines.append("=" * 80)
lines.append(f"[{label} generation sample]")
lines.append("=" * 80)

with torch.no_grad():
    for idx, item in enumerate(prompts, start=1):
        prompt = item.prompt
        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
        input_len = inputs["input_ids"].shape[-1]

        output_ids = model.generate(**inputs, generation_config=gen_config)

        continuation = tokenizer.decode(output_ids[0][input_len:], skip_special_tokens=True).strip()
        full_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)

        lines.append("")
        lines.append("-" * 80)
        lines.append(
            f"[prompt {idx}] position={item.position}, source_chars={item.source_chars}, line={item.line_no}"
        )
        lines.append(prompt)
        lines.append("")
        lines.append(f"[continuation {idx}]")
        lines.append(continuation if continuation else "(생성된 continuation 없음)")
        lines.append("")
        lines.append(f"[full output {idx}]")
        lines.append(full_text)

file_path.write_text("\n".join(lines), encoding="utf-8")

if was_training:
    model.train()

print(f"[generation] saved: {file_path.resolve()}")
return file_path

```

def simple\_repetition\_score(text: str) → Dict[str, Any]:  
words = normalize\_text(text).split()  
if not words:  
return {“words”: 0, “unique\_words”: 0, “unique\_ratio”: 0.0, “dot\_count”: text.count(“.”)}

```auto
unique_words = len(set(words))
return {
    "words": len(words),
    "unique_words": unique_words,
    "unique_ratio": round(unique_words / max(1, len(words)), 4),
    "dot_count": text.count("."),
}

```

def write\_train\_report(  
stats: DataStats,  
plan: TrainPlan,  
prompts: List[PromptItem],  
before\_path: Optional[Path],  
after\_path: Optional[Path],  
elapsed\_seconds: Optional[float],  
) → Path:  
report\_dir = Path(GENERATION\_TEST\_DIR)  
report\_dir.mkdir(parents=True, exist\_ok=True)  
report\_path = report\_dir / “train\_report.txt”

```auto
lines: List[str] = []
lines.append("=" * 80)
lines.append("[train report]")
lines.append("=" * 80)
lines.append("")
lines.append("[config]")
lines.append(f"model : {MODEL_NAME}")
lines.append(f"data path : {DATA_PATH}")
lines.append(f"output dir : {OUTPUT_DIR}")
lines.append(f"max seq length : {MAX_SEQ_LENGTH}")
lines.append(f"bf16/fp16 : {USE_BF16} / {not USE_BF16}")
lines.append("")

lines.append("[data stats]")
for k, v in asdict(stats).items():
    lines.append(f"{k:24}: {v}")
lines.append("")

lines.append("[train plan]")
for k, v in asdict(plan).items():
    lines.append(f"{k:24}: {v}")
lines.append("")

if elapsed_seconds is not None:
    lines.append("[time]")
    lines.append(f"elapsed_seconds : {elapsed_seconds:.1f}")
    lines.append(f"elapsed_minutes : {elapsed_seconds / 60:.2f}")
    lines.append("")

lines.append("[generation files]")
lines.append(f"before : {before_path}")
lines.append(f"after : {after_path}")
lines.append("")

lines.append("[prompts]")
for idx, item in enumerate(prompts, start=1):
    lines.append(
        f"{idx}. position={item.position}, source_chars={item.source_chars}, "
        f"line={item.line_no} | {item.prompt}"
    )
lines.append("")

lines.append("[judge guide]")
lines.append("- before/after continuation의 문체 변화가 있는지 본다.")
lines.append("- after가 원문을 그대로 베끼면 과적합 가능성이 있다.")
lines.append("- after가 점(.) 반복, 같은 구절 반복, 화자명 반복을 보이면 update 수를 줄인다.")
lines.append("- after가 before와 거의 같으면 update 수를 늘리거나 데이터량을 늘린다.")
lines.append("- 짧은 글은 beginning continuation, 긴 글은 middle/late continuation 변화까지 같이 본다.")
lines.append("- KB급 데이터는 지식 주입보다 문체 힌트 정도로 보는 것이 안전하다.")

report_path.write_text("\n".join(lines), encoding="utf-8")
print(f"[report] saved: {report_path.resolve()}")
return report_path

```

# ============================================================

# TrainingArguments 호환 처리

# ============================================================

def add\_training\_args\_compat(kwargs: Dict[str, Any]) → Dict[str, Any]:  
sig = inspect.signature(TrainingArguments. **init** )

```auto
eval_value = kwargs.pop("_eval_strategy_value", "steps")
kwargs = {k: v for k, v in kwargs.items() if v is not None}

if "eval_strategy" in sig.parameters:
    kwargs["eval_strategy"] = eval_value
elif "evaluation_strategy" in sig.parameters:
    kwargs["evaluation_strategy"] = eval_value

if "optim" in sig.parameters:
    kwargs["optim"] = "paged_adamw_8bit"

if "gradient_checkpointing" in sig.parameters:
    kwargs["gradient_checkpointing"] = True

return kwargs

```

class PeriodicCleanupCallback(TrainerCallback):  
def **init** (self, every\_n\_steps: int = 50):  
self.every\_n\_steps = every\_n\_steps

```auto
def on_step_end(self, args, state, control, **kwargs):
    if state.global_step and state.global_step % self.every_n_steps == 0:
        gc.collect()
        if torch.cuda.is_available():
            torch.cuda.empty_cache()

```

# ============================================================

# main

# ============================================================

def main() → None:  
set\_seed(RANDOM\_SEED)

```auto
tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME,
    use_fast=True,
    trust_remote_code=True,
)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

tokenizer.padding_side = "right"

dataset, stats, raw_items = load_jsonl(DATA_PATH, tokenizer)

split = dataset.train_test_split(
    test_size=VALID_RATIO,
    seed=RANDOM_SEED,
    shuffle=True,
)

def _tokenize(examples):
    return tokenize_for_continuation(examples, tokenizer)

tokenized = split.map(
    _tokenize,
    batched=True,
    remove_columns=split["train"].column_names,
    desc="Tokenizing",
)

train_dataset = tokenized["train"].map(
    pack_blocks,
    batched=True,
    desc="Packing train blocks",
)

eval_dataset = tokenized["test"].map(
    pack_blocks,
    batched=True,
    desc="Packing eval blocks",
)

if len(train_dataset) == 0:
    raise ValueError("train_dataset이 0블록입니다. 데이터 또는 MAX_SEQ_LENGTH 설정을 확인하세요.")

if len(eval_dataset) == 0:
    print("[warn] eval block이 0개라서 eval을 끕니다.")

print("=" * 60)
print("[packed blocks]")
print(f"train blocks : {len(train_dataset):,}")
print(f"eval blocks : {len(eval_dataset):,}")
print(f"tokens per block : {MAX_SEQ_LENGTH:,}")
print("=" * 60)

plan = build_train_plan(stats, len(train_dataset), len(eval_dataset)) if AUTO_TUNE else TrainPlan(
    profile="manual",
    mode="epoch",
    train_blocks=len(train_dataset),
    eval_blocks=len(eval_dataset),
    max_steps=-1,
    num_train_epochs=50,
    gradient_accumulation_steps=8,
    optimizer_updates=max(1, (len(train_dataset) * 50) // 8),
    learning_rate=8e-5,
    warmup_ratio=0.03,
    eval_strategy="steps",
    save_strategy="steps",
    eval_steps=100,
    save_steps=100,
    load_best_model_at_end=True,
    early_stopping=True,
)
print_train_plan(plan)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16 if USE_BF16 else torch.float16,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

model.config.use_cache = False
model = prepare_model_for_kbit_training(model)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

prompts = build_prompt_items(raw_items, max_prompts=6) if AUTO_GENERATION_TEST else []
before_path: Optional[Path] = None
after_path: Optional[Path] = None

if AUTO_GENERATION_TEST:
    before_path = generate_samples(
        model=model,
        tokenizer=tokenizer,
        prompts=prompts,
        label="before",
        out_dir=GENERATION_TEST_DIR,
    )

collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

training_args_kwargs = dict(
    output_dir=OUTPUT_DIR,
    _eval_strategy_value=plan.eval_strategy,

    per_device_train_batch_size=1,
    gradient_accumulation_steps=plan.gradient_accumulation_steps,

    num_train_epochs=plan.num_train_epochs,
    max_steps=plan.max_steps,
    learning_rate=plan.learning_rate,
    lr_scheduler_type="cosine",
    warmup_ratio=plan.warmup_ratio,
    weight_decay=0.01,

    logging_steps=10,
    eval_steps=plan.eval_steps,
    save_strategy=plan.save_strategy,
    save_steps=plan.save_steps,
    save_total_limit=SAVE_TOTAL_LIMIT,

    load_best_model_at_end=plan.load_best_model_at_end,
    metric_for_best_model="eval_loss" if plan.load_best_model_at_end else None,
    greater_is_better=False if plan.load_best_model_at_end else None,

    bf16=USE_BF16,
    fp16=not USE_BF16,

    report_to="none",
    remove_unused_columns=False,
    dataloader_num_workers=0,
)

training_args = TrainingArguments(**add_training_args_compat(training_args_kwargs))

callbacks: List[TrainerCallback] = [PeriodicCleanupCallback(every_n_steps=50)]
if plan.early_stopping:
    callbacks.insert(0, EarlyStoppingCallback(early_stopping_patience=3))

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset if plan.eval_strategy != "no" else None,
    data_collator=collator,
    callbacks=callbacks,
)

start_time = time.time()
trainer.train()
elapsed = time.time() - start_time

if AUTO_GENERATION_TEST:
    after_path = generate_samples(
        model=model,
        tokenizer=tokenizer,
        prompts=prompts,
        label="after",
        out_dir=GENERATION_TEST_DIR,
    )

    write_train_report(
        stats=stats,
        plan=plan,
        prompts=prompts,
        before_path=before_path,
        after_path=after_path,
        elapsed_seconds=elapsed,
    )

model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

print(f"[done] saved adapter/tokenizer to: {Path(OUTPUT_DIR).resolve()}")
if AUTO_GENERATION_TEST:
    print(f"[done] generation check dir: {Path(GENERATION_TEST_DIR).resolve()}")

```

if **name** == “ **main** ”:  
main()  
I was trying to use the train.jsonl file universally. This is an attempt to automatically process cases where the number of samples is large but the amount of text is small, or, conversely, when the number of samples is small but the amount of text is large.  
It was a learning experience.  
However, it ended up being an explanatory model rather than the creative model I wanted.  
Chetgpt says it’s my fault.  
I’m insisting that I chose the wrong base model.  
Chet gpt is the one who recommended llama-3-Korean-Bllossom-8B, and now he is saying that I chose the wrong base model, which is absurd.  
What do you guys think?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 6, 2026, 10:00am UTC](https://discuss.huggingface.co/t/i-am-fine-tuning-llama-3-2-8bit-lora-using-chetgpt-but-i-am-doubtful-whether-the-direction-is-correct/177482/2 "2026-07-06T10:00:38Z")

</div>

Hmm… From what you describe, I get the impression that several different tasks may have been mixed together. In that situation, ChatGPT tends by default to **prioritize the task as the user framed it and search for an answer “inside” that frame, so it can sometimes lead you into a dead end**. I cannot tell whether that is what happened in your case, but it may be safer to first locate where you currently are in the larger problem space:

* * *

## Short answer

Your concern about the starting checkpoint is reasonable.

[`MLP-KTLim/llama-3-Korean-Bllossom-8B`](https://huggingface.co/MLP-KTLim/llama-3-Korean-Bllossom-8B) is not a plain Korean base model. Its model card says that it received Korean instruction tuning and DPO-based alignment, and its official example uses a helpful-assistant system prompt and a chat template.

That makes the checkpoint a plausible contributor to explanatory or assistant-like output. It is not the cleanest baseline for testing pure prose continuation.

However, I would separate two questions:

1. **Is the current training objective aimed at the final writing behavior you want?**
2. **Did this particular training run work as intended?**

Both may matter.

A raw-prose LoRA can plausibly become a useful **prose-continuation or style adapter**. It does not, by itself, train the whole stack normally involved in long-form writing: long-output behavior, planning, persistent story state, continuity checking and revision.

Separately, the current script has a few mechanical and evaluation points worth checking before using its output as evidence that the whole direction failed.

The most informative checks before another full run would be:

- compare the checkpoint **without the adapter** under raw continuation, chat request and assistant-prefill inputs;
- verify the actual optimizer-update count;
- inspect whether genuine EOS tokens remain supervised after collation;
- compare adapter-off and adapter-on outputs with unseen prompts and paired generation settings.

## Where the current training run appears to sit

```plaintext
Long-form writing
│
├─ Local prose generation
│ ├─ vocabulary and sentence style
│ ├─ dialogue / description balance
│ ├─ short scene continuation
│ └─ prose-mode behavior
│ ↑
│ └─ The current raw-text LoRA mainly operates here
│
├─ Long-output behavior
├─ Instruction-to-story behavior
├─ Plot and chapter planning
├─ Persistent character / world / timeline state
├─ Continuity checking
└─ Revision and selection

```

The current dataset format, `{"text": "..."}`, and causal next-token objective mainly teach local prose distribution and continuation.

That is a valid component. It is not the same thing as a complete long-form writing system.

A useful distinction is:

```plaintext
Long context
≠
Long output
≠
A coherent long-form narrative

```

A model may accept a long input but still stop after a short response. It may generate many words but lose plot constraints. It may write attractive individual scenes while forgetting character knowledge, chronology or unresolved plot threads across chapters.

## Choose the route by the intended final behavior

| Intended behavior | More direct route | Possible role of the current LoRA |
| --- | --- | --- |
| Continue an existing passage for a few paragraphs | Korean-capable base model + raw-prose LM/LoRA | Main component |
| Write a scene or chapter from instructions | Instruct model + prompt/completion or conversational SFT | Optional prose/style component |
| Produce a very long response in one request | Long-output examples, length conditioning and/or decomposition | May improve prose, but does not provide length control alone |
| Maintain a coherent multi-chapter story | Planner + persistent story state + writer + checker/revision | Writer component |
| Interactive RP or collaborative writing | Chat/RP behavior + character/lore/session memory | Optional narration or character-style component |

So there is not only one possible verdict.

### If the goal is short prose continuation

The current direction still has a destination. I would test a Korean-capable **base** checkpoint, repair the evaluation, and measure whether the adapter improves unseen continuation.

### If the goal is instruction-to-story generation

Raw prose alone does not teach the relationship between an instruction and the desired story response. Prompt/completion examples would be more direct.

### If the goal is coherent multi-chapter writing

The current LoRA may remain useful, but mainly as the prose-writing layer inside a larger workflow.

## Three reasonable next paths

These are alternatives, not a list of mandatory homework.

### Path A — Repair and interpret the current experiment

Choose this if the main target is prose continuation and you want to know whether the current adapter actually works.

Focus on:

- checkpoint behavior before training;
- optimizer-step count;
- EOS supervision;
- held-out paired evaluation.

### Path B — Test the model inside an existing writing workflow

Choose this if the larger goal is long-form writing and you want to see how much planning, memory and revision change the result without building everything yourself.

For example, the same adapter could be tested as the writer model inside an open-source writing environment or pipeline.

### Path C — Retrain with a checkpoint and objective matched to the target

Choose this if the current objective does not match the intended interface.

- raw continuation → base checkpoint + raw prose;
- instruction-to-story → instruct checkpoint + prompt/completion data;
- RP/co-writing → chat/RP data and runtime character/lore state;
- long-output writing → long completions and/or decomposed generation.

> **Why long-form writing is normally split into several capabilities**
>
> The [TRL SFTTrainer documentation](https://huggingface.co/docs/trl/sft_trainer) and [dataset-format guide](https://huggingface.co/docs/trl/main/dataset_formats) distinguish standard language-modeling data, prompt/completion data and conversational data.
> 
> They may all use next-token prediction internally, but the examples teach different contracts.
> 
> ### Raw language modeling
> 
> ```json
> {
> "text": "The rain had continued for three days. Yuna stood by the window..."
> }
> 
> ```
> 
> This directly teaches prose distribution and continuation.
> 
> ### Prompt/completion writing
> 
> ```json
> {
> "prompt": "Write a restrained first-person horror scene set in an empty railway station. Output only the prose.",
> "completion": "The last train had already left when I noticed that the clock..."
> }
> 
> ```
> 
> This teaches:
> 
> ```plaintext
> instruction and constraints
> →
> story response
> 
> ```
> 
> ### Planning
> 
> ```plaintext
> story so far
> + unresolved plot threads
> + chapter objective
> →
> next chapter plan
> 
> ```
> 
> ### Revision
> 
> ```plaintext
> draft
> + continuity report
> →
> revised draft
> 
> ```
> 
> The important statement is not that fine-tuning can never teach planning, memory or revision. It can, with suitable examples and objectives.
> 
> The narrower point is:
> 
> > The current raw-prose objective does not directly supervise those capabilities.
> 
> The [LongWriter paper](https://arxiv.org/abs/2408.07055) reported that models with long input contexts could still struggle to produce long outputs when their supervised post-training data lacked long completions. Its [AgentWrite implementation](https://github.com/THUDM/LongWriter) decomposes an ultra-long task into planned sections; the resulting long outputs were then used as SFT data.
> 
> Story-generation systems also commonly add explicit structure around the writer:
> 
> - [Re3](https://arxiv.org/abs/2210.06774) uses planning, repeated story-state prompting, reranking and revision.
> - Its [open-source implementation](https://github.com/yangkevin2/emnlp22-re3-story-generation) shows how these parts are separated.
> - [DOME](https://arxiv.org/abs/2412.13575) combines dynamic hierarchical outlining with temporal memory.
> - [DOC](https://arxiv.org/abs/2212.10077) uses detailed outline control for longer stories.
> 
> These are examples rather than mandatory architectures. Their common lesson is that attractive next-token generation is only one part of long-form writing.

> **Checkpoint behavior and input-format checks**
>
> The Bllossom model card explicitly describes:
> 
> - Korean instruction tuning;
> - DPO-based alignment;
> - a helpful-assistant system prompt;
> - chat-template usage;
> - EOS and `<|eot_id|>` as generation terminators.
> 
> This does not prove that the checkpoint caused the observed result. The missing control is its output before fine-tuning.
> 
> Using one newly written Korean story prefix, compare:
> 
> ### A. Raw continuation
> 
> Tokenize only the prose prefix and continue it.
> 
> ### B. Chat request
> 
> Ask the model to continue the passage using its official chat template.
> 
> ### C. Assistant prefill
> 
> Place the prose prefix inside the final assistant message and continue that message.
> 
> The Transformers [chat-template guide](https://huggingface.co/docs/transformers/chat_templating) describes `continue_final_message=True` as a response-prefill mechanism.
> 
> Possible interpretations:
> 
> | Observation | More likely explanation |
> | --- | --- |
> | Raw continuation is already explanatory without the adapter | Existing checkpoint behavior |
> | Only the chat request is explanatory | Prompt/template path |
> | Assistant prefill produces prose | Response-start format |
> | All three behave similarly | Look more strongly at training, data and decoding |
> | Only adapter-on output becomes more explanatory | Training-run or dataset issue |
> 
> It is also worth saving raw generated token IDs.
> 
> Bllossom’s official example uses both EOS and `<|eot_id|>` as terminators, while the script appears to pass only `tokenizer.eos_token_id`. Since `skip_special_tokens=True` hides special tokens, an EOT token followed by additional generation may be invisible in the decoded comparison.
> 
> Check:
> 
> ```plaintext
> Was EOS generated?
> Was <|eot_id|> generated?
> Did generation continue after EOT?
> 
> ```
> 
> Also record the exact repository and revision. The thread title, the placeholder model name in the script and the described Bllossom checkpoint do not all use the same Llama version label. Debugging should be based on the exact model ID, tokenizer and revision rather than only “Llama 3.1” or “Llama 3.2.”

> **Mechanical checks in the current training script**
>
> ## 1. `max_steps` and gradient accumulation
> 
> The plan appears to do:
> 
> ```python
> max_steps = target_updates * grad_accum
> optimizer_updates = target_updates
> 
> ```
> 
> The official [Trainer documentation](https://huggingface.co/docs/transformers/main_classes/trainer) and [TrainerState documentation](https://huggingface.co/docs/transformers/main_classes/callback) describe `global_step` and `max_steps` in terms of completed update steps.
> 
> Gradient accumulation controls how many micro-batches contribute to one update. It is normally not multiplied into the desired optimizer-update count a second time.
> 
> If `target_updates` really means optimizer updates, the expected form would normally be:
> 
> ```python
> max_steps = target_updates
> 
> ```
> 
> Under the current profiles:
> 
> | Profile | Printed target updates | Gradient accumulation | `max_steps` passed to Trainer |
> | --- | --- | --- | --- |
> | tiny | 8 | 1 | 8 |
> | small | 16 | 1 | 16 |
> | medium | 32 | 2 | 64 |
> | large | 64 | 4 | 256 |
> | xlarge | 128 | 8 | 1024 |
> 
> That would make medium, large and xlarge perform 2×, 4× and 8× the update count printed as `optimizer_updates`, assuming normal Trainer semantics.
> 
> This does not directly explain an instructional tone, but it changes:
> 
> - effective token exposure;
> - passes over the data;
> - scheduler behavior;
> - overfitting risk;
> - repetition and memorization risk;
> - the interpretation of the report.
> 
> A tiny instrumentation run can settle this:
> 
> ```plaintext
> micro-batches consumed
> backward passes
> optimizer steps
> trainer.state.global_step
> tokens seen
> approximate dataset passes
> 
> ```
> 
> ## 2. EOS supervision
> 
> The script:
> 
> 1. appends EOS to every source row;
> 2. assigns EOS as PAD if the tokenizer has no PAD token;
> 3. uses `DataCollatorForLanguageModeling`.
> 
> Transformers [issue #23530](https://github.com/huggingface/transformers/issues/23530) demonstrates a configuration where `pad_token == eos_token` caused genuine EOS labels to be replaced with `-100`, not only padding positions.
> 
> That report used a particular tokenizer and library version, so it is not proof that the same thing happened here. It is simply inexpensive to audit.
> 
> ```python
> eos_positions = batch["input_ids"].eq(tokenizer.eos_token_id)
> supervised_eos = eos_positions & batch["labels"].ne(-100)
> masked_attended_eos = (
> eos_positions
> & batch["attention_mask"].eq(1)
> & batch["labels"].eq(-100)
> )
> 
> print("EOS in input:", eos_positions.sum().item())
> print("Supervised EOS:", supervised_eos.sum().item())
> print("Attended but masked EOS:", masked_attended_eos.sum().item())
> 
> ```
> 
> If all genuine document-ending EOS tokens are masked, the model is not learning those boundaries.
> 
> ## 3. Automatic training-plan heuristics
> 
> The script chooses learning rate, updates, accumulation and evaluation behavior from file size, token count and block count.
> 
> That is convenient, but equal token counts can describe very different data:
> 
> - one novel;
> - several unrelated novels;
> - independent scenes;
> - one author;
> - many authors;
> - a base checkpoint;
> - an instruction/DPO checkpoint.
> 
> Rather than assuming one calculated update count is correct, a small checkpoint curve can be more informative:
> 
> ```plaintext
> 8 updates
> 16 updates
> 32 updates
> 64 updates
> 
> ```
> 
> Evaluate all four on the same held-out prompts.
> 
> That shows whether style adaptation improves gradually, whether repetition begins, and whether memorization appears before the final checkpoint.
> 
> ## 4. Early stopping
> 
> If evaluation is infrequent relative to the total number of update steps, early stopping may never receive enough evaluation events to act.
> 
> For example, a patience of three evaluations cannot meaningfully stop a run that performs only one scheduled evaluation before completion.
> 
> ## 5. QLoRA itself
> 
> The script uses 4-bit NF4 quantization, double quantization, `prepare_model_for_kbit_training`, and LoRA on the main attention and MLP projection layers.
> 
> That is recognizably within the ordinary QLoRA family described in the [PEFT quantization guide](https://huggingface.co/docs/peft/developer_guides/quantization).
> 
> It does not establish that every hyperparameter is ideal, but 4-bit QLoRA alone is not a strong explanation for a semantic shift toward explanatory prose.
> 
> I would inspect checkpoint behavior, steps, labels, data and evaluation before treating quantization as the primary cause.

> **Why the current before/after comparison is difficult to interpret**
>
> The comparison generation uses:
> 
> ```python
> do_sample=True
> temperature=0.72
> top_p=0.88
> repetition_penalty=1.18
> no_repeat_ngram_size=4
> 
> ```
> 
> A seed is set near the beginning of the program, but the same seed is not restored immediately before both the before-training and after-training generations.
> 
> Training, model initialization and data loading consume random state. Therefore, the two samples are not necessarily paired random draws.
> 
> The automatic comparison prompts also appear to be created from the original `raw_items`, before the train/test split. A prompt may therefore come from a training row.
> 
> This mixes:
> 
> - adapter effects;
> - random sampling variation;
> - memorization;
> - generalization;
> - decoding constraints.
> 
> ## Diagnostic comparison
> 
> Use:
> 
> ```plaintext
> same unseen prompt
> adapter off / adapter on
> do_sample=False
> same stopping tokens
> same output limit
> minimal repetition constraints
> 
> ```
> 
> ## Creative-quality comparison
> 
> Use:
> 
> ```plaintext
> same prompt set
> same generation configuration
> same explicit seed list before and after
> multiple seeds per prompt
> blind comparison where practical
> 
> ```
> 
> ## Useful holdout levels
> 
> 1. A withheld passage from the same work.
> 2. A withheld chapter or work.
> 3. A newly written passage with similar stylistic goals.
> 4. A neutral passage in another style or genre.
> 
> If rows are scenes from the same novel, a random row-level split still shares characters, world facts, author style and plot context across train and evaluation.
> 
> Depending on the question, chapter-group or work-level holdout may be more informative.
> 
> ## Separate local-adapter evaluation from long-form-system evaluation
> 
> ### Local prose evaluation
> 
> - Does it continue in prose mode?
> - Is the local transition coherent?
> - Does it maintain POV and tense?
> - Did the target style appear on unseen text?
> - Did repetition increase?
> - Does it stop naturally?
> - Is it reproducing training text?
> 
> ### Long-form-system evaluation
> 
> - Does it maintain character state?
> - Does it preserve chronology and causality?
> - Does it follow the outline?
> - Does it track unresolved plot threads?
> - Does quality degrade over chapters?
> - Does revision repair detected contradictions?
> 
> A LoRA can succeed on the first list while the overall writing system still fails on the second. That would be a component-boundary result, not necessarily a failed adapter.

> **How to read the possible outcomes**
>
> | Observation | Likely next interpretation |
> | --- | --- |
> | Bllossom is explanatory before training in raw continuation | Checkpoint behavior is a strong contributor |
> | Only chat requests are explanatory | Prompt/template path is important |
> | Assistant prefill fixes the mode | Response-start formatting may be sufficient for some uses |
> | Adapter-on alone increases explanatory output | Inspect data, steps and training trajectory |
> | Genuine EOS labels are masked | Repair labels/collation before retraining |
> | Actual updates exceed the printed target | Repair the training-plan calculation |
> | Only training-derived prompts improve | Memorization or overfitting |
> | Unseen prompts show stable stylistic improvement | LoRA works within its intended local scope |
> | Prose improves but multi-chapter consistency does not | The adapter is a writer component, not the whole system |
> | A base checkpoint adapts while Bllossom does not | Interaction with existing alignment becomes more plausible |
> | A capable model plus planning/memory works without custom training | Orchestration may be a larger bottleneck than adaptation |
> | Every model fails at the same stage in one framework | Inspect the framework’s prompt/state contract |

> **Existing open-source systems that may help define or test the destination**
>
> The choice is not only:
> 
> ```plaintext
> keep fine-tuning
> or
> build a large writing pipeline from scratch
> 
> ```
> 
> Open-source systems exist at several levels.
> 
> ## Human-guided continuation: KoboldCpp
> 
> [KoboldCpp](https://github.com/LostRuins/koboldcpp) is an AGPL-licensed local GGUF runtime with the KoboldAI Lite interface.
> 
> It includes:
> 
> - Storywriter mode;
> - persistent stories;
> - Memory;
> - Author’s Note;
> - World Info;
> - editing, retry and undo;
> - swappable local models and samplers.
> 
> It is not an automatic multi-chapter planner. It is useful for checking whether a checkpoint or adapter works as a human-guided prose-continuation engine under a more appropriate writing interface.
> 
> ## RP, lore and collaborative writing: SillyTavern
> 
> [SillyTavern](https://github.com/SillyTavern/SillyTavern) is a mature open-source frontend for chat, RP and interactive writing.
> 
> Its World Info/lorebook and [Data Bank](https://docs.sillytavern.app/usage/core-concepts/data-bank/) mechanisms are examples of keeping mutable story information outside model weights and injecting relevant information at runtime.
> 
> It is most relevant if the intended goal is character-driven co-writing, RP or persistent lore rather than autonomous novel generation.
> 
> ## Integrated writing workflow: InkOS
> 
> [InkOS](https://github.com/Narcooo/inkos) is an AGPL-licensed open-source story-creation system for novels, scripts and related long-form projects.
> 
> Its source exposes a broader workflow involving planning, context composition, drafting, persistent files, auditing and revision.
> 
> Its value here is not proof that it is the best novel generator. It is a concrete, inspectable reference for what a full writing stack may contain beyond the writer model.
> 
> ## Smaller model-swappable pipeline: GOAT Storytelling Agent
> 
> [GOAT Storytelling Agent](https://github.com/GOAT-AI-lab/GOAT-Storytelling-Agent) implements a top-down flow roughly like:
> 
> ```plaintext
> topic
> → book specification
> → chapter plot
> → scene split
> → scene generation
> 
> ```
> 
> It is closer to a developer or research pipeline than a polished writing application, but it is useful for testing a model as one stage of a larger system.
> 
> ## Research references
> 
> - [LongWriter / AgentWrite](https://github.com/THUDM/LongWriter): long-output decomposition and long-output training data.
> - [Re3 implementation](https://github.com/yangkevin2/emnlp22-re3-story-generation): planning, state prompting, reranking and revision.
> - [DOME](https://arxiv.org/abs/2412.13575): dynamic outlines and temporal memory.
> - [DOC](https://arxiv.org/abs/2212.10077): detailed outline control.
> 
> Research repositories may have older dependencies, proprietary API assumptions or narrow experimental setups. They are best treated as design references unless you intend to reproduce the research environment.
> 
> ## Using an existing system as a diagnostic baseline
> 
> Connect the same model conditions to the same workflow:
> 
> ```plaintext
> A. Bllossom without the adapter
> B. Bllossom with the current adapter
> C. A comparison checkpoint
> 
> ```
> 
> Keep constant:
> 
> ```plaintext
> outline
> story state
> chapter objective
> prompt templates
> sampler
> revision count
> framework version
> 
> ```
> 
> Then compare:
> 
> | Result | Possible interpretation |
> | --- | --- |
> | Planning/state improves all models | Missing orchestration was important |
> | Adapter improves only prose quality | Adapter is useful as the writer layer |
> | Adapter worsens explanation or repetition | Training-run problem remains |
> | Another model reaches the goal without fine-tuning | Model selection or operation may matter more |
> | All models fail at the same workflow stage | Framework contract may be the bottleneck |

> **Checkpoint choices if retraining is still appropriate**
>
> For pure continuation, compare against a Korean-capable **base** checkpoint rather than immediately switching to another RP or instruction model.
> 
> Examples of comparison baselines—not universal recommendations—include:
> 
> - [`kakaocorp/kanana-1.5-8b-base`](https://huggingface.co/kakaocorp/kanana-1.5-8b-base)
> - [`Qwen/Qwen3-8B-Base`](https://huggingface.co/Qwen/Qwen3-8B-Base)
> 
> The useful comparison is:
> 
> ```plaintext
> plain base checkpoint
> versus
> instruction/DPO-aligned checkpoint
> 
> ```
> 
> under the same raw-prose objective.
> 
> For instruction-to-story behavior, an instruction checkpoint may be appropriate, but the data should explicitly demonstrate the desired interface:
> 
> ```plaintext
> premise / constraints
> →
> story text without explanatory preamble
> 
> ```
> 
> For RP or collaborative writing, a chat/RP checkpoint and runtime lore/character state may be more relevant than raw continuation training.
> 
> A community creative or RP model can also be useful as a behavioral baseline, but its model card should be checked for:
> 
> - base model;
> - prompt format;
> - training-data disclosure;
> - language coverage;
> - licensing;
> - independent evaluation;
> - intended distinction between RP dialogue and novel prose.

> **What I would not conclude from the current evidence**
>
> I would not yet conclude that:
> 
> - Bllossom is definitely the sole cause;
> - raw-text fine-tuning is inherently wrong;
> - QLoRA caused the explanatory tone;
> - fixed-length packing destroyed the story structure;
> - more epochs will solve the problem;
> - switching to an RP model will automatically create a long-form writer;
> - a longer context window automatically creates a coherent novel;
> - the current samples prove improvement or degradation;
> - the present adapter is useless.
> 
> The adapter may turn out to be:
> 
> - a failed training run;
> - a functioning prose adapter evaluated incorrectly;
> - a useful adapter attached to a mismatched checkpoint;
> - a scene writer that was expected to be the whole writing system;
> - unnecessary if a stronger model and an existing workflow already meet the actual goal.
> 
> Those outcomes lead to different decisions.

> **Information that would make later diagnosis more concrete**
>
> Any subset of the following would help future readers distinguish checkpoint behavior, training behavior and evaluation noise:
> 
> ```plaintext
> exact model repository and revision
> tokenizer repository and revision
> Transformers / PEFT / bitsandbytes / Torch versions
> GPU and precision configuration
> train_report.txt
> training and evaluation loss history
> trainer.state.global_step
> actual total tokens seen
> trainable parameter count
> EOS / PAD / EOT IDs
> one collator-label audit
> adapter-off and adapter-on outputs for the same unseen prompt
> generation seeds and configuration
> number of JSONL rows
> total tokens
> what one row represents: paragraph, scene, chapter or full work
> number of distinct works and authors
> whether evaluation prompts appeared in training
> 
> ```
> 
> These are not all prerequisites for continuing. Each one simply rules out a different class of explanation.

## Bottom line

The concern about Bllossom is justified: it is already instruction-tuned and DPO-aligned, so it was not the cleanest starting point for a pure prose-continuation experiment.

But changing the checkpoint alone would not answer the larger question.

The current script can plausibly train a **prose-continuation/style adapter**. If the final goal is coherent long-form writing, that adapter is only one possible component. Long-output behavior, instruction following, planning, persistent story state, continuity checking and revision are separate layers.

The central decision is therefore not simply:

```plaintext
keep this LoRA
or
discard it

```

It is:

```plaintext
Is the artifact being trained intended to be:

1. the whole writing system,
2. an instruction-to-story model,
or
3. a prose-generation component inside a larger workflow?

```

Once that is fixed, the route becomes clearer:

```plaintext
Short prose continuation
→ base checkpoint + raw-prose LoRA

Instruction-to-scene or chapter
→ instruct checkpoint + prompt/completion SFT

Very long output
→ long-output examples and/or decomposition

Coherent multi-chapter writing
→ planner + persistent story state + writer + checker/revision
  with optional prose fine-tuning

Interactive RP or co-writing
→ chat/RP behavior + lore/character/session memory

```

Before committing to another full training run, the adapter-off input-format comparison, actual optimizer-step count, EOS-label audit and paired held-out generation should reveal whether the present result is mainly a checkpoint mismatch, a training accident, an evaluation problem, a scope mismatch—or some combination of them.