feat(grpo): optional entropy regularization for chunked GRPO loss#6
Open
dongseokmotif wants to merge 1 commit into
Open
feat(grpo): optional entropy regularization for chunked GRPO loss#6dongseokmotif wants to merge 1 commit into
dongseokmotif wants to merge 1 commit into
Conversation
dongseokmotif
force-pushed
the
feat/grpo-entropy-bonus
branch
from
June 10, 2026 02:27
8c8a706 to
9982957
Compare
wanyaworld
approved these changes
Jun 10, 2026
Member
|
이거 커밋이 뭔가섞인것 같네요? |
Add an `entropy_coef` parameter to the chunked GRPO loss that subtracts `entropy_coef * H` from the per-token loss (entropy bonus), where H = -sum_v p_v * log p_v is computed from the chunk's already-materialized log_probs — no extra logit materialization, no extra memory. Counteracts entropy collapse in long RL runs. - Default entropy_coef=0.0 => exact no-op (backward compatible). - Threaded through LigerFusedLinearPPOBase.forward, the partial / _compute_chunk_loss, GRPOFunction.forward / backward (matching None grad), and LigerFusedLinearGRPOLoss. - Entropy computed differentiably from log_probs inside ppo_loss_fn; gradient flows via the existing chunked-recompute backward. Verified: coef=0 reproduces baseline exactly; nonzero shifts loss by -coef*H (H ~1 nat on random init), monotonic in coef, gradient changes; autograd grad-count balanced (30 forward args = 6 grads + 24 Nones).
dongseokmotif
force-pushed
the
feat/grpo-entropy-bonus
branch
from
June 14, 2026 05:25
9982957 to
3a8fd70
Compare
dongseokmotif
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
main기준으로 entropy 정규화만 담도록 재작성한 PR입니다 (기존에 섞여 있던 clamp 커밋 제거).Adds an optional
entropy_coefto the chunked GRPO loss that subtractsentropy_coef * Hfrom the per-token loss (PPO entropy bonus), whereH = -Σ_v p_v log p_vis computed from the chunk's already-materializedlog_probs— no extra logit pass, no extra memory. Counteracts entropy collapse in long RL runs.Details
entropy_coef=0.0→ exact no-op (backward compatible).LigerFusedLinearPPOBase.forward/_compute_chunk_loss,LigerFusedLinearGRPOFunction.forward/backward(matchingNonegrad), andLigerFusedLinearGRPOLoss.log_probs; gradient flows via the existing chunked-recompute backward.