[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-huggingface-trl-training":3,"mdc-qlwpwd-key":38,"related-repo-huggingface-trl-training":2854,"related-org-huggingface-trl-training":2864},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"trl-training","train and fine-tune LLMs with TRL","Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"huggingface","Hugging Face","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fhuggingface.png",[12,15,18,21,24],{"name":9,"slug":13,"type":14},"hugging-face","tag",{"name":16,"slug":17,"type":14},"LLM","llm",{"name":19,"slug":20,"type":14},"Deep Learning","deep-learning",{"name":22,"slug":23,"type":14},"Python","python",{"name":25,"slug":26,"type":14},"AI Infrastructure","ai-infrastructure",18850,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl","2026-04-06T18:25:32.746828","Apache-2.0",2844,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Train transformer language models with reinforcement learning.","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002FHEAD\u002Ftrl\u002Fskills\u002Ftrl-training","---\nname: trl-training\ndescription: Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.\nlicense: Apache-2.0\nmetadata:\n version: \"1.0.0\"\n author: huggingface\n commands:\n - trl sft\n - trl dpo\n - trl grpo\n - trl kto\n - trl rloo\n - trl reward\n categories:\n - machine-learning\n - llm-training\n - reinforcement-learning\n tags:\n - rlhf\n - supervised-fine-tuning\n - dpo\n - grpo\n - huggingface\n - transformers\n documentation: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fen\u002Fclis\n---\n\n# TRL Training Skill\n\nYou are an expert at using the TRL (Transformers Reinforcement Learning) library to train and fine-tune large language models.\n\n## Overview\n\nTRL provides CLI commands for post-training foundation models using state-of-the-art techniques:\n\n- **SFT** (Supervised Fine-Tuning): Fine-tune models on instruction-following or conversational datasets\n- **DPO** (Direct Preference Optimization): Align models using preference data\n- **GRPO** (Group Relative Policy Optimization): Train models by ranking multiple sampled outputs relative to each other and optimizing based on their comparative rewards.\n- **RLOO** (Reinforce Leave One Out): Online RL training with generation-based rewards\n- **Reward Model Training**: Train reward models for RLHF\n\nTRL is built on top of Hugging Face Transformers and Accelerate, providing seamless integration with the Hugging Face ecosystem.\n\n## Core Commands\n\n### trl sft - Supervised Fine-Tuning\n\nFine-tune language models on instruction-following or conversational datasets.\n\n**Full training:**\n\n```bash\ntrl sft \\\n --model_name_or_path Qwen\u002FQwen2-0.5B \\\n --dataset_name trl-lib\u002FCapybara \\\n --learning_rate 2.0e-5 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '\u003C|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n```\n\n**Train with LoRA adapters:**\n\n```bash\ntrl sft \\\n --model_name_or_path Qwen\u002FQwen2-0.5B \\\n --dataset_name trl-lib\u002FCapybara \\\n --learning_rate 2.0e-4 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '\u003C|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n```\n\n### trl dpo - Direct Preference Optimization\n\nAlign models using preference data (chosen\u002Frejected pairs).\n\n**Full training:**\n\n```bash\ntrl dpo \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --learning_rate 5.0e-7 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns\n```\n\n**Train with LoRA adapters:**\n\n```bash\ntrl dpo \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --learning_rate 5.0e-6 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16\n```\n\n### trl grpo - Group Relative Policy Optimization\n\nTrain models using reward functions or LLM-as-a-judge for evaluating generations and providing rewards.\n\n**Basic usage:**\n\n```bash\ntrl grpo \\\n --model_name_or_path Qwen\u002FQwen2.5-0.5B \\\n --dataset_name trl-lib\u002Fgsm8k \\\n --reward_funcs accuracy_reward \\\n --output_dir Qwen2-0.5B-GRPO \\\n --push_to_hub\n```\n\n### trl rloo - Reinforce Leave One Out\n\nOnline RL training where the model generates text and receives rewards based on custom criteria.\n\n**Basic usage:**\n\n```bash\ntrl rloo \\\n --model_name_or_path Qwen\u002FQwen2.5-0.5B \\\n --dataset_name trl-lib\u002Ftldr \\\n --reward_model_name_or_path sentiment-analysis:nlptown\u002Fbert-base-multilingual-uncased-sentiment \\\n --output_dir Qwen2-0.5B-RLOO \\\n --push_to_hub\n```\n\n### trl reward - Reward Model Training\n\nTrain a reward model to score text quality for RLHF.\n\n**Full training:**\n\n```bash\ntrl reward \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-5 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048\n```\n\n**Train with LoRA adapters:**\n\n```bash\ntrl reward \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward-LoRA \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-4 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048 \\\n --use_peft \\\n --lora_task_type SEQ_CLS \\\n --lora_r 32 \\\n --lora_alpha 16\n```\n\n## Configuration Files\n\nTRL supports YAML configuration files for reproducible training. All CLI arguments can be specified in a config file.\n\n**Example config (sft_config.yaml):**\n\n```yaml\nmodel_name_or_path: Qwen\u002FQwen2.5-0.5B\ndataset_name: trl-lib\u002FCapybara\nlearning_rate: 2.0e-5\nnum_train_epochs: 1\nper_device_train_batch_size: 8\ngradient_accumulation_steps: 2\noutput_dir: .\u002Fsft_output\nuse_peft: true\nlora_r: 16\nlora_alpha: 16\nreport_to: trackio\n```\n\n**Launch with config:**\n\n```bash\ntrl sft --config sft_config.yaml\n```\n\n**Override config values:**\n\n```bash\ntrl sft --config sft_config.yaml --learning_rate 1.0e-5\n```\n\n## Distributed Training\n\nTRL integrates with Accelerate for multi-GPU and multi-node training.\n\n**Multi-GPU training:**\n\n```bash\ntrl sft \\\n --config sft_config.yaml \\\n --num_processes 4\n```\n\n**Use predefined Accelerate configs:**\n\nTRL provides predefined configs: `single_gpu`, `multi_gpu`, `fsdp1`, `fsdp2`, `zero1`, `zero2`, `zero3`\n\n```bash\ntrl sft \\\n --config sft_config.yaml \\\n --accelerate_config zero2\n```\n\n**Custom Accelerate config:**\n\n```bash\n# Generate custom config\naccelerate config\n\n# Use custom config\ntrl sft --config sft_config.yaml --config_file ~\u002F.cache\u002Fhuggingface\u002Faccelerate\u002Fdefault_config.yaml\n```\n\n**Fully Sharded Data Parallel (FSDP):**\n\n```bash\ntrl sft --config sft_config.yaml --accelerate_config fsdp2\n```\n\n**DeepSpeed ZeRO:**\n\n```bash\ntrl sft --config sft_config.yaml --accelerate_config zero3\n```\n\n## Troubleshooting\n\n### CUDA Out of Memory\n\n- Reduce `--per_device_train_batch_size` and increase `--gradient_accumulation_steps`\n- Enable `--use_peft` for LoRA training\n- Use `--gradient_checkpointing` to save memory\n- Try smaller model or longer sequence truncation\n\n### Dataset Loading Issues\n\n- Verify dataset exists: check Hugging Face Hub or local path\n- Check dataset format matches expected columns\n- Use `--dataset_config` for multi-config datasets\n- Inspect dataset: `from datasets import load_dataset; ds = load_dataset(name)`\n\n### Model Loading Issues\n\n- Verify model exists on Hugging Face Hub\n- Check if gated model requires authentication: `hf auth login`\n- For local models, provide absolute path\n- Ensure sufficient disk space and memory\n\n### Slow Training\n\n- Enable dataset `--packing` for short sequences\n- Use larger `--per_device_train_batch_size` if memory allows\n- Enable `--tf32` for faster computation on Ampere GPUs\n- Use `--bf16` on supported hardware\n- Consider multi-GPU training with `--num_processes`\n\n### Generation Issues (GRPO\u002FRLOO)\n\n- Check prompt format in dataset\n- Adjust `--temperature` and `--top_p` for generation\n- Verify the reward function (for GRPO\u002FRLOO)\n\n## Additional Resources\n\n- **Documentation**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- **Examples**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\n\n## Best Practices\n\n1. **Start with SFT**: Always fine-tune base models with SFT before preference alignment\n2. **Use LoRA for efficiency**: Enable `--use_peft` for faster training and lower memory\n3. **Monitor training**: Use `--report_to trackio` (or `--report_to wandb` or `--report_to tensorboard`) for tracking\n4. **Save checkpoints**: TRL automatically saves checkpoints in `--output_dir`\n5. **Test on small datasets first**: Verify pipeline works before full training\n6. **Use configuration files**: Create YAML configs for reproducibility\n7. **Leverage Accelerate**: Use multi-GPU training for faster iteration\n\nWhen helping users with TRL:\n- Always check which training method is appropriate for their use case\n- Verify dataset format matches the expected schema\n- Recommend starting with smaller models for testing\n- Suggest LoRA for resource-constrained environments\n- Point to specific documentation sections for advanced features\n",{"data":39,"body":60},{"name":4,"description":6,"license":30,"metadata":40},{"version":41,"author":8,"commands":42,"categories":49,"tags":53,"documentation":59},"1.0.0",[43,44,45,46,47,48],"trl sft","trl dpo","trl grpo","trl kto","trl rloo","trl reward",[50,51,52],"machine-learning","llm-training","reinforcement-learning",[54,55,56,57,8,58],"rlhf","supervised-fine-tuning","dpo","grpo","transformers","https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fen\u002Fclis",{"type":61,"children":62},"root",[63,72,78,85,90,146,151,157,164,169,177,427,435,683,689,694,701,889,896,1119,1125,1130,1138,1233,1239,1244,1251,1345,1351,1356,1363,1521,1528,1742,1748,1753,1761,1958,1966,1994,2002,2039,2045,2050,2058,2109,2117,2171,2221,2229,2304,2312,2348,2356,2391,2397,2403,2456,2462,2498,2504,2533,2539,2602,2608,2642,2648,2698,2704,2815,2820,2848],{"type":64,"tag":65,"props":66,"children":68},"element","h1",{"id":67},"trl-training-skill",[69],{"type":70,"value":71},"text","TRL Training Skill",{"type":64,"tag":73,"props":74,"children":75},"p",{},[76],{"type":70,"value":77},"You are an expert at using the TRL (Transformers Reinforcement Learning) library to train and fine-tune large language models.",{"type":64,"tag":79,"props":80,"children":82},"h2",{"id":81},"overview",[83],{"type":70,"value":84},"Overview",{"type":64,"tag":73,"props":86,"children":87},{},[88],{"type":70,"value":89},"TRL provides CLI commands for post-training foundation models using state-of-the-art techniques:",{"type":64,"tag":91,"props":92,"children":93},"ul",{},[94,106,116,126,136],{"type":64,"tag":95,"props":96,"children":97},"li",{},[98,104],{"type":64,"tag":99,"props":100,"children":101},"strong",{},[102],{"type":70,"value":103},"SFT",{"type":70,"value":105}," (Supervised Fine-Tuning): Fine-tune models on instruction-following or conversational datasets",{"type":64,"tag":95,"props":107,"children":108},{},[109,114],{"type":64,"tag":99,"props":110,"children":111},{},[112],{"type":70,"value":113},"DPO",{"type":70,"value":115}," (Direct Preference Optimization): Align models using preference data",{"type":64,"tag":95,"props":117,"children":118},{},[119,124],{"type":64,"tag":99,"props":120,"children":121},{},[122],{"type":70,"value":123},"GRPO",{"type":70,"value":125}," (Group Relative Policy Optimization): Train models by ranking multiple sampled outputs relative to each other and optimizing based on their comparative rewards.",{"type":64,"tag":95,"props":127,"children":128},{},[129,134],{"type":64,"tag":99,"props":130,"children":131},{},[132],{"type":70,"value":133},"RLOO",{"type":70,"value":135}," (Reinforce Leave One Out): Online RL training with generation-based rewards",{"type":64,"tag":95,"props":137,"children":138},{},[139,144],{"type":64,"tag":99,"props":140,"children":141},{},[142],{"type":70,"value":143},"Reward Model Training",{"type":70,"value":145},": Train reward models for RLHF",{"type":64,"tag":73,"props":147,"children":148},{},[149],{"type":70,"value":150},"TRL is built on top of Hugging Face Transformers and Accelerate, providing seamless integration with the Hugging Face ecosystem.",{"type":64,"tag":79,"props":152,"children":154},{"id":153},"core-commands",[155],{"type":70,"value":156},"Core Commands",{"type":64,"tag":158,"props":159,"children":161},"h3",{"id":160},"trl-sft-supervised-fine-tuning",[162],{"type":70,"value":163},"trl sft - Supervised Fine-Tuning",{"type":64,"tag":73,"props":165,"children":166},{},[167],{"type":70,"value":168},"Fine-tune language models on instruction-following or conversational datasets.",{"type":64,"tag":73,"props":170,"children":171},{},[172],{"type":64,"tag":99,"props":173,"children":174},{},[175],{"type":70,"value":176},"Full training:",{"type":64,"tag":178,"props":179,"children":184},"pre",{"className":180,"code":181,"language":182,"meta":183,"style":183},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","trl sft \\\n --model_name_or_path Qwen\u002FQwen2-0.5B \\\n --dataset_name trl-lib\u002FCapybara \\\n --learning_rate 2.0e-5 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '\u003C|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n","bash","",[185],{"type":64,"tag":186,"props":187,"children":188},"code",{"__ignoreMap":183},[189,213,231,249,267,286,299,317,335,364,382,400,418],{"type":64,"tag":190,"props":191,"children":194},"span",{"class":192,"line":193},"line",1,[195,201,207],{"type":64,"tag":190,"props":196,"children":198},{"style":197},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[199],{"type":70,"value":200},"trl",{"type":64,"tag":190,"props":202,"children":204},{"style":203},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[205],{"type":70,"value":206}," sft",{"type":64,"tag":190,"props":208,"children":210},{"style":209},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[211],{"type":70,"value":212}," \\\n",{"type":64,"tag":190,"props":214,"children":216},{"class":192,"line":215},2,[217,222,227],{"type":64,"tag":190,"props":218,"children":219},{"style":203},[220],{"type":70,"value":221}," --model_name_or_path",{"type":64,"tag":190,"props":223,"children":224},{"style":203},[225],{"type":70,"value":226}," Qwen\u002FQwen2-0.5B",{"type":64,"tag":190,"props":228,"children":229},{"style":209},[230],{"type":70,"value":212},{"type":64,"tag":190,"props":232,"children":234},{"class":192,"line":233},3,[235,240,245],{"type":64,"tag":190,"props":236,"children":237},{"style":203},[238],{"type":70,"value":239}," --dataset_name",{"type":64,"tag":190,"props":241,"children":242},{"style":203},[243],{"type":70,"value":244}," trl-lib\u002FCapybara",{"type":64,"tag":190,"props":246,"children":247},{"style":209},[248],{"type":70,"value":212},{"type":64,"tag":190,"props":250,"children":252},{"class":192,"line":251},4,[253,258,263],{"type":64,"tag":190,"props":254,"children":255},{"style":203},[256],{"type":70,"value":257}," --learning_rate",{"type":64,"tag":190,"props":259,"children":260},{"style":203},[261],{"type":70,"value":262}," 2.0e-5",{"type":64,"tag":190,"props":264,"children":265},{"style":209},[266],{"type":70,"value":212},{"type":64,"tag":190,"props":268,"children":270},{"class":192,"line":269},5,[271,276,282],{"type":64,"tag":190,"props":272,"children":273},{"style":203},[274],{"type":70,"value":275}," --num_train_epochs",{"type":64,"tag":190,"props":277,"children":279},{"style":278},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[280],{"type":70,"value":281}," 1",{"type":64,"tag":190,"props":283,"children":284},{"style":209},[285],{"type":70,"value":212},{"type":64,"tag":190,"props":287,"children":289},{"class":192,"line":288},6,[290,295],{"type":64,"tag":190,"props":291,"children":292},{"style":203},[293],{"type":70,"value":294}," --packing",{"type":64,"tag":190,"props":296,"children":297},{"style":209},[298],{"type":70,"value":212},{"type":64,"tag":190,"props":300,"children":302},{"class":192,"line":301},7,[303,308,313],{"type":64,"tag":190,"props":304,"children":305},{"style":203},[306],{"type":70,"value":307}," --per_device_train_batch_size",{"type":64,"tag":190,"props":309,"children":310},{"style":278},[311],{"type":70,"value":312}," 2",{"type":64,"tag":190,"props":314,"children":315},{"style":209},[316],{"type":70,"value":212},{"type":64,"tag":190,"props":318,"children":320},{"class":192,"line":319},8,[321,326,331],{"type":64,"tag":190,"props":322,"children":323},{"style":203},[324],{"type":70,"value":325}," --gradient_accumulation_steps",{"type":64,"tag":190,"props":327,"children":328},{"style":278},[329],{"type":70,"value":330}," 8",{"type":64,"tag":190,"props":332,"children":333},{"style":209},[334],{"type":70,"value":212},{"type":64,"tag":190,"props":336,"children":338},{"class":192,"line":337},9,[339,344,350,355,360],{"type":64,"tag":190,"props":340,"children":341},{"style":203},[342],{"type":70,"value":343}," --eos_token",{"type":64,"tag":190,"props":345,"children":347},{"style":346},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[348],{"type":70,"value":349}," '",{"type":64,"tag":190,"props":351,"children":352},{"style":203},[353],{"type":70,"value":354},"\u003C|im_end|>",{"type":64,"tag":190,"props":356,"children":357},{"style":346},[358],{"type":70,"value":359},"'",{"type":64,"tag":190,"props":361,"children":362},{"style":209},[363],{"type":70,"value":212},{"type":64,"tag":190,"props":365,"children":367},{"class":192,"line":366},10,[368,373,378],{"type":64,"tag":190,"props":369,"children":370},{"style":203},[371],{"type":70,"value":372}," --eval_strategy",{"type":64,"tag":190,"props":374,"children":375},{"style":203},[376],{"type":70,"value":377}," steps",{"type":64,"tag":190,"props":379,"children":380},{"style":209},[381],{"type":70,"value":212},{"type":64,"tag":190,"props":383,"children":385},{"class":192,"line":384},11,[386,391,396],{"type":64,"tag":190,"props":387,"children":388},{"style":203},[389],{"type":70,"value":390}," --eval_steps",{"type":64,"tag":190,"props":392,"children":393},{"style":278},[394],{"type":70,"value":395}," 100",{"type":64,"tag":190,"props":397,"children":398},{"style":209},[399],{"type":70,"value":212},{"type":64,"tag":190,"props":401,"children":403},{"class":192,"line":402},12,[404,409,414],{"type":64,"tag":190,"props":405,"children":406},{"style":203},[407],{"type":70,"value":408}," --output_dir",{"type":64,"tag":190,"props":410,"children":411},{"style":203},[412],{"type":70,"value":413}," Qwen2-0.5B-SFT",{"type":64,"tag":190,"props":415,"children":416},{"style":209},[417],{"type":70,"value":212},{"type":64,"tag":190,"props":419,"children":421},{"class":192,"line":420},13,[422],{"type":64,"tag":190,"props":423,"children":424},{"style":203},[425],{"type":70,"value":426}," --push_to_hub\n",{"type":64,"tag":73,"props":428,"children":429},{},[430],{"type":64,"tag":99,"props":431,"children":432},{},[433],{"type":70,"value":434},"Train with LoRA adapters:",{"type":64,"tag":178,"props":436,"children":438},{"className":180,"code":437,"language":182,"meta":183,"style":183},"trl sft \\\n --model_name_or_path Qwen\u002FQwen2-0.5B \\\n --dataset_name trl-lib\u002FCapybara \\\n --learning_rate 2.0e-4 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '\u003C|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n",[439],{"type":64,"tag":186,"props":440,"children":441},{"__ignoreMap":183},[442,457,472,487,503,518,529,544,559,582,597,612,624,641,659,675],{"type":64,"tag":190,"props":443,"children":444},{"class":192,"line":193},[445,449,453],{"type":64,"tag":190,"props":446,"children":447},{"style":197},[448],{"type":70,"value":200},{"type":64,"tag":190,"props":450,"children":451},{"style":203},[452],{"type":70,"value":206},{"type":64,"tag":190,"props":454,"children":455},{"style":209},[456],{"type":70,"value":212},{"type":64,"tag":190,"props":458,"children":459},{"class":192,"line":215},[460,464,468],{"type":64,"tag":190,"props":461,"children":462},{"style":203},[463],{"type":70,"value":221},{"type":64,"tag":190,"props":465,"children":466},{"style":203},[467],{"type":70,"value":226},{"type":64,"tag":190,"props":469,"children":470},{"style":209},[471],{"type":70,"value":212},{"type":64,"tag":190,"props":473,"children":474},{"class":192,"line":233},[475,479,483],{"type":64,"tag":190,"props":476,"children":477},{"style":203},[478],{"type":70,"value":239},{"type":64,"tag":190,"props":480,"children":481},{"style":203},[482],{"type":70,"value":244},{"type":64,"tag":190,"props":484,"children":485},{"style":209},[486],{"type":70,"value":212},{"type":64,"tag":190,"props":488,"children":489},{"class":192,"line":251},[490,494,499],{"type":64,"tag":190,"props":491,"children":492},{"style":203},[493],{"type":70,"value":257},{"type":64,"tag":190,"props":495,"children":496},{"style":203},[497],{"type":70,"value":498}," 2.0e-4",{"type":64,"tag":190,"props":500,"children":501},{"style":209},[502],{"type":70,"value":212},{"type":64,"tag":190,"props":504,"children":505},{"class":192,"line":269},[506,510,514],{"type":64,"tag":190,"props":507,"children":508},{"style":203},[509],{"type":70,"value":275},{"type":64,"tag":190,"props":511,"children":512},{"style":278},[513],{"type":70,"value":281},{"type":64,"tag":190,"props":515,"children":516},{"style":209},[517],{"type":70,"value":212},{"type":64,"tag":190,"props":519,"children":520},{"class":192,"line":288},[521,525],{"type":64,"tag":190,"props":522,"children":523},{"style":203},[524],{"type":70,"value":294},{"type":64,"tag":190,"props":526,"children":527},{"style":209},[528],{"type":70,"value":212},{"type":64,"tag":190,"props":530,"children":531},{"class":192,"line":301},[532,536,540],{"type":64,"tag":190,"props":533,"children":534},{"style":203},[535],{"type":70,"value":307},{"type":64,"tag":190,"props":537,"children":538},{"style":278},[539],{"type":70,"value":312},{"type":64,"tag":190,"props":541,"children":542},{"style":209},[543],{"type":70,"value":212},{"type":64,"tag":190,"props":545,"children":546},{"class":192,"line":319},[547,551,555],{"type":64,"tag":190,"props":548,"children":549},{"style":203},[550],{"type":70,"value":325},{"type":64,"tag":190,"props":552,"children":553},{"style":278},[554],{"type":70,"value":330},{"type":64,"tag":190,"props":556,"children":557},{"style":209},[558],{"type":70,"value":212},{"type":64,"tag":190,"props":560,"children":561},{"class":192,"line":337},[562,566,570,574,578],{"type":64,"tag":190,"props":563,"children":564},{"style":203},[565],{"type":70,"value":343},{"type":64,"tag":190,"props":567,"children":568},{"style":346},[569],{"type":70,"value":349},{"type":64,"tag":190,"props":571,"children":572},{"style":203},[573],{"type":70,"value":354},{"type":64,"tag":190,"props":575,"children":576},{"style":346},[577],{"type":70,"value":359},{"type":64,"tag":190,"props":579,"children":580},{"style":209},[581],{"type":70,"value":212},{"type":64,"tag":190,"props":583,"children":584},{"class":192,"line":366},[585,589,593],{"type":64,"tag":190,"props":586,"children":587},{"style":203},[588],{"type":70,"value":372},{"type":64,"tag":190,"props":590,"children":591},{"style":203},[592],{"type":70,"value":377},{"type":64,"tag":190,"props":594,"children":595},{"style":209},[596],{"type":70,"value":212},{"type":64,"tag":190,"props":598,"children":599},{"class":192,"line":384},[600,604,608],{"type":64,"tag":190,"props":601,"children":602},{"style":203},[603],{"type":70,"value":390},{"type":64,"tag":190,"props":605,"children":606},{"style":278},[607],{"type":70,"value":395},{"type":64,"tag":190,"props":609,"children":610},{"style":209},[611],{"type":70,"value":212},{"type":64,"tag":190,"props":613,"children":614},{"class":192,"line":402},[615,620],{"type":64,"tag":190,"props":616,"children":617},{"style":203},[618],{"type":70,"value":619}," --use_peft",{"type":64,"tag":190,"props":621,"children":622},{"style":209},[623],{"type":70,"value":212},{"type":64,"tag":190,"props":625,"children":626},{"class":192,"line":420},[627,632,637],{"type":64,"tag":190,"props":628,"children":629},{"style":203},[630],{"type":70,"value":631}," --lora_r",{"type":64,"tag":190,"props":633,"children":634},{"style":278},[635],{"type":70,"value":636}," 32",{"type":64,"tag":190,"props":638,"children":639},{"style":209},[640],{"type":70,"value":212},{"type":64,"tag":190,"props":642,"children":644},{"class":192,"line":643},14,[645,650,655],{"type":64,"tag":190,"props":646,"children":647},{"style":203},[648],{"type":70,"value":649}," --lora_alpha",{"type":64,"tag":190,"props":651,"children":652},{"style":278},[653],{"type":70,"value":654}," 16",{"type":64,"tag":190,"props":656,"children":657},{"style":209},[658],{"type":70,"value":212},{"type":64,"tag":190,"props":660,"children":662},{"class":192,"line":661},15,[663,667,671],{"type":64,"tag":190,"props":664,"children":665},{"style":203},[666],{"type":70,"value":408},{"type":64,"tag":190,"props":668,"children":669},{"style":203},[670],{"type":70,"value":413},{"type":64,"tag":190,"props":672,"children":673},{"style":209},[674],{"type":70,"value":212},{"type":64,"tag":190,"props":676,"children":678},{"class":192,"line":677},16,[679],{"type":64,"tag":190,"props":680,"children":681},{"style":203},[682],{"type":70,"value":426},{"type":64,"tag":158,"props":684,"children":686},{"id":685},"trl-dpo-direct-preference-optimization",[687],{"type":70,"value":688},"trl dpo - Direct Preference Optimization",{"type":64,"tag":73,"props":690,"children":691},{},[692],{"type":70,"value":693},"Align models using preference data (chosen\u002Frejected pairs).",{"type":64,"tag":73,"props":695,"children":696},{},[697],{"type":64,"tag":99,"props":698,"children":699},{},[700],{"type":70,"value":176},{"type":64,"tag":178,"props":702,"children":704},{"className":180,"code":703,"language":182,"meta":183,"style":183},"trl dpo \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --learning_rate 5.0e-7 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns\n",[705],{"type":64,"tag":186,"props":706,"children":707},{"__ignoreMap":183},[708,724,740,756,772,787,802,819,834,849,865,881],{"type":64,"tag":190,"props":709,"children":710},{"class":192,"line":193},[711,715,720],{"type":64,"tag":190,"props":712,"children":713},{"style":197},[714],{"type":70,"value":200},{"type":64,"tag":190,"props":716,"children":717},{"style":203},[718],{"type":70,"value":719}," dpo",{"type":64,"tag":190,"props":721,"children":722},{"style":209},[723],{"type":70,"value":212},{"type":64,"tag":190,"props":725,"children":726},{"class":192,"line":215},[727,731,736],{"type":64,"tag":190,"props":728,"children":729},{"style":203},[730],{"type":70,"value":239},{"type":64,"tag":190,"props":732,"children":733},{"style":203},[734],{"type":70,"value":735}," trl-lib\u002Fultrafeedback_binarized",{"type":64,"tag":190,"props":737,"children":738},{"style":209},[739],{"type":70,"value":212},{"type":64,"tag":190,"props":741,"children":742},{"class":192,"line":233},[743,747,752],{"type":64,"tag":190,"props":744,"children":745},{"style":203},[746],{"type":70,"value":221},{"type":64,"tag":190,"props":748,"children":749},{"style":203},[750],{"type":70,"value":751}," Qwen\u002FQwen2-0.5B-Instruct",{"type":64,"tag":190,"props":753,"children":754},{"style":209},[755],{"type":70,"value":212},{"type":64,"tag":190,"props":757,"children":758},{"class":192,"line":251},[759,763,768],{"type":64,"tag":190,"props":760,"children":761},{"style":203},[762],{"type":70,"value":257},{"type":64,"tag":190,"props":764,"children":765},{"style":203},[766],{"type":70,"value":767}," 5.0e-7",{"type":64,"tag":190,"props":769,"children":770},{"style":209},[771],{"type":70,"value":212},{"type":64,"tag":190,"props":773,"children":774},{"class":192,"line":269},[775,779,783],{"type":64,"tag":190,"props":776,"children":777},{"style":203},[778],{"type":70,"value":275},{"type":64,"tag":190,"props":780,"children":781},{"style":278},[782],{"type":70,"value":281},{"type":64,"tag":190,"props":784,"children":785},{"style":209},[786],{"type":70,"value":212},{"type":64,"tag":190,"props":788,"children":789},{"class":192,"line":288},[790,794,798],{"type":64,"tag":190,"props":791,"children":792},{"style":203},[793],{"type":70,"value":307},{"type":64,"tag":190,"props":795,"children":796},{"style":278},[797],{"type":70,"value":312},{"type":64,"tag":190,"props":799,"children":800},{"style":209},[801],{"type":70,"value":212},{"type":64,"tag":190,"props":803,"children":804},{"class":192,"line":301},[805,810,815],{"type":64,"tag":190,"props":806,"children":807},{"style":203},[808],{"type":70,"value":809}," --max_steps",{"type":64,"tag":190,"props":811,"children":812},{"style":278},[813],{"type":70,"value":814}," 1000",{"type":64,"tag":190,"props":816,"children":817},{"style":209},[818],{"type":70,"value":212},{"type":64,"tag":190,"props":820,"children":821},{"class":192,"line":319},[822,826,830],{"type":64,"tag":190,"props":823,"children":824},{"style":203},[825],{"type":70,"value":325},{"type":64,"tag":190,"props":827,"children":828},{"style":278},[829],{"type":70,"value":330},{"type":64,"tag":190,"props":831,"children":832},{"style":209},[833],{"type":70,"value":212},{"type":64,"tag":190,"props":835,"children":836},{"class":192,"line":337},[837,841,845],{"type":64,"tag":190,"props":838,"children":839},{"style":203},[840],{"type":70,"value":372},{"type":64,"tag":190,"props":842,"children":843},{"style":203},[844],{"type":70,"value":377},{"type":64,"tag":190,"props":846,"children":847},{"style":209},[848],{"type":70,"value":212},{"type":64,"tag":190,"props":850,"children":851},{"class":192,"line":366},[852,856,861],{"type":64,"tag":190,"props":853,"children":854},{"style":203},[855],{"type":70,"value":390},{"type":64,"tag":190,"props":857,"children":858},{"style":278},[859],{"type":70,"value":860}," 50",{"type":64,"tag":190,"props":862,"children":863},{"style":209},[864],{"type":70,"value":212},{"type":64,"tag":190,"props":866,"children":867},{"class":192,"line":384},[868,872,877],{"type":64,"tag":190,"props":869,"children":870},{"style":203},[871],{"type":70,"value":408},{"type":64,"tag":190,"props":873,"children":874},{"style":203},[875],{"type":70,"value":876}," Qwen2-0.5B-DPO",{"type":64,"tag":190,"props":878,"children":879},{"style":209},[880],{"type":70,"value":212},{"type":64,"tag":190,"props":882,"children":883},{"class":192,"line":402},[884],{"type":64,"tag":190,"props":885,"children":886},{"style":203},[887],{"type":70,"value":888}," --no_remove_unused_columns\n",{"type":64,"tag":73,"props":890,"children":891},{},[892],{"type":64,"tag":99,"props":893,"children":894},{},[895],{"type":70,"value":434},{"type":64,"tag":178,"props":897,"children":899},{"className":180,"code":898,"language":182,"meta":183,"style":183},"trl dpo \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --learning_rate 5.0e-6 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16\n",[900],{"type":64,"tag":186,"props":901,"children":902},{"__ignoreMap":183},[903,918,933,948,964,979,994,1009,1024,1039,1054,1069,1081,1092,1107],{"type":64,"tag":190,"props":904,"children":905},{"class":192,"line":193},[906,910,914],{"type":64,"tag":190,"props":907,"children":908},{"style":197},[909],{"type":70,"value":200},{"type":64,"tag":190,"props":911,"children":912},{"style":203},[913],{"type":70,"value":719},{"type":64,"tag":190,"props":915,"children":916},{"style":209},[917],{"type":70,"value":212},{"type":64,"tag":190,"props":919,"children":920},{"class":192,"line":215},[921,925,929],{"type":64,"tag":190,"props":922,"children":923},{"style":203},[924],{"type":70,"value":239},{"type":64,"tag":190,"props":926,"children":927},{"style":203},[928],{"type":70,"value":735},{"type":64,"tag":190,"props":930,"children":931},{"style":209},[932],{"type":70,"value":212},{"type":64,"tag":190,"props":934,"children":935},{"class":192,"line":233},[936,940,944],{"type":64,"tag":190,"props":937,"children":938},{"style":203},[939],{"type":70,"value":221},{"type":64,"tag":190,"props":941,"children":942},{"style":203},[943],{"type":70,"value":751},{"type":64,"tag":190,"props":945,"children":946},{"style":209},[947],{"type":70,"value":212},{"type":64,"tag":190,"props":949,"children":950},{"class":192,"line":251},[951,955,960],{"type":64,"tag":190,"props":952,"children":953},{"style":203},[954],{"type":70,"value":257},{"type":64,"tag":190,"props":956,"children":957},{"style":203},[958],{"type":70,"value":959}," 5.0e-6",{"type":64,"tag":190,"props":961,"children":962},{"style":209},[963],{"type":70,"value":212},{"type":64,"tag":190,"props":965,"children":966},{"class":192,"line":269},[967,971,975],{"type":64,"tag":190,"props":968,"children":969},{"style":203},[970],{"type":70,"value":275},{"type":64,"tag":190,"props":972,"children":973},{"style":278},[974],{"type":70,"value":281},{"type":64,"tag":190,"props":976,"children":977},{"style":209},[978],{"type":70,"value":212},{"type":64,"tag":190,"props":980,"children":981},{"class":192,"line":288},[982,986,990],{"type":64,"tag":190,"props":983,"children":984},{"style":203},[985],{"type":70,"value":307},{"type":64,"tag":190,"props":987,"children":988},{"style":278},[989],{"type":70,"value":312},{"type":64,"tag":190,"props":991,"children":992},{"style":209},[993],{"type":70,"value":212},{"type":64,"tag":190,"props":995,"children":996},{"class":192,"line":301},[997,1001,1005],{"type":64,"tag":190,"props":998,"children":999},{"style":203},[1000],{"type":70,"value":809},{"type":64,"tag":190,"props":1002,"children":1003},{"style":278},[1004],{"type":70,"value":814},{"type":64,"tag":190,"props":1006,"children":1007},{"style":209},[1008],{"type":70,"value":212},{"type":64,"tag":190,"props":1010,"children":1011},{"class":192,"line":319},[1012,1016,1020],{"type":64,"tag":190,"props":1013,"children":1014},{"style":203},[1015],{"type":70,"value":325},{"type":64,"tag":190,"props":1017,"children":1018},{"style":278},[1019],{"type":70,"value":330},{"type":64,"tag":190,"props":1021,"children":1022},{"style":209},[1023],{"type":70,"value":212},{"type":64,"tag":190,"props":1025,"children":1026},{"class":192,"line":337},[1027,1031,1035],{"type":64,"tag":190,"props":1028,"children":1029},{"style":203},[1030],{"type":70,"value":372},{"type":64,"tag":190,"props":1032,"children":1033},{"style":203},[1034],{"type":70,"value":377},{"type":64,"tag":190,"props":1036,"children":1037},{"style":209},[1038],{"type":70,"value":212},{"type":64,"tag":190,"props":1040,"children":1041},{"class":192,"line":366},[1042,1046,1050],{"type":64,"tag":190,"props":1043,"children":1044},{"style":203},[1045],{"type":70,"value":390},{"type":64,"tag":190,"props":1047,"children":1048},{"style":278},[1049],{"type":70,"value":860},{"type":64,"tag":190,"props":1051,"children":1052},{"style":209},[1053],{"type":70,"value":212},{"type":64,"tag":190,"props":1055,"children":1056},{"class":192,"line":384},[1057,1061,1065],{"type":64,"tag":190,"props":1058,"children":1059},{"style":203},[1060],{"type":70,"value":408},{"type":64,"tag":190,"props":1062,"children":1063},{"style":203},[1064],{"type":70,"value":876},{"type":64,"tag":190,"props":1066,"children":1067},{"style":209},[1068],{"type":70,"value":212},{"type":64,"tag":190,"props":1070,"children":1071},{"class":192,"line":402},[1072,1077],{"type":64,"tag":190,"props":1073,"children":1074},{"style":203},[1075],{"type":70,"value":1076}," --no_remove_unused_columns",{"type":64,"tag":190,"props":1078,"children":1079},{"style":209},[1080],{"type":70,"value":212},{"type":64,"tag":190,"props":1082,"children":1083},{"class":192,"line":420},[1084,1088],{"type":64,"tag":190,"props":1085,"children":1086},{"style":203},[1087],{"type":70,"value":619},{"type":64,"tag":190,"props":1089,"children":1090},{"style":209},[1091],{"type":70,"value":212},{"type":64,"tag":190,"props":1093,"children":1094},{"class":192,"line":643},[1095,1099,1103],{"type":64,"tag":190,"props":1096,"children":1097},{"style":203},[1098],{"type":70,"value":631},{"type":64,"tag":190,"props":1100,"children":1101},{"style":278},[1102],{"type":70,"value":636},{"type":64,"tag":190,"props":1104,"children":1105},{"style":209},[1106],{"type":70,"value":212},{"type":64,"tag":190,"props":1108,"children":1109},{"class":192,"line":661},[1110,1114],{"type":64,"tag":190,"props":1111,"children":1112},{"style":203},[1113],{"type":70,"value":649},{"type":64,"tag":190,"props":1115,"children":1116},{"style":278},[1117],{"type":70,"value":1118}," 16\n",{"type":64,"tag":158,"props":1120,"children":1122},{"id":1121},"trl-grpo-group-relative-policy-optimization",[1123],{"type":70,"value":1124},"trl grpo - Group Relative Policy Optimization",{"type":64,"tag":73,"props":1126,"children":1127},{},[1128],{"type":70,"value":1129},"Train models using reward functions or LLM-as-a-judge for evaluating generations and providing rewards.",{"type":64,"tag":73,"props":1131,"children":1132},{},[1133],{"type":64,"tag":99,"props":1134,"children":1135},{},[1136],{"type":70,"value":1137},"Basic usage:",{"type":64,"tag":178,"props":1139,"children":1141},{"className":180,"code":1140,"language":182,"meta":183,"style":183},"trl grpo \\\n --model_name_or_path Qwen\u002FQwen2.5-0.5B \\\n --dataset_name trl-lib\u002Fgsm8k \\\n --reward_funcs accuracy_reward \\\n --output_dir Qwen2-0.5B-GRPO \\\n --push_to_hub\n",[1142],{"type":64,"tag":186,"props":1143,"children":1144},{"__ignoreMap":183},[1145,1161,1177,1193,1210,1226],{"type":64,"tag":190,"props":1146,"children":1147},{"class":192,"line":193},[1148,1152,1157],{"type":64,"tag":190,"props":1149,"children":1150},{"style":197},[1151],{"type":70,"value":200},{"type":64,"tag":190,"props":1153,"children":1154},{"style":203},[1155],{"type":70,"value":1156}," grpo",{"type":64,"tag":190,"props":1158,"children":1159},{"style":209},[1160],{"type":70,"value":212},{"type":64,"tag":190,"props":1162,"children":1163},{"class":192,"line":215},[1164,1168,1173],{"type":64,"tag":190,"props":1165,"children":1166},{"style":203},[1167],{"type":70,"value":221},{"type":64,"tag":190,"props":1169,"children":1170},{"style":203},[1171],{"type":70,"value":1172}," Qwen\u002FQwen2.5-0.5B",{"type":64,"tag":190,"props":1174,"children":1175},{"style":209},[1176],{"type":70,"value":212},{"type":64,"tag":190,"props":1178,"children":1179},{"class":192,"line":233},[1180,1184,1189],{"type":64,"tag":190,"props":1181,"children":1182},{"style":203},[1183],{"type":70,"value":239},{"type":64,"tag":190,"props":1185,"children":1186},{"style":203},[1187],{"type":70,"value":1188}," trl-lib\u002Fgsm8k",{"type":64,"tag":190,"props":1190,"children":1191},{"style":209},[1192],{"type":70,"value":212},{"type":64,"tag":190,"props":1194,"children":1195},{"class":192,"line":251},[1196,1201,1206],{"type":64,"tag":190,"props":1197,"children":1198},{"style":203},[1199],{"type":70,"value":1200}," --reward_funcs",{"type":64,"tag":190,"props":1202,"children":1203},{"style":203},[1204],{"type":70,"value":1205}," accuracy_reward",{"type":64,"tag":190,"props":1207,"children":1208},{"style":209},[1209],{"type":70,"value":212},{"type":64,"tag":190,"props":1211,"children":1212},{"class":192,"line":269},[1213,1217,1222],{"type":64,"tag":190,"props":1214,"children":1215},{"style":203},[1216],{"type":70,"value":408},{"type":64,"tag":190,"props":1218,"children":1219},{"style":203},[1220],{"type":70,"value":1221}," Qwen2-0.5B-GRPO",{"type":64,"tag":190,"props":1223,"children":1224},{"style":209},[1225],{"type":70,"value":212},{"type":64,"tag":190,"props":1227,"children":1228},{"class":192,"line":288},[1229],{"type":64,"tag":190,"props":1230,"children":1231},{"style":203},[1232],{"type":70,"value":426},{"type":64,"tag":158,"props":1234,"children":1236},{"id":1235},"trl-rloo-reinforce-leave-one-out",[1237],{"type":70,"value":1238},"trl rloo - Reinforce Leave One Out",{"type":64,"tag":73,"props":1240,"children":1241},{},[1242],{"type":70,"value":1243},"Online RL training where the model generates text and receives rewards based on custom criteria.",{"type":64,"tag":73,"props":1245,"children":1246},{},[1247],{"type":64,"tag":99,"props":1248,"children":1249},{},[1250],{"type":70,"value":1137},{"type":64,"tag":178,"props":1252,"children":1254},{"className":180,"code":1253,"language":182,"meta":183,"style":183},"trl rloo \\\n --model_name_or_path Qwen\u002FQwen2.5-0.5B \\\n --dataset_name trl-lib\u002Ftldr \\\n --reward_model_name_or_path sentiment-analysis:nlptown\u002Fbert-base-multilingual-uncased-sentiment \\\n --output_dir Qwen2-0.5B-RLOO \\\n --push_to_hub\n",[1255],{"type":64,"tag":186,"props":1256,"children":1257},{"__ignoreMap":183},[1258,1274,1289,1305,1322,1338],{"type":64,"tag":190,"props":1259,"children":1260},{"class":192,"line":193},[1261,1265,1270],{"type":64,"tag":190,"props":1262,"children":1263},{"style":197},[1264],{"type":70,"value":200},{"type":64,"tag":190,"props":1266,"children":1267},{"style":203},[1268],{"type":70,"value":1269}," rloo",{"type":64,"tag":190,"props":1271,"children":1272},{"style":209},[1273],{"type":70,"value":212},{"type":64,"tag":190,"props":1275,"children":1276},{"class":192,"line":215},[1277,1281,1285],{"type":64,"tag":190,"props":1278,"children":1279},{"style":203},[1280],{"type":70,"value":221},{"type":64,"tag":190,"props":1282,"children":1283},{"style":203},[1284],{"type":70,"value":1172},{"type":64,"tag":190,"props":1286,"children":1287},{"style":209},[1288],{"type":70,"value":212},{"type":64,"tag":190,"props":1290,"children":1291},{"class":192,"line":233},[1292,1296,1301],{"type":64,"tag":190,"props":1293,"children":1294},{"style":203},[1295],{"type":70,"value":239},{"type":64,"tag":190,"props":1297,"children":1298},{"style":203},[1299],{"type":70,"value":1300}," trl-lib\u002Ftldr",{"type":64,"tag":190,"props":1302,"children":1303},{"style":209},[1304],{"type":70,"value":212},{"type":64,"tag":190,"props":1306,"children":1307},{"class":192,"line":251},[1308,1313,1318],{"type":64,"tag":190,"props":1309,"children":1310},{"style":203},[1311],{"type":70,"value":1312}," --reward_model_name_or_path",{"type":64,"tag":190,"props":1314,"children":1315},{"style":203},[1316],{"type":70,"value":1317}," sentiment-analysis:nlptown\u002Fbert-base-multilingual-uncased-sentiment",{"type":64,"tag":190,"props":1319,"children":1320},{"style":209},[1321],{"type":70,"value":212},{"type":64,"tag":190,"props":1323,"children":1324},{"class":192,"line":269},[1325,1329,1334],{"type":64,"tag":190,"props":1326,"children":1327},{"style":203},[1328],{"type":70,"value":408},{"type":64,"tag":190,"props":1330,"children":1331},{"style":203},[1332],{"type":70,"value":1333}," Qwen2-0.5B-RLOO",{"type":64,"tag":190,"props":1335,"children":1336},{"style":209},[1337],{"type":70,"value":212},{"type":64,"tag":190,"props":1339,"children":1340},{"class":192,"line":288},[1341],{"type":64,"tag":190,"props":1342,"children":1343},{"style":203},[1344],{"type":70,"value":426},{"type":64,"tag":158,"props":1346,"children":1348},{"id":1347},"trl-reward-reward-model-training",[1349],{"type":70,"value":1350},"trl reward - Reward Model Training",{"type":64,"tag":73,"props":1352,"children":1353},{},[1354],{"type":70,"value":1355},"Train a reward model to score text quality for RLHF.",{"type":64,"tag":73,"props":1357,"children":1358},{},[1359],{"type":64,"tag":99,"props":1360,"children":1361},{},[1362],{"type":70,"value":176},{"type":64,"tag":178,"props":1364,"children":1366},{"className":180,"code":1365,"language":182,"meta":183,"style":183},"trl reward \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-5 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048\n",[1367],{"type":64,"tag":186,"props":1368,"children":1369},{"__ignoreMap":183},[1370,1386,1401,1416,1432,1447,1462,1478,1493,1508],{"type":64,"tag":190,"props":1371,"children":1372},{"class":192,"line":193},[1373,1377,1382],{"type":64,"tag":190,"props":1374,"children":1375},{"style":197},[1376],{"type":70,"value":200},{"type":64,"tag":190,"props":1378,"children":1379},{"style":203},[1380],{"type":70,"value":1381}," reward",{"type":64,"tag":190,"props":1383,"children":1384},{"style":209},[1385],{"type":70,"value":212},{"type":64,"tag":190,"props":1387,"children":1388},{"class":192,"line":215},[1389,1393,1397],{"type":64,"tag":190,"props":1390,"children":1391},{"style":203},[1392],{"type":70,"value":221},{"type":64,"tag":190,"props":1394,"children":1395},{"style":203},[1396],{"type":70,"value":751},{"type":64,"tag":190,"props":1398,"children":1399},{"style":209},[1400],{"type":70,"value":212},{"type":64,"tag":190,"props":1402,"children":1403},{"class":192,"line":233},[1404,1408,1412],{"type":64,"tag":190,"props":1405,"children":1406},{"style":203},[1407],{"type":70,"value":239},{"type":64,"tag":190,"props":1409,"children":1410},{"style":203},[1411],{"type":70,"value":735},{"type":64,"tag":190,"props":1413,"children":1414},{"style":209},[1415],{"type":70,"value":212},{"type":64,"tag":190,"props":1417,"children":1418},{"class":192,"line":251},[1419,1423,1428],{"type":64,"tag":190,"props":1420,"children":1421},{"style":203},[1422],{"type":70,"value":408},{"type":64,"tag":190,"props":1424,"children":1425},{"style":203},[1426],{"type":70,"value":1427}," Qwen2-0.5B-Reward",{"type":64,"tag":190,"props":1429,"children":1430},{"style":209},[1431],{"type":70,"value":212},{"type":64,"tag":190,"props":1433,"children":1434},{"class":192,"line":269},[1435,1439,1443],{"type":64,"tag":190,"props":1436,"children":1437},{"style":203},[1438],{"type":70,"value":307},{"type":64,"tag":190,"props":1440,"children":1441},{"style":278},[1442],{"type":70,"value":330},{"type":64,"tag":190,"props":1444,"children":1445},{"style":209},[1446],{"type":70,"value":212},{"type":64,"tag":190,"props":1448,"children":1449},{"class":192,"line":288},[1450,1454,1458],{"type":64,"tag":190,"props":1451,"children":1452},{"style":203},[1453],{"type":70,"value":275},{"type":64,"tag":190,"props":1455,"children":1456},{"style":278},[1457],{"type":70,"value":281},{"type":64,"tag":190,"props":1459,"children":1460},{"style":209},[1461],{"type":70,"value":212},{"type":64,"tag":190,"props":1463,"children":1464},{"class":192,"line":301},[1465,1469,1474],{"type":64,"tag":190,"props":1466,"children":1467},{"style":203},[1468],{"type":70,"value":257},{"type":64,"tag":190,"props":1470,"children":1471},{"style":203},[1472],{"type":70,"value":1473}," 1.0e-5",{"type":64,"tag":190,"props":1475,"children":1476},{"style":209},[1477],{"type":70,"value":212},{"type":64,"tag":190,"props":1479,"children":1480},{"class":192,"line":319},[1481,1485,1489],{"type":64,"tag":190,"props":1482,"children":1483},{"style":203},[1484],{"type":70,"value":372},{"type":64,"tag":190,"props":1486,"children":1487},{"style":203},[1488],{"type":70,"value":377},{"type":64,"tag":190,"props":1490,"children":1491},{"style":209},[1492],{"type":70,"value":212},{"type":64,"tag":190,"props":1494,"children":1495},{"class":192,"line":337},[1496,1500,1504],{"type":64,"tag":190,"props":1497,"children":1498},{"style":203},[1499],{"type":70,"value":390},{"type":64,"tag":190,"props":1501,"children":1502},{"style":278},[1503],{"type":70,"value":860},{"type":64,"tag":190,"props":1505,"children":1506},{"style":209},[1507],{"type":70,"value":212},{"type":64,"tag":190,"props":1509,"children":1510},{"class":192,"line":366},[1511,1516],{"type":64,"tag":190,"props":1512,"children":1513},{"style":203},[1514],{"type":70,"value":1515}," --max_length",{"type":64,"tag":190,"props":1517,"children":1518},{"style":278},[1519],{"type":70,"value":1520}," 2048\n",{"type":64,"tag":73,"props":1522,"children":1523},{},[1524],{"type":64,"tag":99,"props":1525,"children":1526},{},[1527],{"type":70,"value":434},{"type":64,"tag":178,"props":1529,"children":1531},{"className":180,"code":1530,"language":182,"meta":183,"style":183},"trl reward \\\n --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n --dataset_name trl-lib\u002Fultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward-LoRA \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-4 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048 \\\n --use_peft \\\n --lora_task_type SEQ_CLS \\\n --lora_r 32 \\\n --lora_alpha 16\n",[1532],{"type":64,"tag":186,"props":1533,"children":1534},{"__ignoreMap":183},[1535,1550,1565,1580,1596,1611,1626,1642,1657,1672,1688,1699,1716,1731],{"type":64,"tag":190,"props":1536,"children":1537},{"class":192,"line":193},[1538,1542,1546],{"type":64,"tag":190,"props":1539,"children":1540},{"style":197},[1541],{"type":70,"value":200},{"type":64,"tag":190,"props":1543,"children":1544},{"style":203},[1545],{"type":70,"value":1381},{"type":64,"tag":190,"props":1547,"children":1548},{"style":209},[1549],{"type":70,"value":212},{"type":64,"tag":190,"props":1551,"children":1552},{"class":192,"line":215},[1553,1557,1561],{"type":64,"tag":190,"props":1554,"children":1555},{"style":203},[1556],{"type":70,"value":221},{"type":64,"tag":190,"props":1558,"children":1559},{"style":203},[1560],{"type":70,"value":751},{"type":64,"tag":190,"props":1562,"children":1563},{"style":209},[1564],{"type":70,"value":212},{"type":64,"tag":190,"props":1566,"children":1567},{"class":192,"line":233},[1568,1572,1576],{"type":64,"tag":190,"props":1569,"children":1570},{"style":203},[1571],{"type":70,"value":239},{"type":64,"tag":190,"props":1573,"children":1574},{"style":203},[1575],{"type":70,"value":735},{"type":64,"tag":190,"props":1577,"children":1578},{"style":209},[1579],{"type":70,"value":212},{"type":64,"tag":190,"props":1581,"children":1582},{"class":192,"line":251},[1583,1587,1592],{"type":64,"tag":190,"props":1584,"children":1585},{"style":203},[1586],{"type":70,"value":408},{"type":64,"tag":190,"props":1588,"children":1589},{"style":203},[1590],{"type":70,"value":1591}," Qwen2-0.5B-Reward-LoRA",{"type":64,"tag":190,"props":1593,"children":1594},{"style":209},[1595],{"type":70,"value":212},{"type":64,"tag":190,"props":1597,"children":1598},{"class":192,"line":269},[1599,1603,1607],{"type":64,"tag":190,"props":1600,"children":1601},{"style":203},[1602],{"type":70,"value":307},{"type":64,"tag":190,"props":1604,"children":1605},{"style":278},[1606],{"type":70,"value":330},{"type":64,"tag":190,"props":1608,"children":1609},{"style":209},[1610],{"type":70,"value":212},{"type":64,"tag":190,"props":1612,"children":1613},{"class":192,"line":288},[1614,1618,1622],{"type":64,"tag":190,"props":1615,"children":1616},{"style":203},[1617],{"type":70,"value":275},{"type":64,"tag":190,"props":1619,"children":1620},{"style":278},[1621],{"type":70,"value":281},{"type":64,"tag":190,"props":1623,"children":1624},{"style":209},[1625],{"type":70,"value":212},{"type":64,"tag":190,"props":1627,"children":1628},{"class":192,"line":301},[1629,1633,1638],{"type":64,"tag":190,"props":1630,"children":1631},{"style":203},[1632],{"type":70,"value":257},{"type":64,"tag":190,"props":1634,"children":1635},{"style":203},[1636],{"type":70,"value":1637}," 1.0e-4",{"type":64,"tag":190,"props":1639,"children":1640},{"style":209},[1641],{"type":70,"value":212},{"type":64,"tag":190,"props":1643,"children":1644},{"class":192,"line":319},[1645,1649,1653],{"type":64,"tag":190,"props":1646,"children":1647},{"style":203},[1648],{"type":70,"value":372},{"type":64,"tag":190,"props":1650,"children":1651},{"style":203},[1652],{"type":70,"value":377},{"type":64,"tag":190,"props":1654,"children":1655},{"style":209},[1656],{"type":70,"value":212},{"type":64,"tag":190,"props":1658,"children":1659},{"class":192,"line":337},[1660,1664,1668],{"type":64,"tag":190,"props":1661,"children":1662},{"style":203},[1663],{"type":70,"value":390},{"type":64,"tag":190,"props":1665,"children":1666},{"style":278},[1667],{"type":70,"value":860},{"type":64,"tag":190,"props":1669,"children":1670},{"style":209},[1671],{"type":70,"value":212},{"type":64,"tag":190,"props":1673,"children":1674},{"class":192,"line":366},[1675,1679,1684],{"type":64,"tag":190,"props":1676,"children":1677},{"style":203},[1678],{"type":70,"value":1515},{"type":64,"tag":190,"props":1680,"children":1681},{"style":278},[1682],{"type":70,"value":1683}," 2048",{"type":64,"tag":190,"props":1685,"children":1686},{"style":209},[1687],{"type":70,"value":212},{"type":64,"tag":190,"props":1689,"children":1690},{"class":192,"line":384},[1691,1695],{"type":64,"tag":190,"props":1692,"children":1693},{"style":203},[1694],{"type":70,"value":619},{"type":64,"tag":190,"props":1696,"children":1697},{"style":209},[1698],{"type":70,"value":212},{"type":64,"tag":190,"props":1700,"children":1701},{"class":192,"line":402},[1702,1707,1712],{"type":64,"tag":190,"props":1703,"children":1704},{"style":203},[1705],{"type":70,"value":1706}," --lora_task_type",{"type":64,"tag":190,"props":1708,"children":1709},{"style":203},[1710],{"type":70,"value":1711}," SEQ_CLS",{"type":64,"tag":190,"props":1713,"children":1714},{"style":209},[1715],{"type":70,"value":212},{"type":64,"tag":190,"props":1717,"children":1718},{"class":192,"line":420},[1719,1723,1727],{"type":64,"tag":190,"props":1720,"children":1721},{"style":203},[1722],{"type":70,"value":631},{"type":64,"tag":190,"props":1724,"children":1725},{"style":278},[1726],{"type":70,"value":636},{"type":64,"tag":190,"props":1728,"children":1729},{"style":209},[1730],{"type":70,"value":212},{"type":64,"tag":190,"props":1732,"children":1733},{"class":192,"line":643},[1734,1738],{"type":64,"tag":190,"props":1735,"children":1736},{"style":203},[1737],{"type":70,"value":649},{"type":64,"tag":190,"props":1739,"children":1740},{"style":278},[1741],{"type":70,"value":1118},{"type":64,"tag":79,"props":1743,"children":1745},{"id":1744},"configuration-files",[1746],{"type":70,"value":1747},"Configuration Files",{"type":64,"tag":73,"props":1749,"children":1750},{},[1751],{"type":70,"value":1752},"TRL supports YAML configuration files for reproducible training. All CLI arguments can be specified in a config file.",{"type":64,"tag":73,"props":1754,"children":1755},{},[1756],{"type":64,"tag":99,"props":1757,"children":1758},{},[1759],{"type":70,"value":1760},"Example config (sft_config.yaml):",{"type":64,"tag":178,"props":1762,"children":1766},{"className":1763,"code":1764,"language":1765,"meta":183,"style":183},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","model_name_or_path: Qwen\u002FQwen2.5-0.5B\ndataset_name: trl-lib\u002FCapybara\nlearning_rate: 2.0e-5\nnum_train_epochs: 1\nper_device_train_batch_size: 8\ngradient_accumulation_steps: 2\noutput_dir: .\u002Fsft_output\nuse_peft: true\nlora_r: 16\nlora_alpha: 16\nreport_to: trackio\n","yaml",[1767],{"type":64,"tag":186,"props":1768,"children":1769},{"__ignoreMap":183},[1770,1789,1806,1823,1840,1857,1874,1891,1909,1925,1941],{"type":64,"tag":190,"props":1771,"children":1772},{"class":192,"line":193},[1773,1779,1784],{"type":64,"tag":190,"props":1774,"children":1776},{"style":1775},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[1777],{"type":70,"value":1778},"model_name_or_path",{"type":64,"tag":190,"props":1780,"children":1781},{"style":346},[1782],{"type":70,"value":1783},":",{"type":64,"tag":190,"props":1785,"children":1786},{"style":203},[1787],{"type":70,"value":1788}," Qwen\u002FQwen2.5-0.5B\n",{"type":64,"tag":190,"props":1790,"children":1791},{"class":192,"line":215},[1792,1797,1801],{"type":64,"tag":190,"props":1793,"children":1794},{"style":1775},[1795],{"type":70,"value":1796},"dataset_name",{"type":64,"tag":190,"props":1798,"children":1799},{"style":346},[1800],{"type":70,"value":1783},{"type":64,"tag":190,"props":1802,"children":1803},{"style":203},[1804],{"type":70,"value":1805}," trl-lib\u002FCapybara\n",{"type":64,"tag":190,"props":1807,"children":1808},{"class":192,"line":233},[1809,1814,1818],{"type":64,"tag":190,"props":1810,"children":1811},{"style":1775},[1812],{"type":70,"value":1813},"learning_rate",{"type":64,"tag":190,"props":1815,"children":1816},{"style":346},[1817],{"type":70,"value":1783},{"type":64,"tag":190,"props":1819,"children":1820},{"style":278},[1821],{"type":70,"value":1822}," 2.0e-5\n",{"type":64,"tag":190,"props":1824,"children":1825},{"class":192,"line":251},[1826,1831,1835],{"type":64,"tag":190,"props":1827,"children":1828},{"style":1775},[1829],{"type":70,"value":1830},"num_train_epochs",{"type":64,"tag":190,"props":1832,"children":1833},{"style":346},[1834],{"type":70,"value":1783},{"type":64,"tag":190,"props":1836,"children":1837},{"style":278},[1838],{"type":70,"value":1839}," 1\n",{"type":64,"tag":190,"props":1841,"children":1842},{"class":192,"line":269},[1843,1848,1852],{"type":64,"tag":190,"props":1844,"children":1845},{"style":1775},[1846],{"type":70,"value":1847},"per_device_train_batch_size",{"type":64,"tag":190,"props":1849,"children":1850},{"style":346},[1851],{"type":70,"value":1783},{"type":64,"tag":190,"props":1853,"children":1854},{"style":278},[1855],{"type":70,"value":1856}," 8\n",{"type":64,"tag":190,"props":1858,"children":1859},{"class":192,"line":288},[1860,1865,1869],{"type":64,"tag":190,"props":1861,"children":1862},{"style":1775},[1863],{"type":70,"value":1864},"gradient_accumulation_steps",{"type":64,"tag":190,"props":1866,"children":1867},{"style":346},[1868],{"type":70,"value":1783},{"type":64,"tag":190,"props":1870,"children":1871},{"style":278},[1872],{"type":70,"value":1873}," 2\n",{"type":64,"tag":190,"props":1875,"children":1876},{"class":192,"line":301},[1877,1882,1886],{"type":64,"tag":190,"props":1878,"children":1879},{"style":1775},[1880],{"type":70,"value":1881},"output_dir",{"type":64,"tag":190,"props":1883,"children":1884},{"style":346},[1885],{"type":70,"value":1783},{"type":64,"tag":190,"props":1887,"children":1888},{"style":203},[1889],{"type":70,"value":1890}," .\u002Fsft_output\n",{"type":64,"tag":190,"props":1892,"children":1893},{"class":192,"line":319},[1894,1899,1903],{"type":64,"tag":190,"props":1895,"children":1896},{"style":1775},[1897],{"type":70,"value":1898},"use_peft",{"type":64,"tag":190,"props":1900,"children":1901},{"style":346},[1902],{"type":70,"value":1783},{"type":64,"tag":190,"props":1904,"children":1906},{"style":1905},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[1907],{"type":70,"value":1908}," true\n",{"type":64,"tag":190,"props":1910,"children":1911},{"class":192,"line":337},[1912,1917,1921],{"type":64,"tag":190,"props":1913,"children":1914},{"style":1775},[1915],{"type":70,"value":1916},"lora_r",{"type":64,"tag":190,"props":1918,"children":1919},{"style":346},[1920],{"type":70,"value":1783},{"type":64,"tag":190,"props":1922,"children":1923},{"style":278},[1924],{"type":70,"value":1118},{"type":64,"tag":190,"props":1926,"children":1927},{"class":192,"line":366},[1928,1933,1937],{"type":64,"tag":190,"props":1929,"children":1930},{"style":1775},[1931],{"type":70,"value":1932},"lora_alpha",{"type":64,"tag":190,"props":1934,"children":1935},{"style":346},[1936],{"type":70,"value":1783},{"type":64,"tag":190,"props":1938,"children":1939},{"style":278},[1940],{"type":70,"value":1118},{"type":64,"tag":190,"props":1942,"children":1943},{"class":192,"line":384},[1944,1949,1953],{"type":64,"tag":190,"props":1945,"children":1946},{"style":1775},[1947],{"type":70,"value":1948},"report_to",{"type":64,"tag":190,"props":1950,"children":1951},{"style":346},[1952],{"type":70,"value":1783},{"type":64,"tag":190,"props":1954,"children":1955},{"style":203},[1956],{"type":70,"value":1957}," trackio\n",{"type":64,"tag":73,"props":1959,"children":1960},{},[1961],{"type":64,"tag":99,"props":1962,"children":1963},{},[1964],{"type":70,"value":1965},"Launch with config:",{"type":64,"tag":178,"props":1967,"children":1969},{"className":180,"code":1968,"language":182,"meta":183,"style":183},"trl sft --config sft_config.yaml\n",[1970],{"type":64,"tag":186,"props":1971,"children":1972},{"__ignoreMap":183},[1973],{"type":64,"tag":190,"props":1974,"children":1975},{"class":192,"line":193},[1976,1980,1984,1989],{"type":64,"tag":190,"props":1977,"children":1978},{"style":197},[1979],{"type":70,"value":200},{"type":64,"tag":190,"props":1981,"children":1982},{"style":203},[1983],{"type":70,"value":206},{"type":64,"tag":190,"props":1985,"children":1986},{"style":203},[1987],{"type":70,"value":1988}," --config",{"type":64,"tag":190,"props":1990,"children":1991},{"style":203},[1992],{"type":70,"value":1993}," sft_config.yaml\n",{"type":64,"tag":73,"props":1995,"children":1996},{},[1997],{"type":64,"tag":99,"props":1998,"children":1999},{},[2000],{"type":70,"value":2001},"Override config values:",{"type":64,"tag":178,"props":2003,"children":2005},{"className":180,"code":2004,"language":182,"meta":183,"style":183},"trl sft --config sft_config.yaml --learning_rate 1.0e-5\n",[2006],{"type":64,"tag":186,"props":2007,"children":2008},{"__ignoreMap":183},[2009],{"type":64,"tag":190,"props":2010,"children":2011},{"class":192,"line":193},[2012,2016,2020,2024,2029,2034],{"type":64,"tag":190,"props":2013,"children":2014},{"style":197},[2015],{"type":70,"value":200},{"type":64,"tag":190,"props":2017,"children":2018},{"style":203},[2019],{"type":70,"value":206},{"type":64,"tag":190,"props":2021,"children":2022},{"style":203},[2023],{"type":70,"value":1988},{"type":64,"tag":190,"props":2025,"children":2026},{"style":203},[2027],{"type":70,"value":2028}," sft_config.yaml",{"type":64,"tag":190,"props":2030,"children":2031},{"style":203},[2032],{"type":70,"value":2033}," --learning_rate",{"type":64,"tag":190,"props":2035,"children":2036},{"style":203},[2037],{"type":70,"value":2038}," 1.0e-5\n",{"type":64,"tag":79,"props":2040,"children":2042},{"id":2041},"distributed-training",[2043],{"type":70,"value":2044},"Distributed Training",{"type":64,"tag":73,"props":2046,"children":2047},{},[2048],{"type":70,"value":2049},"TRL integrates with Accelerate for multi-GPU and multi-node training.",{"type":64,"tag":73,"props":2051,"children":2052},{},[2053],{"type":64,"tag":99,"props":2054,"children":2055},{},[2056],{"type":70,"value":2057},"Multi-GPU training:",{"type":64,"tag":178,"props":2059,"children":2061},{"className":180,"code":2060,"language":182,"meta":183,"style":183},"trl sft \\\n --config sft_config.yaml \\\n --num_processes 4\n",[2062],{"type":64,"tag":186,"props":2063,"children":2064},{"__ignoreMap":183},[2065,2080,2096],{"type":64,"tag":190,"props":2066,"children":2067},{"class":192,"line":193},[2068,2072,2076],{"type":64,"tag":190,"props":2069,"children":2070},{"style":197},[2071],{"type":70,"value":200},{"type":64,"tag":190,"props":2073,"children":2074},{"style":203},[2075],{"type":70,"value":206},{"type":64,"tag":190,"props":2077,"children":2078},{"style":209},[2079],{"type":70,"value":212},{"type":64,"tag":190,"props":2081,"children":2082},{"class":192,"line":215},[2083,2088,2092],{"type":64,"tag":190,"props":2084,"children":2085},{"style":203},[2086],{"type":70,"value":2087}," --config",{"type":64,"tag":190,"props":2089,"children":2090},{"style":203},[2091],{"type":70,"value":2028},{"type":64,"tag":190,"props":2093,"children":2094},{"style":209},[2095],{"type":70,"value":212},{"type":64,"tag":190,"props":2097,"children":2098},{"class":192,"line":233},[2099,2104],{"type":64,"tag":190,"props":2100,"children":2101},{"style":203},[2102],{"type":70,"value":2103}," --num_processes",{"type":64,"tag":190,"props":2105,"children":2106},{"style":278},[2107],{"type":70,"value":2108}," 4\n",{"type":64,"tag":73,"props":2110,"children":2111},{},[2112],{"type":64,"tag":99,"props":2113,"children":2114},{},[2115],{"type":70,"value":2116},"Use predefined Accelerate configs:",{"type":64,"tag":73,"props":2118,"children":2119},{},[2120,2122,2128,2130,2136,2137,2143,2144,2150,2151,2157,2158,2164,2165],{"type":70,"value":2121},"TRL provides predefined configs: ",{"type":64,"tag":186,"props":2123,"children":2125},{"className":2124},[],[2126],{"type":70,"value":2127},"single_gpu",{"type":70,"value":2129},", ",{"type":64,"tag":186,"props":2131,"children":2133},{"className":2132},[],[2134],{"type":70,"value":2135},"multi_gpu",{"type":70,"value":2129},{"type":64,"tag":186,"props":2138,"children":2140},{"className":2139},[],[2141],{"type":70,"value":2142},"fsdp1",{"type":70,"value":2129},{"type":64,"tag":186,"props":2145,"children":2147},{"className":2146},[],[2148],{"type":70,"value":2149},"fsdp2",{"type":70,"value":2129},{"type":64,"tag":186,"props":2152,"children":2154},{"className":2153},[],[2155],{"type":70,"value":2156},"zero1",{"type":70,"value":2129},{"type":64,"tag":186,"props":2159,"children":2161},{"className":2160},[],[2162],{"type":70,"value":2163},"zero2",{"type":70,"value":2129},{"type":64,"tag":186,"props":2166,"children":2168},{"className":2167},[],[2169],{"type":70,"value":2170},"zero3",{"type":64,"tag":178,"props":2172,"children":2174},{"className":180,"code":2173,"language":182,"meta":183,"style":183},"trl sft \\\n --config sft_config.yaml \\\n --accelerate_config zero2\n",[2175],{"type":64,"tag":186,"props":2176,"children":2177},{"__ignoreMap":183},[2178,2193,2208],{"type":64,"tag":190,"props":2179,"children":2180},{"class":192,"line":193},[2181,2185,2189],{"type":64,"tag":190,"props":2182,"children":2183},{"style":197},[2184],{"type":70,"value":200},{"type":64,"tag":190,"props":2186,"children":2187},{"style":203},[2188],{"type":70,"value":206},{"type":64,"tag":190,"props":2190,"children":2191},{"style":209},[2192],{"type":70,"value":212},{"type":64,"tag":190,"props":2194,"children":2195},{"class":192,"line":215},[2196,2200,2204],{"type":64,"tag":190,"props":2197,"children":2198},{"style":203},[2199],{"type":70,"value":2087},{"type":64,"tag":190,"props":2201,"children":2202},{"style":203},[2203],{"type":70,"value":2028},{"type":64,"tag":190,"props":2205,"children":2206},{"style":209},[2207],{"type":70,"value":212},{"type":64,"tag":190,"props":2209,"children":2210},{"class":192,"line":233},[2211,2216],{"type":64,"tag":190,"props":2212,"children":2213},{"style":203},[2214],{"type":70,"value":2215}," --accelerate_config",{"type":64,"tag":190,"props":2217,"children":2218},{"style":203},[2219],{"type":70,"value":2220}," zero2\n",{"type":64,"tag":73,"props":2222,"children":2223},{},[2224],{"type":64,"tag":99,"props":2225,"children":2226},{},[2227],{"type":70,"value":2228},"Custom Accelerate config:",{"type":64,"tag":178,"props":2230,"children":2232},{"className":180,"code":2231,"language":182,"meta":183,"style":183},"# Generate custom config\naccelerate config\n\n# Use custom config\ntrl sft --config sft_config.yaml --config_file ~\u002F.cache\u002Fhuggingface\u002Faccelerate\u002Fdefault_config.yaml\n",[2233],{"type":64,"tag":186,"props":2234,"children":2235},{"__ignoreMap":183},[2236,2245,2258,2267,2275],{"type":64,"tag":190,"props":2237,"children":2238},{"class":192,"line":193},[2239],{"type":64,"tag":190,"props":2240,"children":2242},{"style":2241},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[2243],{"type":70,"value":2244},"# Generate custom config\n",{"type":64,"tag":190,"props":2246,"children":2247},{"class":192,"line":215},[2248,2253],{"type":64,"tag":190,"props":2249,"children":2250},{"style":197},[2251],{"type":70,"value":2252},"accelerate",{"type":64,"tag":190,"props":2254,"children":2255},{"style":203},[2256],{"type":70,"value":2257}," config\n",{"type":64,"tag":190,"props":2259,"children":2260},{"class":192,"line":233},[2261],{"type":64,"tag":190,"props":2262,"children":2264},{"emptyLinePlaceholder":2263},true,[2265],{"type":70,"value":2266},"\n",{"type":64,"tag":190,"props":2268,"children":2269},{"class":192,"line":251},[2270],{"type":64,"tag":190,"props":2271,"children":2272},{"style":2241},[2273],{"type":70,"value":2274},"# Use custom config\n",{"type":64,"tag":190,"props":2276,"children":2277},{"class":192,"line":269},[2278,2282,2286,2290,2294,2299],{"type":64,"tag":190,"props":2279,"children":2280},{"style":197},[2281],{"type":70,"value":200},{"type":64,"tag":190,"props":2283,"children":2284},{"style":203},[2285],{"type":70,"value":206},{"type":64,"tag":190,"props":2287,"children":2288},{"style":203},[2289],{"type":70,"value":1988},{"type":64,"tag":190,"props":2291,"children":2292},{"style":203},[2293],{"type":70,"value":2028},{"type":64,"tag":190,"props":2295,"children":2296},{"style":203},[2297],{"type":70,"value":2298}," --config_file",{"type":64,"tag":190,"props":2300,"children":2301},{"style":203},[2302],{"type":70,"value":2303}," ~\u002F.cache\u002Fhuggingface\u002Faccelerate\u002Fdefault_config.yaml\n",{"type":64,"tag":73,"props":2305,"children":2306},{},[2307],{"type":64,"tag":99,"props":2308,"children":2309},{},[2310],{"type":70,"value":2311},"Fully Sharded Data Parallel (FSDP):",{"type":64,"tag":178,"props":2313,"children":2315},{"className":180,"code":2314,"language":182,"meta":183,"style":183},"trl sft --config sft_config.yaml --accelerate_config fsdp2\n",[2316],{"type":64,"tag":186,"props":2317,"children":2318},{"__ignoreMap":183},[2319],{"type":64,"tag":190,"props":2320,"children":2321},{"class":192,"line":193},[2322,2326,2330,2334,2338,2343],{"type":64,"tag":190,"props":2323,"children":2324},{"style":197},[2325],{"type":70,"value":200},{"type":64,"tag":190,"props":2327,"children":2328},{"style":203},[2329],{"type":70,"value":206},{"type":64,"tag":190,"props":2331,"children":2332},{"style":203},[2333],{"type":70,"value":1988},{"type":64,"tag":190,"props":2335,"children":2336},{"style":203},[2337],{"type":70,"value":2028},{"type":64,"tag":190,"props":2339,"children":2340},{"style":203},[2341],{"type":70,"value":2342}," --accelerate_config",{"type":64,"tag":190,"props":2344,"children":2345},{"style":203},[2346],{"type":70,"value":2347}," fsdp2\n",{"type":64,"tag":73,"props":2349,"children":2350},{},[2351],{"type":64,"tag":99,"props":2352,"children":2353},{},[2354],{"type":70,"value":2355},"DeepSpeed ZeRO:",{"type":64,"tag":178,"props":2357,"children":2359},{"className":180,"code":2358,"language":182,"meta":183,"style":183},"trl sft --config sft_config.yaml --accelerate_config zero3\n",[2360],{"type":64,"tag":186,"props":2361,"children":2362},{"__ignoreMap":183},[2363],{"type":64,"tag":190,"props":2364,"children":2365},{"class":192,"line":193},[2366,2370,2374,2378,2382,2386],{"type":64,"tag":190,"props":2367,"children":2368},{"style":197},[2369],{"type":70,"value":200},{"type":64,"tag":190,"props":2371,"children":2372},{"style":203},[2373],{"type":70,"value":206},{"type":64,"tag":190,"props":2375,"children":2376},{"style":203},[2377],{"type":70,"value":1988},{"type":64,"tag":190,"props":2379,"children":2380},{"style":203},[2381],{"type":70,"value":2028},{"type":64,"tag":190,"props":2383,"children":2384},{"style":203},[2385],{"type":70,"value":2342},{"type":64,"tag":190,"props":2387,"children":2388},{"style":203},[2389],{"type":70,"value":2390}," zero3\n",{"type":64,"tag":79,"props":2392,"children":2394},{"id":2393},"troubleshooting",[2395],{"type":70,"value":2396},"Troubleshooting",{"type":64,"tag":158,"props":2398,"children":2400},{"id":2399},"cuda-out-of-memory",[2401],{"type":70,"value":2402},"CUDA Out of Memory",{"type":64,"tag":91,"props":2404,"children":2405},{},[2406,2425,2438,2451],{"type":64,"tag":95,"props":2407,"children":2408},{},[2409,2411,2417,2419],{"type":70,"value":2410},"Reduce ",{"type":64,"tag":186,"props":2412,"children":2414},{"className":2413},[],[2415],{"type":70,"value":2416},"--per_device_train_batch_size",{"type":70,"value":2418}," and increase ",{"type":64,"tag":186,"props":2420,"children":2422},{"className":2421},[],[2423],{"type":70,"value":2424},"--gradient_accumulation_steps",{"type":64,"tag":95,"props":2426,"children":2427},{},[2428,2430,2436],{"type":70,"value":2429},"Enable ",{"type":64,"tag":186,"props":2431,"children":2433},{"className":2432},[],[2434],{"type":70,"value":2435},"--use_peft",{"type":70,"value":2437}," for LoRA training",{"type":64,"tag":95,"props":2439,"children":2440},{},[2441,2443,2449],{"type":70,"value":2442},"Use ",{"type":64,"tag":186,"props":2444,"children":2446},{"className":2445},[],[2447],{"type":70,"value":2448},"--gradient_checkpointing",{"type":70,"value":2450}," to save memory",{"type":64,"tag":95,"props":2452,"children":2453},{},[2454],{"type":70,"value":2455},"Try smaller model or longer sequence truncation",{"type":64,"tag":158,"props":2457,"children":2459},{"id":2458},"dataset-loading-issues",[2460],{"type":70,"value":2461},"Dataset Loading Issues",{"type":64,"tag":91,"props":2463,"children":2464},{},[2465,2470,2475,2487],{"type":64,"tag":95,"props":2466,"children":2467},{},[2468],{"type":70,"value":2469},"Verify dataset exists: check Hugging Face Hub or local path",{"type":64,"tag":95,"props":2471,"children":2472},{},[2473],{"type":70,"value":2474},"Check dataset format matches expected columns",{"type":64,"tag":95,"props":2476,"children":2477},{},[2478,2479,2485],{"type":70,"value":2442},{"type":64,"tag":186,"props":2480,"children":2482},{"className":2481},[],[2483],{"type":70,"value":2484},"--dataset_config",{"type":70,"value":2486}," for multi-config datasets",{"type":64,"tag":95,"props":2488,"children":2489},{},[2490,2492],{"type":70,"value":2491},"Inspect dataset: ",{"type":64,"tag":186,"props":2493,"children":2495},{"className":2494},[],[2496],{"type":70,"value":2497},"from datasets import load_dataset; ds = load_dataset(name)",{"type":64,"tag":158,"props":2499,"children":2501},{"id":2500},"model-loading-issues",[2502],{"type":70,"value":2503},"Model Loading Issues",{"type":64,"tag":91,"props":2505,"children":2506},{},[2507,2512,2523,2528],{"type":64,"tag":95,"props":2508,"children":2509},{},[2510],{"type":70,"value":2511},"Verify model exists on Hugging Face Hub",{"type":64,"tag":95,"props":2513,"children":2514},{},[2515,2517],{"type":70,"value":2516},"Check if gated model requires authentication: ",{"type":64,"tag":186,"props":2518,"children":2520},{"className":2519},[],[2521],{"type":70,"value":2522},"hf auth login",{"type":64,"tag":95,"props":2524,"children":2525},{},[2526],{"type":70,"value":2527},"For local models, provide absolute path",{"type":64,"tag":95,"props":2529,"children":2530},{},[2531],{"type":70,"value":2532},"Ensure sufficient disk space and memory",{"type":64,"tag":158,"props":2534,"children":2536},{"id":2535},"slow-training",[2537],{"type":70,"value":2538},"Slow Training",{"type":64,"tag":91,"props":2540,"children":2541},{},[2542,2555,2567,2579,2591],{"type":64,"tag":95,"props":2543,"children":2544},{},[2545,2547,2553],{"type":70,"value":2546},"Enable dataset ",{"type":64,"tag":186,"props":2548,"children":2550},{"className":2549},[],[2551],{"type":70,"value":2552},"--packing",{"type":70,"value":2554}," for short sequences",{"type":64,"tag":95,"props":2556,"children":2557},{},[2558,2560,2565],{"type":70,"value":2559},"Use larger ",{"type":64,"tag":186,"props":2561,"children":2563},{"className":2562},[],[2564],{"type":70,"value":2416},{"type":70,"value":2566}," if memory allows",{"type":64,"tag":95,"props":2568,"children":2569},{},[2570,2571,2577],{"type":70,"value":2429},{"type":64,"tag":186,"props":2572,"children":2574},{"className":2573},[],[2575],{"type":70,"value":2576},"--tf32",{"type":70,"value":2578}," for faster computation on Ampere GPUs",{"type":64,"tag":95,"props":2580,"children":2581},{},[2582,2583,2589],{"type":70,"value":2442},{"type":64,"tag":186,"props":2584,"children":2586},{"className":2585},[],[2587],{"type":70,"value":2588},"--bf16",{"type":70,"value":2590}," on supported hardware",{"type":64,"tag":95,"props":2592,"children":2593},{},[2594,2596],{"type":70,"value":2595},"Consider multi-GPU training with ",{"type":64,"tag":186,"props":2597,"children":2599},{"className":2598},[],[2600],{"type":70,"value":2601},"--num_processes",{"type":64,"tag":158,"props":2603,"children":2605},{"id":2604},"generation-issues-grporloo",[2606],{"type":70,"value":2607},"Generation Issues (GRPO\u002FRLOO)",{"type":64,"tag":91,"props":2609,"children":2610},{},[2611,2616,2637],{"type":64,"tag":95,"props":2612,"children":2613},{},[2614],{"type":70,"value":2615},"Check prompt format in dataset",{"type":64,"tag":95,"props":2617,"children":2618},{},[2619,2621,2627,2629,2635],{"type":70,"value":2620},"Adjust ",{"type":64,"tag":186,"props":2622,"children":2624},{"className":2623},[],[2625],{"type":70,"value":2626},"--temperature",{"type":70,"value":2628}," and ",{"type":64,"tag":186,"props":2630,"children":2632},{"className":2631},[],[2633],{"type":70,"value":2634},"--top_p",{"type":70,"value":2636}," for generation",{"type":64,"tag":95,"props":2638,"children":2639},{},[2640],{"type":70,"value":2641},"Verify the reward function (for GRPO\u002FRLOO)",{"type":64,"tag":79,"props":2643,"children":2645},{"id":2644},"additional-resources",[2646],{"type":70,"value":2647},"Additional Resources",{"type":64,"tag":91,"props":2649,"children":2650},{},[2651,2669,2683],{"type":64,"tag":95,"props":2652,"children":2653},{},[2654,2659,2661],{"type":64,"tag":99,"props":2655,"children":2656},{},[2657],{"type":70,"value":2658},"Documentation",{"type":70,"value":2660},": ",{"type":64,"tag":2662,"props":2663,"children":2667},"a",{"href":2664,"rel":2665},"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl",[2666],"nofollow",[2668],{"type":70,"value":2664},{"type":64,"tag":95,"props":2670,"children":2671},{},[2672,2677,2678],{"type":64,"tag":99,"props":2673,"children":2674},{},[2675],{"type":70,"value":2676},"GitHub",{"type":70,"value":2660},{"type":64,"tag":2662,"props":2679,"children":2681},{"href":28,"rel":2680},[2666],[2682],{"type":70,"value":28},{"type":64,"tag":95,"props":2684,"children":2685},{},[2686,2691,2692],{"type":64,"tag":99,"props":2687,"children":2688},{},[2689],{"type":70,"value":2690},"Examples",{"type":70,"value":2660},{"type":64,"tag":2662,"props":2693,"children":2696},{"href":2694,"rel":2695},"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples",[2666],[2697],{"type":70,"value":2694},{"type":64,"tag":79,"props":2699,"children":2701},{"id":2700},"best-practices",[2702],{"type":70,"value":2703},"Best Practices",{"type":64,"tag":2705,"props":2706,"children":2707},"ol",{},[2708,2718,2735,2769,2785,2795,2805],{"type":64,"tag":95,"props":2709,"children":2710},{},[2711,2716],{"type":64,"tag":99,"props":2712,"children":2713},{},[2714],{"type":70,"value":2715},"Start with SFT",{"type":70,"value":2717},": Always fine-tune base models with SFT before preference alignment",{"type":64,"tag":95,"props":2719,"children":2720},{},[2721,2726,2728,2733],{"type":64,"tag":99,"props":2722,"children":2723},{},[2724],{"type":70,"value":2725},"Use LoRA for efficiency",{"type":70,"value":2727},": Enable ",{"type":64,"tag":186,"props":2729,"children":2731},{"className":2730},[],[2732],{"type":70,"value":2435},{"type":70,"value":2734}," for faster training and lower memory",{"type":64,"tag":95,"props":2736,"children":2737},{},[2738,2743,2745,2751,2753,2759,2761,2767],{"type":64,"tag":99,"props":2739,"children":2740},{},[2741],{"type":70,"value":2742},"Monitor training",{"type":70,"value":2744},": Use ",{"type":64,"tag":186,"props":2746,"children":2748},{"className":2747},[],[2749],{"type":70,"value":2750},"--report_to trackio",{"type":70,"value":2752}," (or ",{"type":64,"tag":186,"props":2754,"children":2756},{"className":2755},[],[2757],{"type":70,"value":2758},"--report_to wandb",{"type":70,"value":2760}," or ",{"type":64,"tag":186,"props":2762,"children":2764},{"className":2763},[],[2765],{"type":70,"value":2766},"--report_to tensorboard",{"type":70,"value":2768},") for tracking",{"type":64,"tag":95,"props":2770,"children":2771},{},[2772,2777,2779],{"type":64,"tag":99,"props":2773,"children":2774},{},[2775],{"type":70,"value":2776},"Save checkpoints",{"type":70,"value":2778},": TRL automatically saves checkpoints in ",{"type":64,"tag":186,"props":2780,"children":2782},{"className":2781},[],[2783],{"type":70,"value":2784},"--output_dir",{"type":64,"tag":95,"props":2786,"children":2787},{},[2788,2793],{"type":64,"tag":99,"props":2789,"children":2790},{},[2791],{"type":70,"value":2792},"Test on small datasets first",{"type":70,"value":2794},": Verify pipeline works before full training",{"type":64,"tag":95,"props":2796,"children":2797},{},[2798,2803],{"type":64,"tag":99,"props":2799,"children":2800},{},[2801],{"type":70,"value":2802},"Use configuration files",{"type":70,"value":2804},": Create YAML configs for reproducibility",{"type":64,"tag":95,"props":2806,"children":2807},{},[2808,2813],{"type":64,"tag":99,"props":2809,"children":2810},{},[2811],{"type":70,"value":2812},"Leverage Accelerate",{"type":70,"value":2814},": Use multi-GPU training for faster iteration",{"type":64,"tag":73,"props":2816,"children":2817},{},[2818],{"type":70,"value":2819},"When helping users with TRL:",{"type":64,"tag":91,"props":2821,"children":2822},{},[2823,2828,2833,2838,2843],{"type":64,"tag":95,"props":2824,"children":2825},{},[2826],{"type":70,"value":2827},"Always check which training method is appropriate for their use case",{"type":64,"tag":95,"props":2829,"children":2830},{},[2831],{"type":70,"value":2832},"Verify dataset format matches the expected schema",{"type":64,"tag":95,"props":2834,"children":2835},{},[2836],{"type":70,"value":2837},"Recommend starting with smaller models for testing",{"type":64,"tag":95,"props":2839,"children":2840},{},[2841],{"type":70,"value":2842},"Suggest LoRA for resource-constrained environments",{"type":64,"tag":95,"props":2844,"children":2845},{},[2846],{"type":70,"value":2847},"Point to specific documentation sections for advanced features",{"type":64,"tag":2849,"props":2850,"children":2851},"style",{},[2852],{"type":70,"value":2853},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2855,"total":193},[2856],{"slug":4,"name":4,"fn":5,"description":6,"org":2857,"tags":2858,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2859,2860,2861,2862,2863],{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":22,"slug":23,"type":14},{"items":2865,"total":3028},[2866,2882,2890,2907,2921,2934,2947,2962,2976,2986,2999,3013],{"slug":2867,"name":2867,"fn":2868,"description":2869,"org":2870,"tags":2871,"stars":2879,"repoUrl":2880,"updatedAt":2881},"train-sentence-transformers","train sentence-transformers models","Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval \u002F pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2872,2873,2874,2875,2876],{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":22,"slug":23,"type":14},{"name":2877,"slug":2878,"type":14},"Search","search",18914,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fsentence-transformers","2026-05-08T05:09:16.820066",{"slug":4,"name":4,"fn":5,"description":6,"org":2883,"tags":2884,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2885,2886,2887,2888,2889],{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":22,"slug":23,"type":14},{"slug":2891,"name":2891,"fn":2892,"description":2893,"org":2894,"tags":2895,"stars":2904,"repoUrl":2905,"updatedAt":2906},"hf-cli","manage Hugging Face Hub resources via CLI","Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2896,2899,2902,2903],{"name":2897,"slug":2898,"type":14},"CLI","cli",{"name":2900,"slug":2901,"type":14},"Datasets","datasets",{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},10861,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills","2026-04-06T18:25:34.020855",{"slug":2908,"name":2908,"fn":2909,"description":2910,"org":2911,"tags":2912,"stars":2904,"repoUrl":2905,"updatedAt":2920},"hf-cloud-aws-context-discovery","discover local AWS environment context","Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like \"use my AWS account\", \"deploy to AWS\", \"use my profile\", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2913,2916,2917],{"name":2914,"slug":2915,"type":14},"AWS","aws",{"name":2897,"slug":2898,"type":14},{"name":2918,"slug":2919,"type":14},"Configuration","configuration","2026-07-08T05:55:33.716099",{"slug":2922,"name":2922,"fn":2923,"description":2924,"org":2925,"tags":2926,"stars":2904,"repoUrl":2905,"updatedAt":2933},"hf-cloud-python-env-setup","set up Python environments for AWS","Set up an isolated Python environment for SageMaker \u002F AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to \"set up the environment\". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2927,2928,2929,2932],{"name":25,"slug":26,"type":14},{"name":2914,"slug":2915,"type":14},{"name":2930,"slug":2931,"type":14},"Engineering","engineering",{"name":22,"slug":23,"type":14},"2026-07-08T05:55:32.505017",{"slug":2935,"name":2935,"fn":2936,"description":2937,"org":2938,"tags":2939,"stars":2904,"repoUrl":2905,"updatedAt":2946},"hf-cloud-sagemaker-deployment-planner","plan model deployments to Amazon SageMaker","Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like \"deploy a model\", \"host this LLM on AWS\", \"serve this embedding model\", \"deploy a reranker\", \"deploy a text-to-image \u002F diffusion model\", \"host this for async inference\", \"create an endpoint\", \"serve my fine-tuned model\", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. \"I just want to get this running on AWS, you figure it out\"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image \u002F diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2940,2941,2942,2945],{"name":25,"slug":26,"type":14},{"name":2914,"slug":2915,"type":14},{"name":2943,"slug":2944,"type":14},"Deployment","deployment",{"name":9,"slug":13,"type":14},"2026-07-08T05:55:37.387689",{"slug":2948,"name":2948,"fn":2949,"description":2950,"org":2951,"tags":2952,"stars":2904,"repoUrl":2905,"updatedAt":2961},"hf-cloud-sagemaker-iam-preflight","configure SageMaker IAM roles","Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2953,2954,2955,2958],{"name":25,"slug":26,"type":14},{"name":2914,"slug":2915,"type":14},{"name":2956,"slug":2957,"type":14},"Permissions","permissions",{"name":2959,"slug":2960,"type":14},"Security","security","2026-07-08T05:55:34.948771",{"slug":2963,"name":2963,"fn":2964,"description":2965,"org":2966,"tags":2967,"stars":2904,"repoUrl":2905,"updatedAt":2975},"hf-cloud-sagemaker-production-defaults","create production-ready SageMaker endpoints","Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2968,2969,2970,2971,2972],{"name":25,"slug":26,"type":14},{"name":2914,"slug":2915,"type":14},{"name":2943,"slug":2944,"type":14},{"name":9,"slug":13,"type":14},{"name":2973,"slug":2974,"type":14},"Monitoring","monitoring","2026-07-08T05:55:38.664702",{"slug":2977,"name":2977,"fn":2978,"description":2979,"org":2980,"tags":2981,"stars":2904,"repoUrl":2905,"updatedAt":2985},"hf-cloud-serving-image-selection","select SageMaker serving containers","Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says \"deploy this LLM\", \"host this HuggingFace model\", \"serve this fine-tuned model\", \"deploy this embedding model\", \"host a reranker\", \"serve a sentence-transformers model\", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings\u002Fcross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2982,2983,2984],{"name":25,"slug":26,"type":14},{"name":2914,"slug":2915,"type":14},{"name":2943,"slug":2944,"type":14},"2026-07-08T05:55:36.173465",{"slug":2987,"name":2987,"fn":2988,"description":2989,"org":2990,"tags":2991,"stars":2904,"repoUrl":2905,"updatedAt":2998},"hf-mcp","access Hugging Face Hub via MCP","Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2992,2993,2994,2995],{"name":2900,"slug":2901,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":2996,"slug":2997,"type":14},"MCP","mcp","2026-04-06T18:25:50.364185",{"slug":3000,"name":3000,"fn":3001,"description":3002,"org":3003,"tags":3004,"stars":2904,"repoUrl":2905,"updatedAt":3012},"hf-mem","estimate memory for Hugging Face models","Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3005,3006,3007,3008,3009],{"name":25,"slug":26,"type":14},{"name":2897,"slug":2898,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":3010,"slug":3011,"type":14},"Performance","performance","2026-06-13T07:23:57.101435",{"slug":3014,"name":3014,"fn":3015,"description":3016,"org":3017,"tags":3018,"stars":2904,"repoUrl":2905,"updatedAt":3027},"huggingface-best","find and compare Hugging Face models","Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: \"best model for X\", \"what model should I use for\", \"top models for [task]\", \"which model runs on my laptop\u002Fmachine\u002Fdevice\", \"recommend a model for\", \"what LLM should I use for\", \"compare models for\", \"what's state of the art for\", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3019,3022,3023,3024],{"name":3020,"slug":3021,"type":14},"Analytics","analytics",{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":3025,"slug":3026,"type":14},"Research","research","2026-04-24T05:09:45.870658",37]