{"id":89372,"date":"2026-08-22T09:51:24","date_gmt":"2026-08-22T06:51:24","guid":{"rendered":"https:\/\/twodots.gr\/?p=89372"},"modified":"2026-08-22T09:51:27","modified_gmt":"2026-08-22T06:51:27","slug":"dpo-preference-data-audit-llm","status":"publish","type":"post","link":"https:\/\/twodots.gr\/hr\/dpo-preference-data-audit-llm\/","title":{"rendered":"DPO \u03c7\u03c9\u03c1\u03af\u03c2 \u03b1\u03c5\u03c4\u03b1\u03c0\u03ac\u03c4\u03b5\u03c2: \u03c0\u03ce\u03c2 \u03b5\u03bb\u03ad\u03b3\u03c7\u03b5\u03b9\u03c2 \u03c4\u03b1 preference data \u03c0\u03c1\u03b9\u03bd \u03b5\u03ba\u03c0\u03b1\u03b9\u03b4\u03b5\u03cd\u03c3\u03b5\u03b9\u03c2 \u03ad\u03bd\u03b1 LLM"},"content":{"rendered":"<div class=\"td-article-lede\">\n<p><strong>\u03a4\u03bf DPO \u03b4\u03b5\u03bd \u03b4\u03b9\u03bf\u03c1\u03b8\u03ce\u03bd\u03b5\u03b9 \u03c4\u03b1 preference data\u00b7 \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 \u03c4\u03bf \u03c3\u03ae\u03bc\u03b1 \u03c0\u03bf\u03c5 \u03c0\u03b5\u03c1\u03b9\u03ad\u03c7\u03bf\u03c5\u03bd.<\/strong> \u0391\u03bd \u03c4\u03bf \u00abchosen\u00bb \u03b5\u03af\u03bd\u03b1\u03b9 \u03c3\u03c5\u03bd\u03ae\u03b8\u03c9\u03c2 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf, \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03cc \u03bb\u03b5\u03be\u03b9\u03bb\u03cc\u03b3\u03b9\u03bf \u03ae \u03c0\u03c1\u03bf\u03ad\u03c1\u03c7\u03b5\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03ac\u03bb\u03bb\u03bf conversational context \u03b1\u03c0\u03cc \u03c4\u03bf \u00abrejected\u00bb, \u03ad\u03bd\u03b1 LLM \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03bc\u03ac\u03b8\u03b5\u03b9 \u03c4\u03bf shortcut \u03b1\u03bd\u03c4\u03af \u03b3\u03b9\u03b1 \u03c4\u03b7 \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03cc\u03c4\u03b7\u03c4\u03b1, \u03c4\u03b7\u03bd \u03b1\u03c3\u03c6\u03ac\u03bb\u03b5\u03b9\u03b1 \u03ae \u03c4\u03bf \u03cd\u03c6\u03bf\u03c2 \u03c0\u03bf\u03c5 \u03b8\u03ad\u03bb\u03b5\u03b9 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ac \u03b7 \u03bf\u03bc\u03ac\u03b4\u03b1.<\/p>\n<p>\u0393\u03b9\u2019 \u03b1\u03c5\u03c4\u03cc \u03c4\u03bf audit \u03c0\u03c1\u03bf\u03b7\u03b3\u03b5\u03af\u03c4\u03b1\u03b9 \u03c4\u03bf\u03c5 fine-tuning. \u0397 \u03c3\u03c9\u03c3\u03c4\u03ae \u03c1\u03bf\u03ae \u03b5\u03bb\u03ad\u03b3\u03c7\u03b5\u03b9 \u03c0\u03c1\u03ce\u03c4\u03b1 \u03c4\u03b7\u03bd \u03b1\u03ba\u03b5\u03c1\u03b1\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03c9\u03bd pairs, \u03c4\u03b9\u03c2 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ad\u03c2 \u03bc\u03ae\u03ba\u03bf\u03c5\u03c2, \u03c4\u03b1 lexical shortcuts, \u03c4\u03b1 \u03cc\u03c1\u03b9\u03b1 \u03c4\u03bf\u03c5 tokenizer \u03ba\u03b1\u03b9 \u03c4\u03b7\u03bd \u03ba\u03ac\u03bb\u03c5\u03c8\u03b7 \u03c4\u03c9\u03bd \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ce\u03bd use cases. \u039c\u03b5\u03c4\u03ac \u03b5\u03ba\u03c0\u03b1\u03b9\u03b4\u03b5\u03cd\u03b5\u03b9 \u03bc\u03b5 DPO \u03ba\u03b1\u03b9 \u03b4\u03b9\u03b1\u03b2\u03ac\u03b6\u03b5\u03b9 reward accuracy \u03ba\u03b1\u03b9 margins \u03b1\u03bd\u03ac slice \u2014 \u03cc\u03c7\u03b9 \u03bc\u03cc\u03bd\u03bf \u03c9\u03c2 \u03ad\u03bd\u03b1\u03bd \u03c3\u03c5\u03bd\u03bf\u03bb\u03b9\u03ba\u03cc \u03bc\u03ad\u03c3\u03bf \u03cc\u03c1\u03bf.<\/p>\n<\/div>\n<div class=\"td-article-toc\">\n<div class=\"td-toc-title\">\u03a0\u03b5\u03c1\u03b9\u03b5\u03c7\u03cc\u03bc\u03b5\u03bd\u03b1<\/div>\n<ul>\n<li><a href=\"#dpo-data-audit-before-training\">\u03a4\u03b9 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03bf DPO \u03ba\u03b1\u03b9 \u03b3\u03b9\u03b1\u03c4\u03af \u03c4\u03bf audit \u03c0\u03c1\u03bf\u03b7\u03b3\u03b5\u03af\u03c4\u03b1\u03b9<\/a><\/li>\n<li><a href=\"#hh-rlhf-subsets\">\u03a4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 subsets, \u03cc\u03c7\u03b9 \u03ad\u03bd\u03b1 \u03b5\u03bd\u03b9\u03b1\u03af\u03bf \u03c3\u03ae\u03bc\u03b1<\/a><\/li>\n<li><a href=\"#preference-pair-integrity\">\u0397 \u03b1\u03ba\u03b5\u03c1\u03b1\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03bf\u03c5 preference pair<\/a><\/li>\n<li><a href=\"#length-bias\">Length bias: \u03cc\u03c4\u03b1\u03bd \u03b7 \u03ad\u03ba\u03c4\u03b1\u03c3\u03b7 \u03b3\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 shortcut<\/a><\/li>\n<li><a href=\"#lexical-shortcuts\">Lexical shortcuts \u03bc\u03b5 TF-IDF \u03ba\u03b1\u03b9 logistic regression<\/a><\/li>\n<li><a href=\"#tokenizer-aware-formatting\">Tokenizer-aware filtering \u03ba\u03b1\u03b9 conversational format<\/a><\/li>\n<li><a href=\"#dpo-trl-lora\">DPO, TRL \u03ba\u03b1\u03b9 LoRA: \u03c4\u03b9 \u03c1\u03c5\u03b8\u03bc\u03af\u03b6\u03b5\u03b9 \u03c4\u03bf pipeline<\/a><\/li>\n<li><a href=\"#metrics-by-slice\">Reward accuracy \u03ba\u03b1\u03b9 margins \u03b1\u03bd\u03ac slice<\/a><\/li>\n<li><a href=\"#smoke-test-not-benchmark\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03b1 30 steps \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 benchmark<\/a><\/li>\n<li><a href=\"#seven-step-preference-audit\">Preference data audit \u03c3\u03b5 \u03b5\u03c0\u03c4\u03ac \u03b2\u03ae\u03bc\u03b1\u03c4\u03b1<\/a><\/li>\n<li><a href=\"#production-eval-contract\">\u0391\u03c0\u03cc \u03c4\u03bf notebook \u03c3\u03b5 production eval contract<\/a><\/li>\n<\/ul>\n<\/div>\n<h2 id=\"dpo-data-audit-before-training\">\u03a4\u03b9 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03bf DPO \u03ba\u03b1\u03b9 \u03b3\u03b9\u03b1\u03c4\u03af \u03c4\u03bf audit \u03c0\u03c1\u03bf\u03b7\u03b3\u03b5\u03af\u03c4\u03b1\u03b9<\/h2>\n<p>\u03a4\u03bf Direct Preference Optimization \u03c0\u03c1\u03bf\u03c3\u03b1\u03c1\u03bc\u03cc\u03b6\u03b5\u03b9 \u03bc\u03b9\u03b1 policy \u03c0\u03ac\u03bd\u03c9 \u03c3\u03b5 \u03b6\u03b5\u03cd\u03b3\u03b7 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03c9\u03bd \u03b3\u03b9\u03b1 \u03c4\u03bf \u03af\u03b4\u03b9\u03bf prompt: \u03bc\u03af\u03b1 \u03c0\u03c1\u03bf\u03c4\u03b9\u03bc\u03ce\u03bc\u03b5\u03bd\u03b7 completion \u03ba\u03b1\u03b9 \u03bc\u03af\u03b1 \u03b1\u03c0\u03bf\u03c1\u03c1\u03b9\u03c6\u03b8\u03b5\u03af\u03c3\u03b1. \u0397 \u03b1\u03c1\u03c7\u03b9\u03ba\u03ae \u03b5\u03c1\u03b3\u03b1\u03c3\u03af\u03b1 \u03b3\u03b9\u03b1 \u03c4\u03bf DPO \u03ad\u03b4\u03b5\u03b9\u03be\u03b5 \u03cc\u03c4\u03b9 \u03b7 \u03c0\u03c1\u03bf\u03c4\u03af\u03bc\u03b7\u03c3\u03b7 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03b9\u03b7\u03b8\u03b5\u03af \u03bc\u03b5 \u03ad\u03bd\u03b1\u03bd \u03ac\u03bc\u03b5\u03c3\u03bf classification-style \u03c3\u03c4\u03cc\u03c7\u03bf \u03c9\u03c2 \u03c0\u03c1\u03bf\u03c2 \u03bc\u03b9\u03b1 reference policy, \u03c7\u03c9\u03c1\u03af\u03c2 \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03b5\u03ba\u03c0\u03b1\u03af\u03b4\u03b5\u03c5\u03c3\u03b7 reward model \u03ba\u03b1\u03b9 \u03c7\u03c9\u03c1\u03af\u03c2 \u03c4\u03bf\u03bd \u03ba\u03bb\u03b1\u03c3\u03b9\u03ba\u03cc \u03b2\u03c1\u03cc\u03c7\u03bf reinforcement learning \u03c4\u03bf\u03c5 PPO.<\/p>\n<p>\u0391\u03c5\u03c4\u03cc \u03b1\u03c0\u03bb\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03c4\u03b7\u03bd optimization \u03c0\u03bb\u03b5\u03c5\u03c1\u03ac, \u03cc\u03c7\u03b9 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03bf\u03c5 \u03c3\u03ae\u03bc\u03b1\u03c4\u03bf\u03c2. \u0397 \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1 \u00abchosen\u00bb \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03bc\u03b9\u03b1 \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7 \u03c0\u03c1\u03bf\u03c4\u03b9\u03bc\u03ae\u03b8\u03b7\u03ba\u03b5 \u03bc\u03ad\u03c3\u03b1 \u03c3\u03c4\u03b7 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03b7 \u03b4\u03b9\u03b1\u03b4\u03b9\u03ba\u03b1\u03c3\u03af\u03b1 \u03c3\u03c5\u03bb\u03bb\u03bf\u03b3\u03ae\u03c2\u00b7 \u03b4\u03b5\u03bd \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03b1\u03c5\u03c4\u03bf\u03bc\u03ac\u03c4\u03c9\u03c2 \u03cc\u03c4\u03b9 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03bd\u03c4\u03b9\u03ba\u03b5\u03b9\u03bc\u03b5\u03bd\u03b9\u03ba\u03ac \u03b1\u03ba\u03c1\u03b9\u03b2\u03ae\u03c2, \u03b1\u03c3\u03c6\u03b1\u03bb\u03ae\u03c2, \u03c3\u03cd\u03bd\u03c4\u03bf\u03bc\u03b7 \u03ae \u03ba\u03b1\u03c4\u03ac\u03bb\u03bb\u03b7\u03bb\u03b7 \u03b3\u03b9\u03b1 \u03ba\u03ac\u03b8\u03b5 \u03c0\u03c1\u03bf\u03ca\u03cc\u03bd. \u0397 \u03c4\u03c1\u03ad\u03c7\u03bf\u03c5\u03c3\u03b1 \u03c4\u03b5\u03ba\u03bc\u03b7\u03c1\u03af\u03c9\u03c3\u03b7 \u03c4\u03bf\u03c5 TRL \u03c0\u03c1\u03bf\u03b5\u03b9\u03b4\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03bc\u03ac\u03bb\u03b9\u03c3\u03c4\u03b1 \u03cc\u03c4\u03b9 chosen \u03ba\u03b1\u03b9 rejected completions \u03bc\u03c0\u03bf\u03c1\u03bf\u03cd\u03bd \u03ba\u03b1\u03b9 \u03bf\u03b9 \u03b4\u03cd\u03bf \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03ba\u03b1\u03bb\u03ad\u03c2 \u03ae \u03ba\u03b1\u03ba\u03ad\u03c2 \u03c3\u03b5 \u03b1\u03c0\u03cc\u03bb\u03c5\u03c4\u03bf\u03c5\u03c2 \u03cc\u03c1\u03bf\u03c5\u03c2.<\/p>\n<p>\u0393\u03b9\u03b1 \u03bc\u03b9\u03b1 \u03b5\u03c0\u03b9\u03c7\u03b5\u03af\u03c1\u03b7\u03c3\u03b7, \u03c4\u03bf \u03bb\u03ac\u03b8\u03bf\u03c2 \u03c3\u03ae\u03bc\u03b1 \u03b3\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 \u03bb\u03b5\u03b9\u03c4\u03bf\u03c5\u03c1\u03b3\u03b9\u03ba\u03ae \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac. \u0388\u03bd\u03b1 assistant \u03b5\u03be\u03c5\u03c0\u03b7\u03c1\u03ad\u03c4\u03b7\u03c3\u03b7\u03c2 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03bc\u03ac\u03b8\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03b7 \u03c6\u03bb\u03c5\u03b1\u03c1\u03af\u03b1 \u03b1\u03bd\u03c4\u03b1\u03bc\u03b5\u03af\u03b2\u03b5\u03c4\u03b1\u03b9, \u03ad\u03bd\u03b1 content workflow \u03cc\u03c4\u03b9 \u03bf\u03b9 \u03b2\u03b5\u03b2\u03b1\u03b9\u03cc\u03c4\u03b7\u03c4\u03b5\u03c2 \u03ba\u03b5\u03c1\u03b4\u03af\u03b6\u03bf\u03c5\u03bd \u03c4\u03b7\u03bd \u03b5\u03c0\u03b9\u03c6\u03c5\u03bb\u03b1\u03ba\u03c4\u03b9\u03ba\u03ae \u03b1\u03ba\u03c1\u03af\u03b2\u03b5\u03b9\u03b1 \u03ba\u03b1\u03b9 \u03ad\u03bd\u03b1 agent \u03cc\u03c4\u03b9 \u03b7 \u03c3\u03c5\u03bc\u03c6\u03c9\u03bd\u03af\u03b1 \u03bc\u03b5 \u03c4\u03bf\u03bd \u03c7\u03c1\u03ae\u03c3\u03c4\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c0\u03c1\u03bf\u03c4\u03b9\u03bc\u03cc\u03c4\u03b5\u03c1\u03b7 \u03b1\u03c0\u03cc \u03c4\u03b7 \u03b4\u03b9\u03cc\u03c1\u03b8\u03c9\u03c3\u03b7. \u038c\u03c0\u03c9\u03c2 \u03ba\u03b1\u03b9 \u03c3\u03c4\u03b7 <a href=\"https:\/\/twodots.gr\/multi-llm-agents-diakyvernisi-metra-perissotero-apo-prompt\/\">\u03b4\u03b9\u03b1\u03ba\u03c5\u03b2\u03ad\u03c1\u03bd\u03b7\u03c3\u03b7 multi-LLM agents<\/a>, \u03b7 \u03b1\u03be\u03b9\u03bf\u03c0\u03b9\u03c3\u03c4\u03af\u03b1 \u03b4\u03b5\u03bd \u03c0\u03c1\u03bf\u03ba\u03cd\u03c0\u03c4\u03b5\u03b9 \u03bc\u03cc\u03bd\u03bf \u03b1\u03c0\u03cc \u03c4\u03bf\u03bd optimizer \u03b1\u03bb\u03bb\u03ac \u03b1\u03c0\u03cc \u03c4\u03b1 \u03b4\u03b5\u03b4\u03bf\u03bc\u03ad\u03bd\u03b1, \u03c4\u03b1 \u03cc\u03c1\u03b9\u03b1 \u03ba\u03b1\u03b9 \u03c4\u03bf\u03bd \u03c4\u03c1\u03cc\u03c0\u03bf \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7\u03c2.<\/p>\n<div class=\"td-article-note\">\n<p><strong>\u03a4\u03bf \u00abchosen\u00bb \u03b5\u03af\u03bd\u03b1\u03b9 \u03c3\u03c7\u03b5\u03c4\u03b9\u03ba\u03ae \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1, \u03cc\u03c7\u03b9 \u03c0\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03b9\u03b7\u03c4\u03b9\u03ba\u03cc \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2:<\/strong> \u03c0\u03c1\u03b9\u03bd \u03b1\u03c0\u03cc \u03c4\u03b7\u03bd \u03b5\u03ba\u03c0\u03b1\u03af\u03b4\u03b5\u03c5\u03c3\u03b7, \u03b7 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03bf\u03c1\u03af\u03c3\u03b5\u03b9 \u03c0\u03bf\u03b9\u03b1 \u03b4\u03b9\u03ac\u03c3\u03c4\u03b1\u03c3\u03b7 \u03c0\u03c1\u03bf\u03c4\u03af\u03bc\u03b7\u03c3\u03b7\u03c2 \u03b5\u03ba\u03c6\u03c1\u03ac\u03b6\u03b5\u03b9 \u03ba\u03ac\u03b8\u03b5 pair, \u03c0\u03bf\u03b9\u03bf\u03c2 \u03c4\u03bf \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b5 \u03ba\u03b1\u03b9 \u03c0\u03bf\u03b9\u03b1 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b7 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03b8\u03b1 \u03b4\u03b5\u03af\u03be\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03c4\u03bf \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03ad\u03bc\u03b1\u03b8\u03b5 \u03c4\u03bf \u03c3\u03c9\u03c3\u03c4\u03cc \u03c3\u03ae\u03bc\u03b1.<\/p>\n<\/div>\n<h2 id=\"hh-rlhf-subsets\">\u03a4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 subsets, \u03cc\u03c7\u03b9 \u03ad\u03bd\u03b1 \u03b5\u03bd\u03b9\u03b1\u03af\u03bf \u03c3\u03ae\u03bc\u03b1<\/h2>\n<p>\u03a4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf \u03b5\u03ba\u03c0\u03b1\u03b9\u03b4\u03b5\u03c5\u03c4\u03b9\u03ba\u03cc workflow \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 \u03c4\u03bc\u03ae\u03bc\u03b1\u03c4\u03b1 \u03c4\u03bf\u03c5 Anthropic HH-RLHF: <em>helpful-base<\/em>, <em>helpful-rejection-sampled<\/em>, <em>helpful-online<\/em> \u03ba\u03b1\u03b9 <em>harmless-base<\/em>. \u03a4\u03bf \u03b5\u03c0\u03af\u03c3\u03b7\u03bc\u03bf repository \u03b5\u03be\u03b7\u03b3\u03b5\u03af \u03cc\u03c4\u03b9 \u03c4\u03b1 helpful \u03b4\u03b5\u03b4\u03bf\u03bc\u03ad\u03bd\u03b1 \u03c0\u03c1\u03bf\u03ad\u03c1\u03c7\u03bf\u03bd\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ad\u03c2 \u03c6\u03ac\u03c3\u03b5\u03b9\u03c2 \u03c3\u03c5\u03bb\u03bb\u03bf\u03b3\u03ae\u03c2 \u2014 base models, rejection sampling \u03ba\u03b1\u03b9 iterated online process \u2014 \u03b5\u03bd\u03ce \u03c4\u03b1 harmless \u03b4\u03b5\u03b4\u03bf\u03bc\u03ad\u03bd\u03b1 \u03c3\u03c5\u03bb\u03bb\u03ad\u03c7\u03b8\u03b7\u03ba\u03b1\u03bd \u03b3\u03b9\u03b1 base models. \u0391\u03c5\u03c4\u03ad\u03c2 \u03bf\u03b9 \u03c0\u03c1\u03bf\u03b5\u03bb\u03b5\u03cd\u03c3\u03b5\u03b9\u03c2 \u03b4\u03b5\u03bd \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b1\u03bd\u03c4\u03b9\u03bc\u03b5\u03c4\u03c9\u03c0\u03af\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03c3\u03b1\u03bd \u03ad\u03bd\u03b1 \u03bf\u03bc\u03bf\u03b9\u03bf\u03b3\u03b5\u03bd\u03ad\u03c2 corpus.<\/p>\n<p>\u03a3\u03c4\u03bf walkthrough \u03b4\u03b5\u03b9\u03b3\u03bc\u03b1\u03c4\u03bf\u03bb\u03b7\u03c0\u03c4\u03bf\u03cd\u03bd\u03c4\u03b1\u03b9 120 training \u03ba\u03b1\u03b9 30 test \u03b5\u03b3\u03b3\u03c1\u03b1\u03c6\u03ad\u03c2 \u03b1\u03c0\u03cc \u03ba\u03ac\u03b8\u03b5 subset \u03bc\u03b5 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc seed 17. \u039f\u03b9 \u03b1\u03c1\u03b9\u03b8\u03bc\u03bf\u03af \u03b5\u03af\u03bd\u03b1\u03b9 \u03c0\u03b1\u03c1\u03ac\u03bc\u03b5\u03c4\u03c1\u03bf\u03b9 \u03b5\u03bd\u03cc\u03c2 \u03bc\u03b9\u03ba\u03c1\u03bf\u03cd, \u03b1\u03bd\u03b1\u03c0\u03b1\u03c1\u03b1\u03b3\u03ce\u03b3\u03b9\u03bc\u03bf\u03c5 \u03c0\u03b5\u03b9\u03c1\u03ac\u03bc\u03b1\u03c4\u03bf\u03c2 \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03c3\u03cd\u03c3\u03c4\u03b1\u03c3\u03b7 \u03b3\u03b9\u03b1 production. \u0397 \u03c3\u03c9\u03c3\u03c4\u03ae \u03b4\u03b5\u03b9\u03b3\u03bc\u03b1\u03c4\u03bf\u03bb\u03b7\u03c8\u03af\u03b1 \u03b5\u03be\u03b1\u03c1\u03c4\u03ac\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03c4\u03b9\u03c2 \u03b3\u03bb\u03ce\u03c3\u03c3\u03b5\u03c2, \u03c4\u03b1 intents, \u03c4\u03b9\u03c2 \u03ba\u03b1\u03c4\u03b7\u03b3\u03bf\u03c1\u03af\u03b5\u03c2 \u03ba\u03b9\u03bd\u03b4\u03cd\u03bd\u03bf\u03c5 \u03ba\u03b1\u03b9 \u03c4\u03b7 \u03c3\u03c5\u03c7\u03bd\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03c9\u03bd \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ce\u03bd \u03b1\u03b9\u03c4\u03b7\u03bc\u03ac\u03c4\u03c9\u03bd.<\/p>\n<p>\u0397 \u03b4\u03b9\u03b1\u03c4\u03ae\u03c1\u03b7\u03c3\u03b7 \u03c4\u03bf\u03c5 \u03c0\u03b5\u03b4\u03af\u03bf\u03c5 <code>source_subset<\/code> \u03b5\u03c0\u03b9\u03c4\u03c1\u03ad\u03c0\u03b5\u03b9 slice-level \u03b1\u03bd\u03ac\u03b3\u03bd\u03c9\u03c3\u03b7. \u0388\u03bd\u03b1 aggregate metric \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03c6\u03b1\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 \u03b8\u03b5\u03c4\u03b9\u03ba\u03cc \u03b5\u03bd\u03ce \u03c4\u03bf \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03b2\u03b5\u03bb\u03c4\u03b9\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c3\u03c4\u03b1 helpful pairs \u03ba\u03b1\u03b9 \u03c5\u03c0\u03bf\u03c7\u03c9\u03c1\u03b5\u03af \u03c3\u03c4\u03bf harmless-base. \u0397 \u03af\u03b4\u03b9\u03b1 \u03bb\u03bf\u03b3\u03b9\u03ba\u03ae \u03b9\u03c3\u03c7\u03cd\u03b5\u03b9 \u03c3\u03c4\u03b1 \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b7\u03c3\u03b9\u03b1\u03ba\u03ac <a href=\"https:\/\/twodots.gr\/analytics-platforms-data-decisions\/\">analytics \u03c0\u03bf\u03c5 \u03bc\u03b5\u03c4\u03b1\u03c4\u03c1\u03ad\u03c0\u03bf\u03c5\u03bd \u03b4\u03b5\u03b4\u03bf\u03bc\u03ad\u03bd\u03b1 \u03c3\u03b5 \u03b1\u03c0\u03bf\u03c6\u03ac\u03c3\u03b5\u03b9\u03c2<\/a>: \u03bf \u03bc\u03ad\u03c3\u03bf\u03c2 \u03cc\u03c1\u03bf\u03c2 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c7\u03c1\u03ae\u03c3\u03b9\u03bc\u03bf\u03c2 \u03bc\u03cc\u03bd\u03bf \u03cc\u03c4\u03b1\u03bd \u03b4\u03b5\u03bd \u03ba\u03c1\u03cd\u03b2\u03b5\u03b9 \u03c4\u03bf segment \u03cc\u03c0\u03bf\u03c5 \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03b7 \u03b2\u03bb\u03ac\u03b2\u03b7.<\/p>\n<h2 id=\"preference-pair-integrity\">\u0397 \u03b1\u03ba\u03b5\u03c1\u03b1\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03bf\u03c5 preference pair<\/h2>\n<p>\u03a0\u03c1\u03b9\u03bd \u03c5\u03c0\u03bf\u03bb\u03bf\u03b3\u03b9\u03c3\u03c4\u03b5\u03af \u03bf\u03c0\u03bf\u03b9\u03bf\u03b4\u03ae\u03c0\u03bf\u03c4\u03b5 metric, \u03c4\u03b1 raw transcripts \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03bc\u03b5\u03c4\u03b1\u03c4\u03c1\u03b1\u03c0\u03bf\u03cd\u03bd \u03c3\u03b5 \u03b4\u03bf\u03bc\u03b7\u03bc\u03ad\u03bd\u03b1 \u03bc\u03b7\u03bd\u03cd\u03bc\u03b1\u03c4\u03b1 \u03bc\u03b5 \u03c1\u03cc\u03bb\u03bf\u03c5\u03c2 \u03c7\u03c1\u03ae\u03c3\u03c4\u03b7 \u03ba\u03b1\u03b9 assistant. \u0388\u03bd\u03b1\u03c2 \u03ad\u03b3\u03ba\u03c5\u03c1\u03bf\u03c2 \u03b4\u03b9\u03ac\u03bb\u03bf\u03b3\u03bf\u03c2 \u03b1\u03c1\u03c7\u03af\u03b6\u03b5\u03b9 \u03b1\u03c0\u03cc \u03c7\u03c1\u03ae\u03c3\u03c4\u03b7, \u03c4\u03b5\u03bb\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 \u03c3\u03b5 assistant, \u03b5\u03bd\u03b1\u03bb\u03bb\u03ac\u03c3\u03c3\u03b5\u03b9 \u03c3\u03c9\u03c3\u03c4\u03ac \u03c1\u03cc\u03bb\u03bf\u03c5\u03c2 \u03ba\u03b1\u03b9 \u03b4\u03b5\u03bd \u03c0\u03b5\u03c1\u03b9\u03ad\u03c7\u03b5\u03b9 \u03ba\u03b5\u03bd\u03cc \u03c0\u03b5\u03c1\u03b9\u03b5\u03c7\u03cc\u03bc\u03b5\u03bd\u03bf. \u0393\u03b9\u03b1 explicit preference format, \u03c4\u03bf TRL \u03c0\u03c1\u03bf\u03c4\u03b5\u03af\u03bd\u03b5\u03b9 \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03cc prompt \u03ba\u03b1\u03b9 \u03b4\u03cd\u03bf completions: chosen \u03ba\u03b1\u03b9 rejected.<\/p>\n<p>\u039f \u03c0\u03b9\u03bf \u03ba\u03c1\u03af\u03c3\u03b9\u03bc\u03bf\u03c2 \u03ad\u03bb\u03b5\u03b3\u03c7\u03bf\u03c2 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b7 chosen \u03ba\u03b1\u03b9 \u03b7 rejected \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7 \u03bd\u03b1 \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 \u03c4\u03bf \u03af\u03b4\u03b9\u03bf prefix \u03b4\u03b9\u03b1\u03bb\u03cc\u03b3\u03bf\u03c5. \u0391\u03bd \u03b1\u03bb\u03bb\u03ac\u03b6\u03b5\u03b9 \u03ba\u03b1\u03b9 \u03c4\u03bf prompt, \u03c4\u03bf pair \u03b4\u03b5\u03bd \u03b1\u03c0\u03bf\u03bc\u03bf\u03bd\u03ce\u03bd\u03b5\u03b9 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03b7\u03c2 \u03c4\u03b5\u03bb\u03b9\u03ba\u03ae\u03c2 \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7\u03c2. \u039f optimizer \u03b2\u03bb\u03ad\u03c0\u03b5\u03b9 \u03b4\u03cd\u03bf \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac \u03c0\u03b1\u03c1\u03b1\u03b4\u03b5\u03af\u03b3\u03bc\u03b1\u03c4\u03b1 \u03ba\u03b1\u03b9 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b1\u03c0\u03bf\u03b4\u03ce\u03c3\u03b5\u03b9 \u03c4\u03b7\u03bd \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1 \u03c3\u03c4\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03c4\u03bf\u03c5 context \u03b1\u03bd\u03c4\u03af \u03c3\u03c4\u03b7\u03bd \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7.<\/p>\n<p>\u039f\u03b9 \u03bc\u03b7 \u03ad\u03b3\u03ba\u03c5\u03c1\u03b5\u03c2 \u03b5\u03b3\u03b3\u03c1\u03b1\u03c6\u03ad\u03c2 \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03bd\u03b1 \u03c6\u03b9\u03bb\u03c4\u03c1\u03ac\u03c1\u03bf\u03bd\u03c4\u03b1\u03b9 \u03c3\u03b9\u03c9\u03c0\u03b7\u03c1\u03ac. \u03a4\u03bf pipeline \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 reason codes \u03cc\u03c0\u03c9\u03c2 <code>empty_turn<\/code>, <code>role_order<\/code>, <code>prefix_mismatch<\/code> \u03ba\u03b1\u03b9 <code>duplicate_pair<\/code>, \u03bc\u03b1\u03b6\u03af \u03bc\u03b5 rejection rate \u03b1\u03bd\u03ac \u03c0\u03b7\u03b3\u03ae. \u039c\u03b9\u03b1 \u03b1\u03c0\u03cc\u03c4\u03bf\u03bc\u03b7 \u03b1\u03cd\u03be\u03b7\u03c3\u03b7 \u03c3\u03c4\u03bf <code>prefix_mismatch<\/code> \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c8\u03b5\u03b9 \u03b1\u03bb\u03bb\u03b1\u03b3\u03ae schema, \u03bb\u03ac\u03b8\u03bf\u03c2 join \u03c3\u03c4\u03b1 logs \u03ae \u03c0\u03c1\u03cc\u03b2\u03bb\u03b7\u03bc\u03b1 \u03c3\u03c4\u03bf feedback collector \u03c0\u03c1\u03b9\u03bd \u03c3\u03c0\u03b1\u03c4\u03b1\u03bb\u03b7\u03b8\u03b5\u03af \u03c7\u03c1\u03cc\u03bd\u03bf\u03c2 GPU.<\/p>\n<h2 id=\"length-bias\">Length bias: \u03cc\u03c4\u03b1\u03bd \u03b7 \u03ad\u03ba\u03c4\u03b1\u03c3\u03b7 \u03b3\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 shortcut<\/h2>\n<p>\u03a4\u03bf audit \u03bc\u03b5\u03c4\u03c1\u03ac \u03c4\u03b9\u03c2 \u03bb\u03ad\u03be\u03b5\u03b9\u03c2 \u03ae, \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b1, \u03c4\u03b1 completion tokens \u03c3\u03c4\u03b7 chosen \u03ba\u03b1\u03b9 \u03c4\u03b7 rejected \u03c0\u03bb\u03b5\u03c5\u03c1\u03ac, \u03c4\u03b7 \u03bc\u03b5\u03c4\u03b1\u03be\u03cd \u03c4\u03bf\u03c5\u03c2 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03ba\u03b1\u03b9 \u03c4\u03bf \u03bc\u03ae\u03ba\u03bf\u03c2 \u03c4\u03bf\u03c5 prompt. \u0395\u03be\u03b5\u03c4\u03ac\u03b6\u03b5\u03b9 \u03c4\u03cc\u03c3\u03bf \u03c4\u03bf\u03bd \u03bc\u03ad\u03c3\u03bf \u03cc\u03c1\u03bf \u03b1\u03bd\u03ac subset \u03cc\u03c3\u03bf \u03ba\u03b1\u03b9 \u03c4\u03b7\u03bd \u03ba\u03b1\u03c4\u03b1\u03bd\u03bf\u03bc\u03ae \u03c4\u03c9\u03bd \u03b6\u03b5\u03c5\u03b3\u03ce\u03bd. \u0388\u03bd\u03b1\u03c2 \u03bc\u03bf\u03bd\u03b1\u03b4\u03b9\u03ba\u03cc\u03c2 \u03bc\u03ad\u03c3\u03bf\u03c2 \u03cc\u03c1\u03bf\u03c2 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03ba\u03c1\u03cd\u03c8\u03b5\u03b9 \u03b4\u03cd\u03bf \u03b1\u03bd\u03c4\u03af\u03b8\u03b5\u03c4\u03b5\u03c2 \u03bf\u03bc\u03ac\u03b4\u03b5\u03c2 \u03ae \u03bb\u03af\u03b3\u03b1 \u03b1\u03ba\u03c1\u03b1\u03af\u03b1 \u03c0\u03b1\u03c1\u03b1\u03b4\u03b5\u03af\u03b3\u03bc\u03b1\u03c4\u03b1.<\/p>\n<p>\u03a4\u03bf \u03b5\u03c1\u03ce\u03c4\u03b7\u03bc\u03b1 \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03bd \u03bc\u03b9\u03b1 \u03ba\u03b1\u03bb\u03ae \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b7. \u0395\u03af\u03bd\u03b1\u03b9 \u03b1\u03bd \u03c4\u03bf \u03bc\u03ae\u03ba\u03bf\u03c2 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c0\u03b5\u03b9 \u03c4\u03cc\u03c3\u03bf \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03ac \u03c4\u03b7\u03bd \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1 \u03ce\u03c3\u03c4\u03b5 \u03c4\u03bf \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03bd\u03b1 \u03ba\u03b5\u03c1\u03b4\u03af\u03b6\u03b5\u03b9 \u03c7\u03c9\u03c1\u03af\u03c2 \u03bd\u03b1 \u03bc\u03b1\u03b8\u03b1\u03af\u03bd\u03b5\u03b9 \u03c4\u03b7 \u03b6\u03b7\u03c4\u03bf\u03cd\u03bc\u03b5\u03bd\u03b7 \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac. \u03a3\u03b5 \u03ad\u03bd\u03b1\u03bd <a href=\"https:\/\/twodots.gr\/6-ai-agents-customer-support-gia-epicheiriseis\/\">AI agent \u03b3\u03b9\u03b1 customer support<\/a>, \u03c4\u03bf \u03bb\u03ac\u03b8\u03bf\u03c2 shortcut \u03b1\u03c5\u03be\u03ac\u03bd\u03b5\u03b9 latency \u03ba\u03b1\u03b9 inference cost, \u03ba\u03bf\u03c5\u03c1\u03ac\u03b6\u03b5\u03b9 \u03c4\u03bf\u03bd \u03c0\u03b5\u03bb\u03ac\u03c4\u03b7 \u03ba\u03b1\u03b9 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03ba\u03c1\u03cd\u03b2\u03b5\u03b9 \u03c4\u03b7\u03bd \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ae \u03bb\u03cd\u03c3\u03b7 \u03bc\u03ad\u03c3\u03b1 \u03c3\u03b5 \u03c0\u03b5\u03c1\u03b9\u03c4\u03c4\u03cc \u03ba\u03b5\u03af\u03bc\u03b5\u03bd\u03bf.<\/p>\n<p>\u03a7\u03c1\u03b5\u03b9\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03c4\u03bf\u03c5\u03bb\u03ac\u03c7\u03b9\u03c3\u03c4\u03bf\u03bd \u03c4\u03c1\u03b5\u03b9\u03c2 \u03cc\u03c8\u03b5\u03b9\u03c2: \u03c0\u03bf\u03c3\u03bf\u03c3\u03c4\u03cc pairs \u03cc\u03c0\u03bf\u03c5 \u03c4\u03bf chosen \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf, \u03ba\u03b1\u03c4\u03b1\u03bd\u03bf\u03bc\u03ae \u03c4\u03b7\u03c2 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac\u03c2 tokens \u03ba\u03b1\u03b9 \u03b5\u03c0\u03af\u03b4\u03bf\u03c3\u03b7 \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5 \u03bc\u03ad\u03c3\u03b1 \u03c3\u03b5 \u03bc\u03ae\u03ba\u03bf\u03c2-\u03b9\u03c3\u03bf\u03c1\u03c1\u03bf\u03c0\u03b7\u03bc\u03ad\u03bd\u03b1 buckets. \u0391\u03bd \u03b7 \u03b2\u03b5\u03bb\u03c4\u03af\u03c9\u03c3\u03b7 \u03b5\u03be\u03b1\u03c6\u03b1\u03bd\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03cc\u03c4\u03b1\u03bd chosen \u03ba\u03b1\u03b9 rejected \u03ad\u03c7\u03bf\u03c5\u03bd \u03c0\u03b1\u03c1\u03cc\u03bc\u03bf\u03b9\u03bf \u03bc\u03ae\u03ba\u03bf\u03c2, \u03c4\u03bf \u03b1\u03c0\u03bf\u03c4\u03ad\u03bb\u03b5\u03c3\u03bc\u03b1 \u03b1\u03c0\u03b1\u03b9\u03c4\u03b5\u03af \u03ad\u03c1\u03b5\u03c5\u03bd\u03b1 \u03c0\u03c1\u03b9\u03bd \u03c0\u03b1\u03c1\u03bf\u03c5\u03c3\u03b9\u03b1\u03c3\u03c4\u03b5\u03af \u03c9\u03c2 \u03c0\u03c1\u03cc\u03bf\u03b4\u03bf\u03c2 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2.<\/p>\n<h2 id=\"lexical-shortcuts\">Lexical shortcuts \u03bc\u03b5 TF-IDF \u03ba\u03b1\u03b9 logistic regression<\/h2>\n<p>\u0388\u03bd\u03b1 \u03b1\u03c0\u03bb\u03cc diagnostic \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03ae\u03c3\u03b5\u03b9 TF-IDF \u03c3\u03b5 unigrams \u03ba\u03b1\u03b9 bigrams \u03ba\u03b1\u03b9 logistic regression \u03b3\u03b9\u03b1 \u03bd\u03b1 \u03b4\u03b9\u03b1\u03c7\u03c9\u03c1\u03af\u03c3\u03b5\u03b9 chosen \u03b1\u03c0\u03cc rejected \u03ba\u03b5\u03af\u03bc\u03b5\u03bd\u03b1. \u03a4\u03bf split \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b3\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 \u03b1\u03bd\u03ac pair ID, \u03ce\u03c3\u03c4\u03b5 \u03bf\u03b9 \u03b4\u03cd\u03bf \u03c0\u03bb\u03b5\u03c5\u03c1\u03ad\u03c2 \u03c4\u03bf\u03c5 \u03af\u03b4\u03b9\u03bf\u03c5 \u03b6\u03b5\u03cd\u03b3\u03bf\u03c5\u03c2 \u03bd\u03b1 \u03bc\u03b7 \u03b4\u03b9\u03b1\u03c1\u03c1\u03b5\u03cd\u03c3\u03bf\u03c5\u03bd \u03c3\u03b5 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac train \u03ba\u03b1\u03b9 test sets. \u0394\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac, \u03b7 \u03b5\u03c0\u03af\u03b4\u03bf\u03c3\u03b7 \u03b8\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03b5\u03c7\u03bd\u03b7\u03c4\u03ac \u03b1\u03b9\u03c3\u03b9\u03cc\u03b4\u03bf\u03be\u03b7.<\/p>\n<p>Accuracy \u03ba\u03b1\u03b9 ROC-AUC \u03b5\u03b4\u03ce \u03b4\u03b5\u03bd \u03b2\u03b1\u03b8\u03bc\u03bf\u03bb\u03bf\u03b3\u03bf\u03cd\u03bd \u03c4\u03bf LLM. \u039c\u03b5\u03c4\u03c1\u03bf\u03cd\u03bd \u03c0\u03cc\u03c3\u03bf \u03b5\u03cd\u03ba\u03bf\u03bb\u03b1 \u03b5\u03c0\u03b9\u03c6\u03b1\u03bd\u03b5\u03b9\u03b1\u03ba\u03ac lexical patterns \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03bf\u03c5\u03bd \u03c4\u03b7\u03bd \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1. \u0397 \u03c3\u03cd\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03bc\u03b5 permuted labels \u03b4\u03af\u03bd\u03b5\u03b9 baseline \u03b8\u03bf\u03c1\u03cd\u03b2\u03bf\u03c5 \u03ba\u03b1\u03b9 \u03bc\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 \u03c4\u03bf\u03bd \u03ba\u03af\u03bd\u03b4\u03c5\u03bd\u03bf \u03bd\u03b1 \u03b5\u03c1\u03bc\u03b7\u03bd\u03b5\u03c5\u03c4\u03b5\u03af \u03ba\u03ac\u03b8\u03b5 \u03bc\u03b9\u03ba\u03c1\u03ae \u03b1\u03c0\u03cc\u03ba\u03bb\u03b9\u03c3\u03b7 \u03b1\u03c0\u03cc \u03c4\u03bf 0,5 \u03c9\u03c2 \u03bf\u03c5\u03c3\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03cc \u03c3\u03ae\u03bc\u03b1.<\/p>\n<p>\u0397 \u03c0\u03b1\u03c1\u03b1\u03c4\u03b7\u03c1\u03b7\u03c3\u03b9\u03bc\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03ba\u03b1\u03b9 \u03c0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03bf\u03cd\u03c2. \u03a3\u03b5 corpus \u03b1\u03c3\u03c6\u03ac\u03bb\u03b5\u03b9\u03b1\u03c2, \u03c4\u03b1 \u03ba\u03bf\u03c1\u03c5\u03c6\u03b1\u03af\u03b1 n-grams \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03c0\u03b5\u03c1\u03b9\u03ad\u03c7\u03bf\u03c5\u03bd \u03c0\u03c1\u03bf\u03c3\u03b2\u03bb\u03b7\u03c4\u03b9\u03ba\u03cc \u03ae \u03b5\u03c5\u03b1\u03af\u03c3\u03b8\u03b7\u03c4\u03bf \u03c5\u03bb\u03b9\u03ba\u03cc. \u0391\u03c0\u03bf\u03b8\u03b7\u03ba\u03b5\u03cd\u03c3\u03c4\u03b5 aggregated \u03b4\u03b5\u03af\u03ba\u03c4\u03b5\u03c2 \u03ba\u03b1\u03b9 \u03b5\u03bb\u03b5\u03b3\u03c7\u03cc\u03bc\u03b5\u03bd\u03b1 hashes \u03cc\u03c0\u03bf\u03c5 \u03b1\u03c1\u03ba\u03bf\u03cd\u03bd\u00b7 \u03bc\u03b7 \u03bc\u03b5\u03c4\u03b1\u03c6\u03ad\u03c1\u03b5\u03c4\u03b5 \u03b1\u03b4\u03b9\u03b1\u03ba\u03c1\u03af\u03c4\u03c9\u03c2 raw feature strings \u03c3\u03b5 dashboards, tickets \u03ba\u03b1\u03b9 logs.<\/p>\n<div class=\"td-comparison\">\n<p class=\"td-comparison-title\">\u03a4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b9\u03c2 \u03ad\u03bb\u03b5\u03b3\u03c7\u03bf\u03b9 \u03c0\u03bf\u03c5 \u03b1\u03c0\u03bf\u03bc\u03bf\u03bd\u03ce\u03bd\u03bf\u03c5\u03bd \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc preference signal<\/p>\n<div class=\"td-comparison-cards td-comparison-cards--horizontal\">\n<div class=\"td-comparison-grid td-comparison-grid--four\">\n<div class=\"td-platform-card\">\n<h3>\u038a\u03b4\u03b9\u03bf prompt<\/h3>\n<p>Chosen \u03ba\u03b1\u03b9 rejected \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03ad\u03c7\u03bf\u03c5\u03bd \u03c4\u03bf \u03af\u03b4\u03b9\u03bf conversational prefix. \u0394\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03cc context \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03c4\u03bf pair \u03b4\u03b5\u03bd \u03bc\u03b5\u03c4\u03c1\u03ac \u03bc\u03cc\u03bd\u03bf \u03c4\u03b7\u03bd \u03c4\u03b5\u03bb\u03b9\u03ba\u03ae \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7.<\/p>\n<\/div>\n<div class=\"td-platform-card\">\n<h3>\u0399\u03c3\u03bf\u03c1\u03c1\u03bf\u03c0\u03af\u03b1 \u03bc\u03ae\u03ba\u03bf\u03c5\u03c2<\/h3>\n<p>\u039c\u03b5\u03c4\u03c1\u03ae\u03c3\u03c4\u03b5 token deltas \u03ba\u03b1\u03b9 accuracy \u03c3\u03b5 \u03bc\u03ae\u03ba\u03bf\u03c2-\u03b9\u03c3\u03bf\u03c1\u03c1\u03bf\u03c0\u03b7\u03bc\u03ad\u03bd\u03b1 buckets. \u0388\u03c4\u03c3\u03b9 \u03be\u03b5\u03c7\u03c9\u03c1\u03af\u03b6\u03b5\u03b9 \u03b7 \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b1\u03c0\u03cc \u03c4\u03b7\u03bd \u03b1\u03c0\u03bb\u03ae \u03c0\u03c1\u03bf\u03c4\u03af\u03bc\u03b7\u03c3\u03b7 \u03c3\u03c4\u03b7 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b7 completion.<\/p>\n<\/div>\n<div class=\"td-platform-card td-platform-card--navy\">\n<h3>Lexical baseline<\/h3>\n<p>\u0388\u03bd\u03b1 pair-safe TF-IDF diagnostic \u03b4\u03b5\u03af\u03c7\u03bd\u03b5\u03b9 \u03b1\u03bd \u03bb\u03ad\u03be\u03b5\u03b9\u03c2 \u03ae \u03c6\u03c1\u03ac\u03c3\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c0\u03bf\u03c5\u03bd \u03b5\u03cd\u03ba\u03bf\u03bb\u03b1 \u03c4\u03b7\u03bd \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1, \u03c0\u03c1\u03b9\u03bd \u03c4\u03bf \u03af\u03b4\u03b9\u03bf shortcut \u03c0\u03b5\u03c1\u03ac\u03c3\u03b5\u03b9 \u03c3\u03c4\u03b7\u03bd policy.<\/p>\n<\/div>\n<div class=\"td-platform-card\">\n<h3>\u039a\u03ac\u03bb\u03c5\u03c8\u03b7 slices<\/h3>\n<p>\u0391\u03be\u03b9\u03bf\u03bb\u03bf\u03b3\u03ae\u03c3\u03c4\u03b5 subset, \u03b3\u03bb\u03ce\u03c3\u03c3\u03b1, intent \u03ba\u03b1\u03b9 risk class \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ac. \u03a4\u03bf aggregate reward accuracy \u03b4\u03b5\u03bd \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03ba\u03c1\u03cd\u03b2\u03b5\u03b9 \u03c4\u03bf segment \u03c0\u03bf\u03c5 \u03c5\u03c0\u03bf\u03c7\u03c9\u03c1\u03b5\u03af.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<h2 id=\"tokenizer-aware-formatting\">Tokenizer-aware filtering \u03ba\u03b1\u03b9 conversational format<\/h2>\n<p>\u039c\u03b5\u03c4\u03ac \u03c4\u03bf semantic audit, \u03ba\u03ac\u03b8\u03b5 pair \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c0\u03b5\u03c1\u03ac\u03c3\u03b5\u03b9 \u03b1\u03c0\u03cc \u03c4\u03bf\u03bd \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc tokenizer \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5. \u03a3\u03c4\u03bf \u03c0\u03b1\u03c1\u03ac\u03b4\u03b5\u03b9\u03b3\u03bc\u03b1 \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af\u03c4\u03b1\u03b9 \u03c4\u03bf Qwen2.5-0.5B-Instruct, \u03bc\u03b5 max prompt length 256 \u03ba\u03b1\u03b9 \u03c3\u03c5\u03bd\u03bf\u03bb\u03b9\u03ba\u03cc max length 512. \u03a4\u03bf pipeline \u03bc\u03b5\u03c4\u03c1\u03ac prompt \u03bc\u03b1\u03b6\u03af \u03bc\u03b5 \u03ba\u03ac\u03b8\u03b5 completion \u03ba\u03b1\u03b9 \u03b1\u03c0\u03bf\u03c1\u03c1\u03af\u03c0\u03c4\u03b5\u03b9 \u03cc\u03c3\u03b1 \u03b6\u03b5\u03cd\u03b3\u03b7 \u03c5\u03c0\u03b5\u03c1\u03b2\u03b1\u03af\u03bd\u03bf\u03c5\u03bd \u03c4\u03b1 \u03cc\u03c1\u03b9\u03b1.<\/p>\n<p>\u0397 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03bc\u03b5 \u03c7\u03b1\u03c1\u03b1\u03ba\u03c4\u03ae\u03c1\u03b5\u03c2 \u03ae \u03bb\u03ad\u03be\u03b5\u03b9\u03c2 \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af. \u0397 tokenization \u03b1\u03bb\u03bb\u03ac\u03b6\u03b5\u03b9 \u03b1\u03bd\u03ac \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf, chat template \u03ba\u03b1\u03b9 \u03b3\u03bb\u03ce\u03c3\u03c3\u03b1. \u0388\u03bd\u03b1 \u03cc\u03c1\u03b9\u03bf \u03c1\u03c5\u03b8\u03bc\u03b9\u03c3\u03bc\u03ad\u03bd\u03bf \u03bc\u03cc\u03bd\u03bf \u03c3\u03c4\u03b1 \u03b1\u03b3\u03b3\u03bb\u03b9\u03ba\u03ac \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03ba\u03cc\u03b2\u03b5\u03b9 \u03b4\u03c5\u03c3\u03b1\u03bd\u03ac\u03bb\u03bf\u03b3\u03b1 \u03b5\u03bb\u03bb\u03b7\u03bd\u03b9\u03ba\u03ac \u03ae \u03c0\u03bf\u03bb\u03cd\u03b3\u03bb\u03c9\u03c3\u03c3\u03b1 \u03c0\u03b1\u03c1\u03b1\u03b4\u03b5\u03af\u03b3\u03bc\u03b1\u03c4\u03b1. \u03a0\u03b1\u03c1\u03b1\u03ba\u03bf\u03bb\u03bf\u03c5\u03b8\u03ae\u03c3\u03c4\u03b5 rejection rate \u03ba\u03b1\u03b9 retained-token distribution \u03b1\u03bd\u03ac \u03b3\u03bb\u03ce\u03c3\u03c3\u03b1, intent \u03ba\u03b1\u03b9 \u03ba\u03b1\u03bd\u03ac\u03bb\u03b9.<\/p>\n<p>\u03a4\u03bf chat template \u03b1\u03c0\u03bf\u03c4\u03b5\u03bb\u03b5\u03af \u03bc\u03ad\u03c1\u03bf\u03c2 \u03c4\u03bf\u03c5 input contract. \u0391\u03bd \u03bb\u03b5\u03af\u03c0\u03b5\u03b9, \u03c4\u03bf walkthrough \u03b5\u03b3\u03ba\u03b1\u03b8\u03b9\u03c3\u03c4\u03ac ChatML fallback, \u03b1\u03bb\u03bb\u03ac \u03c3\u03b5 production \u03b7 \u03b5\u03c0\u03b9\u03bb\u03bf\u03b3\u03ae \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c1\u03b7\u03c4\u03ae, versioned \u03ba\u03b1\u03b9 \u03b4\u03bf\u03ba\u03b9\u03bc\u03b1\u03c3\u03bc\u03ad\u03bd\u03b7 \u03bc\u03b5 \u03c4\u03bf \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03c0\u03bf\u03c5 \u03b8\u03b1 \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b7\u03b8\u03b5\u03af. \u0397 <a href=\"https:\/\/twodots.gr\/asfaleia-llm-meta-to-fine-tuning-prompt-template-ropd\/\">\u03b1\u03c3\u03c6\u03ac\u03bb\u03b5\u03b9\u03b1 \u03b5\u03bd\u03cc\u03c2 LLM \u03bc\u03b5\u03c4\u03ac \u03c4\u03bf fine-tuning \u03b5\u03c0\u03b7\u03c1\u03b5\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03ba\u03b1\u03b9 \u03b1\u03c0\u03cc \u03c4\u03bf prompt template<\/a>, \u03ac\u03c1\u03b1 \u03b1\u03bb\u03bb\u03b1\u03b3\u03ae \u03bc\u03bf\u03c1\u03c6\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7\u03c2 \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03b8\u03ce\u03bf refactor.<\/p>\n<h2 id=\"dpo-trl-lora\">DPO, TRL \u03ba\u03b1\u03b9 LoRA: \u03c4\u03b9 \u03c1\u03c5\u03b8\u03bc\u03af\u03b6\u03b5\u03b9 \u03c4\u03bf pipeline<\/h2>\n<p>\u03a4\u03bf \u03b5\u03ba\u03c0\u03b1\u03b9\u03b4\u03b5\u03c5\u03c4\u03b9\u03ba\u03cc setup \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af Qwen2.5-0.5B-Instruct, beta 0,1, learning rate 5e-6, batch size 1, gradient accumulation 8, warmup ratio 0,1 \u03ba\u03b1\u03b9 30 max steps. \u0397 LoRA \u03bf\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03bc\u03b5 rank 16, alpha 32 \u03ba\u03b1\u03b9 dropout 0,05. \u0391\u03c5\u03c4\u03ad\u03c2 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c0\u03b1\u03c1\u03ac\u03bc\u03b5\u03c4\u03c1\u03bf\u03b9 smoke test, \u03cc\u03c7\u03b9 \u03ba\u03b1\u03b8\u03bf\u03bb\u03b9\u03ba\u03ad\u03c2 \u03b2\u03ad\u03bb\u03c4\u03b9\u03c3\u03c4\u03b5\u03c2 \u03c4\u03b9\u03bc\u03ad\u03c2.<\/p>\n<p>\u03a3\u03c4\u03bf DPO, \u03b7 policy \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 \u03bc\u03b5 reference policy. \u039c\u03b5 PEFT\/LoRA, \u03b7 \u03c0\u03b1\u03b3\u03c9\u03bc\u03ad\u03bd\u03b7 \u03b2\u03ac\u03c3\u03b7 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u2014 \u03b1\u03bd\u03ac\u03bb\u03bf\u03b3\u03b1 \u03bc\u03b5 \u03c4\u03b7 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03b7 \u03c5\u03bb\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03ad\u03ba\u03b4\u03bf\u03c3\u03b7 \u2014 \u03bd\u03b1 \u03bb\u03b5\u03b9\u03c4\u03bf\u03c5\u03c1\u03b3\u03ae\u03c3\u03b5\u03b9 \u03c9\u03c2 reference \u03c7\u03c9\u03c1\u03af\u03c2 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03bf \u03b4\u03b5\u03cd\u03c4\u03b5\u03c1\u03bf \u03c0\u03bb\u03ae\u03c1\u03b5\u03c2 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf. \u0397 \u03c4\u03c1\u03ad\u03c7\u03bf\u03c5\u03c3\u03b1 \u03c4\u03b5\u03ba\u03bc\u03b7\u03c1\u03af\u03c9\u03c3\u03b7 \u03c4\u03bf\u03c5 TRL \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b7 \u03c0\u03b7\u03b3\u03ae \u03b1\u03bb\u03ae\u03b8\u03b5\u03b9\u03b1\u03c2 \u03b3\u03b9\u03b1 \u03c4\u03b1 \u03c5\u03c0\u03bf\u03c3\u03c4\u03b7\u03c1\u03b9\u03b6\u03cc\u03bc\u03b5\u03bd\u03b1 arguments, \u03c4\u03bf dataset format \u03ba\u03b1\u03b9 \u03c4\u03b7 \u03b4\u03b9\u03b1\u03c7\u03b5\u03af\u03c1\u03b9\u03c3\u03b7 adapters.<\/p>\n<p>\u0397 \u03c3\u03c5\u03bc\u03b2\u03b1\u03c4\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b5\u03ba\u03b4\u03cc\u03c3\u03b5\u03c9\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03ad\u03c1\u03bf\u03c2 \u03c4\u03b7\u03c2 \u03b1\u03bd\u03b1\u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03b9\u03bc\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2. \u039a\u03b1\u03c4\u03b1\u03b3\u03c1\u03ac\u03c8\u03c4\u03b5 Python, PyTorch, Transformers, TRL, PEFT, tokenizer revision, model revision, hardware, precision mode, seed \u03ba\u03b1\u03b9 resolved configuration. \u0388\u03bd\u03b1 pipeline \u03c0\u03bf\u03c5 \u03b5\u03c0\u03b9\u03b8\u03b5\u03c9\u03c1\u03b5\u03af signatures \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03c7\u03b5\u03b9\u03c1\u03b9\u03c3\u03c4\u03b5\u03af \u03bc\u03b5\u03c4\u03b1\u03ba\u03b9\u03bd\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03c4\u03c1\u03c9\u03bd \u03bc\u03b5\u03c4\u03b1\u03be\u03cd <code>DPOConfig<\/code> \u03ba\u03b1\u03b9 <code>DPOTrainer<\/code>, \u03b1\u03bb\u03bb\u03ac \u03b4\u03b5\u03bd \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03ba\u03c1\u03cd\u03b2\u03b5\u03b9 \u03c4\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03b1\u03c0\u03cc \u03c4\u03bf run manifest.<\/p>\n<h2 id=\"metrics-by-slice\">Reward accuracy \u03ba\u03b1\u03b9 margins \u03b1\u03bd\u03ac slice<\/h2>\n<p>\u039c\u03b5\u03c4\u03ac \u03c4\u03b7\u03bd \u03b5\u03ba\u03c0\u03b1\u03af\u03b4\u03b5\u03c5\u03c3\u03b7, \u03c4\u03bf held-out evaluation \u03c0\u03b1\u03c1\u03b1\u03ba\u03bf\u03bb\u03bf\u03c5\u03b8\u03b5\u03af loss, reward accuracy \u03ba\u03b1\u03b9 reward margins. \u0393\u03b9\u03b1 \u03ba\u03ac\u03b8\u03b5 pair, \u03c4\u03bf DPO margin \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03b9 \u03c4\u03b7 \u03bc\u03b5\u03c4\u03b1\u03ba\u03af\u03bd\u03b7\u03c3\u03b7 \u03c4\u03b7\u03c2 policy \u03c9\u03c2 \u03c0\u03c1\u03bf\u03c2 \u03c4\u03b7 reference policy \u03b3\u03b9\u03b1 chosen \u03ba\u03b1\u03b9 rejected completions. \u0398\u03b5\u03c4\u03b9\u03ba\u03cc margin \u03b4\u03b5\u03af\u03c7\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03b7 policy \u03bc\u03b5\u03c4\u03b1\u03ba\u03b9\u03bd\u03ae\u03b8\u03b7\u03ba\u03b5 \u03c0\u03c1\u03bf\u03c2 \u03c4\u03b7 chosen \u03c0\u03bb\u03b5\u03c5\u03c1\u03ac\u00b7 \u03b4\u03b5\u03bd \u03b1\u03c0\u03bf\u03b4\u03b5\u03b9\u03ba\u03bd\u03cd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03b7 \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7 \u03ad\u03b3\u03b9\u03bd\u03b5 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ad\u03c3\u03c4\u03b5\u03c1\u03b7, \u03b1\u03c3\u03c6\u03b1\u03bb\u03ad\u03c3\u03c4\u03b5\u03c1\u03b7 \u03ae \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b7 \u03b3\u03b9\u03b1 \u03c4\u03bf\u03bd \u03c0\u03b5\u03bb\u03ac\u03c4\u03b7.<\/p>\n<p>\u0397 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c3\u03c5\u03bd\u03b4\u03ad\u03c3\u03b5\u03b9 \u03c4\u03b1 metrics \u03bc\u03b5 slices \u03c0\u03bf\u03c5 \u03b1\u03bd\u03c4\u03b9\u03c3\u03c4\u03bf\u03b9\u03c7\u03bf\u03cd\u03bd \u03c3\u03c4\u03bf \u03c0\u03c1\u03bf\u03ca\u03cc\u03bd: \u03b5\u03c0\u03b9\u03c3\u03c4\u03c1\u03bf\u03c6\u03ad\u03c2, \u03c4\u03b5\u03c7\u03bd\u03b9\u03ba\u03ae \u03c5\u03c0\u03bf\u03c3\u03c4\u03ae\u03c1\u03b9\u03be\u03b7, catalog questions, policy requests, \u03b5\u03c0\u03b9\u03ba\u03af\u03bd\u03b4\u03c5\u03bd\u03b5\u03c2 \u03c0\u03c1\u03bf\u03c4\u03c1\u03bf\u03c0\u03ad\u03c2, \u03b5\u03bb\u03bb\u03b7\u03bd\u03b9\u03ba\u03ac \u03ba\u03b1\u03b9 \u03b1\u03b3\u03b3\u03bb\u03b9\u03ba\u03ac. \u0397 \u03af\u03b4\u03b9\u03b1 \u03b1\u03c1\u03c7\u03ae \u03b2\u03c1\u03af\u03c3\u03ba\u03b5\u03c4\u03b1\u03b9 \u03c0\u03af\u03c3\u03c9 \u03b1\u03c0\u03cc \u03c4\u03b7\u03bd <a href=\"https:\/\/twodots.gr\/ai-reasoning-kanones-egkyrotita-orthotita\/\">\u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7 AI reasoning \u03bc\u03b5 \u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03b5\u03b3\u03ba\u03c5\u03c1\u03cc\u03c4\u03b7\u03c4\u03b1 \u03ba\u03b1\u03b9 \u03bf\u03c1\u03b8\u03cc\u03c4\u03b7\u03c4\u03b1<\/a>: \u03ad\u03bd\u03b1 proxy metric \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03cc\u03c4\u03b1\u03bd \u03b4\u03b5\u03bd \u03bc\u03b5\u03c4\u03c1\u03ac \u03c4\u03b7\u03bd \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b7\u03c3\u03b9\u03b1\u03ba\u03ae \u03b9\u03b4\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c0\u03bf\u03c5 \u03bc\u03b1\u03c2 \u03b5\u03bd\u03b4\u03b9\u03b1\u03c6\u03ad\u03c1\u03b5\u03b9.<\/p>\n<p>\u0388\u03bd\u03b1 \u03c7\u03c1\u03ae\u03c3\u03b9\u03bc\u03bf diagnostic \u03b5\u03be\u03b5\u03c4\u03ac\u03b6\u03b5\u03b9 \u03c0\u03cc\u03c3\u03bf \u03c3\u03c5\u03c7\u03bd\u03ac \u03b7 \u03c3\u03c9\u03c3\u03c4\u03ae \u03ba\u03b1\u03c4\u03ac\u03c4\u03b1\u03be\u03b7 \u03c3\u03c5\u03bc\u03c0\u03af\u03c0\u03c4\u03b5\u03b9 \u03bc\u03b5 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf chosen. \u03a4\u03b9\u03bc\u03ae \u03ba\u03bf\u03bd\u03c4\u03ac \u03c3\u03c4\u03bf 0,5 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03ba\u03b1\u03b8\u03b7\u03c3\u03c5\u03c7\u03b1\u03c3\u03c4\u03b9\u03ba\u03ae \u03bc\u03cc\u03bd\u03bf \u03bc\u03ad\u03c3\u03b1 \u03c3\u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf balanced test\u00b7 \u03c4\u03b9\u03bc\u03ae \u03ba\u03bf\u03bd\u03c4\u03ac \u03c3\u03c4\u03bf 1 \u03b1\u03c0\u03b1\u03b9\u03c4\u03b5\u03af \u03ad\u03c1\u03b5\u03c5\u03bd\u03b1. \u039a\u03b1\u03bc\u03af\u03b1 \u03b1\u03c0\u03cc \u03c4\u03b9\u03c2 \u03b4\u03cd\u03bf \u03b4\u03b5\u03bd \u03b1\u03c0\u03bf\u03b4\u03b5\u03b9\u03ba\u03bd\u03cd\u03b5\u03b9 \u03b1\u03b9\u03c4\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c7\u03c9\u03c1\u03af\u03c2 length-controlled evaluation \u03ba\u03b1\u03b9 \u03c0\u03bf\u03b9\u03bf\u03c4\u03b9\u03ba\u03ae \u03b5\u03be\u03ad\u03c4\u03b1\u03c3\u03b7 \u03c4\u03c9\u03bd completions.<\/p>\n<div class=\"td-decision-band\">\n<div class=\"td-decision-band-content\"><strong>Release \u03bc\u03cc\u03bd\u03bf \u03cc\u03c4\u03b1\u03bd \u03b7 \u03b2\u03b5\u03bb\u03c4\u03af\u03c9\u03c3\u03b7 \u03b5\u03c0\u03b9\u03b2\u03b9\u03ce\u03bd\u03b5\u03b9 \u03c7\u03c9\u03c1\u03af\u03c2 \u03c4\u03bf shortcut.<\/strong><span>\u039f\u03c1\u03af\u03c3\u03c4\u03b5 \u03c0\u03c1\u03b9\u03bd \u03c4\u03bf training \u03cc\u03c1\u03b9\u03b1 \u03b1\u03bd\u03ac subset, \u03b3\u03bb\u03ce\u03c3\u03c3\u03b1, intent \u03ba\u03b1\u03b9 risk class\u00b7 \u03b1\u03c0\u03b1\u03b9\u03c4\u03ae\u03c3\u03c4\u03b5 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc margin \u03c3\u03b5 length-controlled pairs, \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03ba\u03b1\u03bc\u03af\u03b1 \u03ba\u03c1\u03af\u03c3\u03b9\u03bc\u03b7 \u03c5\u03c0\u03bf\u03c7\u03ce\u03c1\u03b7\u03c3\u03b7 \u03c0\u03af\u03c3\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf\u03bd aggregate \u03bc\u03ad\u03c3\u03bf \u03cc\u03c1\u03bf.<\/span><\/div>\n<\/div>\n<h2 id=\"smoke-test-not-benchmark\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03b1 30 steps \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 benchmark<\/h2>\n<p>\u0397 \u03b5\u03ba\u03c4\u03ad\u03bb\u03b5\u03c3\u03b7 30 steps \u03c3\u03b5 CPU \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03b9\u03ce\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 data loading, parsing, tokenization, training, evaluation \u03ba\u03b1\u03b9 saving \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03bf\u03bd\u03c4\u03b1\u03b9. \u0394\u03b5\u03bd \u03c0\u03b1\u03c1\u03ac\u03b3\u03b5\u03b9 production model \u03bf\u03cd\u03c4\u03b5 \u03b1\u03c0\u03bf\u03b4\u03b5\u03b9\u03ba\u03bd\u03cd\u03b5\u03b9 \u03c4\u03b7\u03bd \u03c5\u03c0\u03b5\u03c1\u03bf\u03c7\u03ae \u03c4\u03bf\u03c5 DPO, \u03c4\u03bf\u03c5 Qwen2.5-0.5B-Instruct \u03ae \u03c4\u03bf\u03c5 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf\u03c5 dataset.<\/p>\n<p>Reward accuracy \u03ba\u03bf\u03bd\u03c4\u03ac \u03c3\u03c4\u03bf 0,5 \u03c3\u03b5 \u03c4\u03cc\u03c3\u03bf \u03c3\u03cd\u03bd\u03c4\u03bf\u03bc\u03bf run \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03b1\u03c0\u03bb\u03ce\u03c2 \u03bd\u03b1 \u03b4\u03b5\u03af\u03c7\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03b7 policy \u03b4\u03b5\u03bd \u03b5\u03af\u03c7\u03b5 \u03c7\u03c1\u03cc\u03bd\u03bf \u03bd\u03b1 \u03bc\u03b5\u03c4\u03b1\u03ba\u03b9\u03bd\u03b7\u03b8\u03b5\u03af. \u03a0\u03b5\u03c1\u03b9\u03c3\u03c3\u03cc\u03c4\u03b5\u03c1\u03b1 steps \u03c3\u03b5 GPU \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b1\u03bb\u03bb\u03ac\u03be\u03bf\u03c5\u03bd \u03c4\u03bf \u03b1\u03c0\u03bf\u03c4\u03ad\u03bb\u03b5\u03c3\u03bc\u03b1, \u03b1\u03bb\u03bb\u03ac \u03ba\u03b1\u03b9 \u03c0\u03ac\u03bb\u03b9 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03b5\u03c0\u03b1\u03bd\u03b1\u03bb\u03ae\u03c8\u03b5\u03b9\u03c2, \u03c0\u03c1\u03bf\u03ba\u03b1\u03b8\u03bf\u03c1\u03b9\u03c3\u03bc\u03ad\u03bd\u03b1 acceptance criteria, uncertainty \u03cc\u03c0\u03bf\u03c5 \u03c4\u03b1\u03b9\u03c1\u03b9\u03ac\u03b6\u03b5\u03b9 \u03ba\u03b1\u03b9 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b7 \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7.<\/p>\n<p>\u0397 \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b7\u03bc\u03b1\u03c4\u03b9\u03ba\u03ae \u03b1\u03bd\u03b1\u03c6\u03bf\u03c1\u03ac \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03be\u03b5\u03c7\u03c9\u03c1\u03af\u03b6\u03b5\u03b9 \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b9\u03c2 \u03ba\u03b1\u03c4\u03b1\u03c3\u03c4\u03ac\u03c3\u03b5\u03b9\u03c2: pipeline pass, training convergence, offline quality pass \u03ba\u03b1\u03b9 production readiness. \u0397 \u03c0\u03c1\u03ce\u03c4\u03b7 \u03b4\u03b5\u03bd \u03c3\u03c5\u03bd\u03b5\u03c0\u03ac\u03b3\u03b5\u03c4\u03b1\u03b9 \u03c4\u03b9\u03c2 \u03ac\u03bb\u03bb\u03b5\u03c2 \u03c4\u03c1\u03b5\u03b9\u03c2. \u0391\u03c5\u03c4\u03ae \u03b7 \u03b4\u03b9\u03ac\u03ba\u03c1\u03b9\u03c3\u03b7 \u03c0\u03c1\u03bf\u03c3\u03c4\u03b1\u03c4\u03b5\u03cd\u03b5\u03b9 stakeholders \u03b1\u03c0\u03cc \u03b5\u03bd\u03c4\u03c5\u03c0\u03c9\u03c3\u03b9\u03b1\u03ba\u03ac demos \u03c0\u03bf\u03c5 \u03b4\u03b5\u03bd \u03ad\u03c7\u03bf\u03c5\u03bd \u03b1\u03ba\u03cc\u03bc\u03b7 \u03b1\u03c0\u03bf\u03b4\u03b5\u03af\u03be\u03b5\u03b9 \u03b1\u03be\u03af\u03b1 \u03ae \u03b1\u03c3\u03c6\u03ac\u03bb\u03b5\u03b9\u03b1.<\/p>\n<h2 id=\"seven-step-preference-audit\">Preference data audit \u03c3\u03b5 \u03b5\u03c0\u03c4\u03ac \u03b2\u03ae\u03bc\u03b1\u03c4\u03b1<\/h2>\n<p>\u03a4\u03bf audit \u03b3\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c0\u03b9\u03bf \u03b1\u03be\u03b9\u03cc\u03c0\u03b9\u03c3\u03c4\u03bf \u03cc\u03c4\u03b1\u03bd \u03ba\u03ac\u03b8\u03b5 \u03b2\u03ae\u03bc\u03b1 \u03c0\u03b1\u03c1\u03ac\u03b3\u03b5\u03b9 \u03c4\u03b5\u03ba\u03bc\u03ae\u03c1\u03b9\u03bf \u03c0\u03bf\u03c5 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03c0\u03b1\u03bd\u03b1\u03bb\u03b7\u03c6\u03b8\u03b5\u03af \u03c3\u03c4\u03bf \u03b5\u03c0\u03cc\u03bc\u03b5\u03bd\u03bf dataset version. \u0397 <a href=\"https:\/\/twodots.gr\/ai-agents-confidence-context-data-human-oversight\/\">\u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03b5\u03c0\u03bf\u03c0\u03c4\u03b5\u03af\u03b1 \u03c3\u03b5 AI workflows<\/a> \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c4\u03b1 \u03af\u03b4\u03b9\u03b1 \u03ba\u03b1\u03b8\u03b1\u03c1\u03ac \u03c3\u03b7\u03bc\u03b5\u03af\u03b1 \u03b5\u03bb\u03ad\u03b3\u03c7\u03bf\u03c5: \u03b3\u03bd\u03c9\u03c3\u03c4\u03cc input, \u03c5\u03c0\u03b5\u03cd\u03b8\u03c5\u03bd\u03bf owner \u03ba\u03b1\u03b9 \u03b1\u03c0\u03cc\u03c6\u03b1\u03c3\u03b7 \u03c0\u03bf\u03c5 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b1\u03bd\u03b1\u03c4\u03c1\u03b1\u03c0\u03b5\u03af.<\/p>\n<div class=\"td-step-list\">\n<p class=\"td-step-list-title\">\u0391\u03c0\u03cc \u03c4\u03bf\u03bd \u03bf\u03c1\u03b9\u03c3\u03bc\u03cc \u03c4\u03b7\u03c2 \u03c0\u03c1\u03bf\u03c4\u03af\u03bc\u03b7\u03c3\u03b7\u03c2 \u03ad\u03c9\u03c2 \u03c4\u03bf release gate<\/p>\n<ol>\n<li><span class=\"td-step-kicker\">\u0392\u03ae\u03bc\u03b1 1<\/span><strong>\u039f\u03c1\u03af\u03c3\u03c4\u03b5 \u03c4\u03b9 \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u00ab\u03c0\u03c1\u03bf\u03c4\u03b9\u03bc\u03ce\u03bc\u03b5\u03bd\u03b7\u00bb \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7<\/strong>\n<p>\u03a7\u03c9\u03c1\u03af\u03c3\u03c4\u03b5 \u03b1\u03ba\u03c1\u03af\u03b2\u03b5\u03b9\u03b1, \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03cc\u03c4\u03b7\u03c4\u03b1, \u03b1\u03c3\u03c6\u03ac\u03bb\u03b5\u03b9\u03b1, \u03cd\u03c6\u03bf\u03c2, \u03ad\u03ba\u03c4\u03b1\u03c3\u03b7 \u03ba\u03b1\u03b9 escalation. \u039c\u03af\u03b1 \u03b1\u03c3\u03b1\u03c6\u03ae\u03c2 \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1 \u03b4\u03b5\u03bd \u03bb\u03ad\u03b5\u03b9 \u03c3\u03c4\u03bf \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03c0\u03bf\u03b9\u03bf trade-off \u03c0\u03c1\u03bf\u03ad\u03c7\u03b5\u03b9.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0392\u03ae\u03bc\u03b1 2<\/span><strong>\u0395\u03c0\u03b1\u03bb\u03b7\u03b8\u03b5\u03cd\u03c3\u03c4\u03b5 schema, \u03c1\u03cc\u03bb\u03bf\u03c5\u03c2 \u03ba\u03b1\u03b9 \u03ba\u03bf\u03b9\u03bd\u03cc prompt<\/strong>\n<p>\u0391\u03c0\u03bf\u03c1\u03c1\u03af\u03c8\u03c4\u03b5 \u03ba\u03b5\u03bd\u03ac turns, \u03bb\u03ac\u03b8\u03bf\u03c2 \u03b5\u03bd\u03b1\u03bb\u03bb\u03b1\u03b3\u03ae \u03c1\u03cc\u03bb\u03c9\u03bd, prefix mismatch, duplicates \u03ba\u03b1\u03b9 pairs \u03cc\u03c0\u03bf\u03c5 \u03b1\u03bb\u03bb\u03ac\u03b6\u03b5\u03b9 \u03bc\u03b1\u03b6\u03af \u03c4\u03bf context \u03ba\u03b1\u03b9 \u03b7 completion.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0392\u03ae\u03bc\u03b1 3<\/span><strong>\u039c\u03b5\u03c4\u03c1\u03ae\u03c3\u03c4\u03b5 length bias \u03b1\u03bd\u03ac slice<\/strong>\n<p>\u03a5\u03c0\u03bf\u03bb\u03bf\u03b3\u03af\u03c3\u03c4\u03b5 token deltas, \u03c0\u03bf\u03c3\u03bf\u03c3\u03c4\u03cc \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf\u03c5 chosen \u03ba\u03b1\u03b9 \u03b5\u03c0\u03af\u03b4\u03bf\u03c3\u03b7 \u03c3\u03b5 length-controlled buckets \u03b3\u03b9\u03b1 \u03ba\u03ac\u03b8\u03b5 subset, \u03b3\u03bb\u03ce\u03c3\u03c3\u03b1 \u03ba\u03b1\u03b9 intent.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0392\u03ae\u03bc\u03b1 4<\/span><strong>\u0394\u03bf\u03ba\u03b9\u03bc\u03ac\u03c3\u03c4\u03b5 lexical shortcuts \u03c7\u03c9\u03c1\u03af\u03c2 leakage<\/strong>\n<p>\u039a\u03ac\u03bd\u03c4\u03b5 split \u03b1\u03bd\u03ac pair ID, \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03c4\u03b5 TF-IDF classifier \u03bc\u03b5 permuted labels \u03ba\u03b1\u03b9 \u03c0\u03c1\u03bf\u03c3\u03c4\u03b1\u03c4\u03b5\u03cd\u03c3\u03c4\u03b5 \u03b5\u03c5\u03b1\u03af\u03c3\u03b8\u03b7\u03c4\u03b1 n-grams \u03b1\u03c0\u03cc \u03c0\u03b5\u03c1\u03b9\u03c4\u03c4\u03ae \u03ad\u03ba\u03b8\u03b5\u03c3\u03b7 \u03c3\u03c4\u03b1 logs.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0392\u03ae\u03bc\u03b1 5<\/span><strong>\u03a6\u03b9\u03bb\u03c4\u03c1\u03ac\u03c1\u03b5\u03c4\u03b5 \u03bc\u03b5 \u03c4\u03bf\u03bd \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc tokenizer<\/strong>\n<p>\u0395\u03c6\u03b1\u03c1\u03bc\u03cc\u03c3\u03c4\u03b5 \u03c4\u03bf versioned chat template, \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03c4\u03b5 prompt \u03ba\u03b1\u03b9 completions \u03ba\u03b1\u03b9 \u03b5\u03bb\u03ad\u03b3\u03be\u03c4\u03b5 \u03c0\u03bf\u03b9\u03b1 \u03b3\u03bb\u03ce\u03c3\u03c3\u03b1 \u03ae \u03ba\u03b1\u03c4\u03b7\u03b3\u03bf\u03c1\u03af\u03b1 \u03c7\u03ac\u03bd\u03b5\u03b9 \u03b4\u03c5\u03c3\u03b1\u03bd\u03ac\u03bb\u03bf\u03b3\u03b1 \u03b4\u03b5\u03af\u03b3\u03bc\u03b1\u03c4\u03b1.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0392\u03ae\u03bc\u03b1 6<\/span><strong>\u039a\u03bb\u03b5\u03b9\u03b4\u03ce\u03c3\u03c4\u03b5 manifest \u03ba\u03b1\u03b9 reproducible configuration<\/strong>\n<p>\u0391\u03c0\u03bf\u03b8\u03b7\u03ba\u03b5\u03cd\u03c3\u03c4\u03b5 dataset hash, model \u03ba\u03b1\u03b9 tokenizer revision, \u03b2\u03b9\u03b2\u03bb\u03b9\u03bf\u03b8\u03ae\u03ba\u03b5\u03c2, seed, hardware, precision, LoRA \u03ba\u03b1\u03b9 resolved DPO arguments \u03b3\u03b9\u03b1 \u03ba\u03ac\u03b8\u03b5 run.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0392\u03ae\u03bc\u03b1 7<\/span><strong>\u0391\u03be\u03b9\u03bf\u03bb\u03bf\u03b3\u03ae\u03c3\u03c4\u03b5 \u03b1\u03bd\u03ac use case \u03c0\u03c1\u03b9\u03bd \u03b1\u03c0\u03cc release<\/strong>\n<p>\u03a3\u03c5\u03bd\u03b4\u03c5\u03ac\u03c3\u03c4\u03b5 reward metrics, length-controlled tests, \u03c0\u03bf\u03b9\u03bf\u03c4\u03b9\u03ba\u03ad\u03c2 \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03c3\u03b5\u03b9\u03c2, safety evals \u03ba\u03b1\u03b9 \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7\u00b7 \u03bc\u03c0\u03bb\u03bf\u03ba\u03ac\u03c1\u03b5\u03c4\u03b5 \u03c4\u03b7 \u03b4\u03b9\u03ac\u03b8\u03b5\u03c3\u03b7 \u03b1\u03bd \u03c5\u03c0\u03bf\u03c7\u03c9\u03c1\u03b5\u03af \u03ba\u03c1\u03af\u03c3\u03b9\u03bc\u03bf slice.<\/p>\n<\/li>\n<\/ol>\n<\/div>\n<h2 id=\"production-eval-contract\">\u0391\u03c0\u03cc \u03c4\u03bf notebook \u03c3\u03b5 production eval contract<\/h2>\n<p>\u0388\u03bd\u03b1 production workflow \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 versioned datasets, immutable manifests, hashes, reason-coded filtering, \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ac safety evals \u03ba\u03b1\u03b9 monitoring \u03bc\u03b5\u03c4\u03ac \u03c4\u03b7 \u03b4\u03b9\u03ac\u03b8\u03b5\u03c3\u03b7. \u039a\u03ac\u03b8\u03b5 checkpoint \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c3\u03c5\u03bd\u03b4\u03ad\u03b5\u03c4\u03b1\u03b9 \u03bc\u03b5 \u03c4\u03b7\u03bd \u03b1\u03ba\u03c1\u03b9\u03b2\u03ae \u03ad\u03ba\u03b4\u03bf\u03c3\u03b7 \u03b4\u03b5\u03b4\u03bf\u03bc\u03ad\u03bd\u03c9\u03bd, \u03c4\u03bf\u03bd \u03ba\u03ce\u03b4\u03b9\u03ba\u03b1 preprocessing, \u03c4\u03b1 acceptance criteria \u03ba\u03b1\u03b9 \u03c4\u03b7\u03bd \u03b1\u03c0\u03cc\u03c6\u03b1\u03c3\u03b7 \u03ad\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7\u03c2.<\/p>\n<p>\u0393\u03b9\u03b1 marketing, e-commerce \u03ba\u03b1\u03b9 customer support, \u03b7 \u03ad\u03bd\u03bd\u03bf\u03b9\u03b1 \u03c4\u03b7\u03c2 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03bc\u03b5\u03c4\u03b1\u03c6\u03c1\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 observable outcomes: \u03c0\u03b9\u03c3\u03c4\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c3\u03b5 catalog \u03ba\u03b1\u03b9 policy data, \u03c3\u03c9\u03c3\u03c4\u03ae \u03ba\u03bb\u03b9\u03bc\u03ac\u03ba\u03c9\u03c3\u03b7 \u03c3\u03b5 \u03ac\u03bd\u03b8\u03c1\u03c9\u03c0\u03bf, \u03b1\u03c0\u03bf\u03c6\u03c5\u03b3\u03ae \u03bc\u03b7 \u03c5\u03c0\u03bf\u03c3\u03c4\u03b7\u03c1\u03b9\u03b6\u03cc\u03bc\u03b5\u03bd\u03c9\u03bd \u03c5\u03c0\u03bf\u03c3\u03c7\u03ad\u03c3\u03b5\u03c9\u03bd, \u03ba\u03b1\u03c4\u03ac\u03bb\u03bb\u03b7\u03bb\u03b7 \u03ad\u03ba\u03c4\u03b1\u03c3\u03b7, task completion \u03ba\u03b1\u03b9 \u03b1\u03c3\u03c6\u03b1\u03bb\u03ae\u03c2 \u03ac\u03c1\u03bd\u03b7\u03c3\u03b7. \u03a4\u03bf deployment monitoring \u03b5\u03bb\u03ad\u03b3\u03c7\u03b5\u03b9 drift \u03c3\u03b5 \u03b1\u03c5\u03c4\u03ac \u03c4\u03b1 outcomes, \u03cc\u03c7\u03b9 \u03bc\u03cc\u03bd\u03bf \u03c4\u03b5\u03c7\u03bd\u03b9\u03ba\u03ae \u03b4\u03b9\u03b1\u03b8\u03b5\u03c3\u03b9\u03bc\u03cc\u03c4\u03b7\u03c4\u03b1.<\/p>\n<p>\u03a4\u03bf NIST Generative AI Profile \u03b1\u03bd\u03c4\u03b9\u03bc\u03b5\u03c4\u03c9\u03c0\u03af\u03b6\u03b5\u03b9 \u03c4\u03b7 \u03b4\u03b9\u03b1\u03c7\u03b5\u03af\u03c1\u03b9\u03c3\u03b7 \u03ba\u03b9\u03bd\u03b4\u03cd\u03bd\u03bf\u03c5 \u03c3\u03b5 \u03cc\u03bb\u03bf \u03c4\u03bf\u03bd \u03ba\u03cd\u03ba\u03bb\u03bf \u03b6\u03c9\u03ae\u03c2 \u03c3\u03c7\u03b5\u03b4\u03b9\u03b1\u03c3\u03bc\u03bf\u03cd, \u03b1\u03bd\u03ac\u03c0\u03c4\u03c5\u03be\u03b7\u03c2, \u03c7\u03c1\u03ae\u03c3\u03b7\u03c2 \u03ba\u03b1\u03b9 \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7\u03c2. \u03a3\u03c4\u03b7\u03bd \u03c0\u03c1\u03ac\u03be\u03b7, \u03b1\u03c5\u03c4\u03cc \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03c4\u03bf preference dataset \u03b5\u03af\u03bd\u03b1\u03b9 governed asset: \u03ad\u03c7\u03b5\u03b9 owner, provenance, \u03b5\u03c0\u03b9\u03c4\u03c1\u03b5\u03c0\u03cc\u03bc\u03b5\u03bd\u03b5\u03c2 \u03c7\u03c1\u03ae\u03c3\u03b5\u03b9\u03c2, quality gates \u03ba\u03b1\u03b9 \u03b4\u03b9\u03b1\u03b4\u03b9\u03ba\u03b1\u03c3\u03af\u03b1 \u03b1\u03c0\u03cc\u03c3\u03c5\u03c1\u03c3\u03b7\u03c2. \u03a0\u03b1\u03c1\u03cc\u03bc\u03bf\u03b9\u03b1, \u03b7 <a href=\"https:\/\/twodots.gr\/automatopoiisi-change-management-apodeixeis-riskou\/\">\u03b1\u03c5\u03c4\u03bf\u03bc\u03b1\u03c4\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7 \u03c3\u03c4\u03bf change management \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03b1\u03c0\u03bf\u03b4\u03b5\u03af\u03be\u03b5\u03b9\u03c2 \u03b1\u03bd\u03ac\u03bb\u03bf\u03b3\u03b5\u03c2 \u03c4\u03bf\u03c5 \u03c1\u03af\u03c3\u03ba\u03bf\u03c5<\/a>, \u03cc\u03c7\u03b9 \u03c4\u03bf \u03af\u03b4\u03b9\u03bf \u03c4\u03c5\u03c0\u03b9\u03ba\u03cc review \u03b3\u03b9\u03b1 \u03ba\u03ac\u03b8\u03b5 \u03b1\u03bb\u03bb\u03b1\u03b3\u03ae.<\/p>\n<p>\u03a4\u03bf \u03ba\u03b5\u03bd\u03c4\u03c1\u03b9\u03ba\u03cc \u03bc\u03ac\u03b8\u03b7\u03bc\u03b1 \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03b5\u03b9 \u03b1\u03c0\u03bb\u03cc: \u03c4\u03bf DPO \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03c4\u03bf \u03c3\u03ae\u03bc\u03b1 \u03c0\u03bf\u03c5 \u03c4\u03bf\u03c5 \u03b4\u03af\u03bd\u03b5\u03c4\u03b5, \u03cc\u03c7\u03b9 \u03b5\u03ba\u03b5\u03af\u03bd\u03bf \u03c0\u03bf\u03c5 \u03b8\u03b1 \u03b8\u03ad\u03bb\u03b1\u03c4\u03b5 \u03bd\u03b1 \u03b5\u03af\u03c7\u03b1\u03c4\u03b5 \u03c3\u03c5\u03bb\u03bb\u03ad\u03be\u03b5\u03b9. \u0388\u03bd\u03b1 \u03ba\u03b1\u03bb\u03cc audit \u03ba\u03ac\u03bd\u03b5\u03b9 \u03c4\u03bf \u03c3\u03ae\u03bc\u03b1 \u03b5\u03bb\u03ad\u03b3\u03be\u03b9\u03bc\u03bf \u03c0\u03c1\u03b9\u03bd \u03b3\u03af\u03bd\u03b5\u03b9 \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5.<\/p>\n<section class=\"td-service-cta\">\n<div class=\"td-service-cta-content\">\n<p class=\"td-service-cta-kicker\">\u0391\u03c5\u03c4\u03bf\u03bc\u03b1\u03c4\u03b9\u03c3\u03bc\u03bf\u03af \u0395\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03ae\u03c3\u03b5\u03c9\u03bd &amp; AI<\/p>\n<h3>\u03a3\u03c7\u03b5\u03b4\u03b9\u03ac\u03c3\u03c4\u03b5 AI workflows \u03bc\u03b5 \u03b5\u03bb\u03ad\u03b3\u03be\u03b9\u03bc\u03b1 \u03b4\u03b5\u03b4\u03bf\u03bc\u03ad\u03bd\u03b1 \u03ba\u03b1\u03b9 \u03c3\u03b1\u03c6\u03ae release gates<\/h3>\n<p>\u0397 TWO DOTS \u03c7\u03b1\u03c1\u03c4\u03bf\u03b3\u03c1\u03b1\u03c6\u03b5\u03af \u03b4\u03b5\u03b4\u03bf\u03bc\u03ad\u03bd\u03b1, integrations, \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b5\u03c2 \u03b5\u03b3\u03ba\u03c1\u03af\u03c3\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c0\u03c1\u03b9\u03bd \u03ad\u03bd\u03b1 AI workflow \u03c3\u03c5\u03bd\u03b4\u03b5\u03b8\u03b5\u03af \u03bc\u03b5 CRM, e-shop \u03ae customer support. \u0388\u03c4\u03c3\u03b9 \u03c4\u03bf production \u03b1\u03c0\u03bf\u03c4\u03ad\u03bb\u03b5\u03c3\u03bc\u03b1 \u03c3\u03c4\u03b7\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ac use cases \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03bc\u03cc\u03bd\u03bf \u03c3\u03b5 \u03ad\u03bd\u03b1 \u03ba\u03b1\u03bb\u03cc demo.<\/p>\n<div class=\"td-service-cta-actions\"><a class=\"td-service-cta-button\" href=\"https:\/\/twodots.gr\/aftomatismoi-epicheiriseon-ai\/\">\u0394\u03b5\u03af\u03c4\u03b5 \u03c4\u03bf\u03c5\u03c2 \u0391\u03c5\u03c4\u03bf\u03bc\u03b1\u03c4\u03b9\u03c3\u03bc\u03bf\u03cd\u03c2 \u0395\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03ae\u03c3\u03b5\u03c9\u03bd &amp; AI<\/a><\/div>\n<\/div>\n<\/section>\n<section id=\"sychnes-erotiseis\" class=\"td-faq-section\">\n<div class=\"td-faq\">\n<p class=\"td-faq-heading\">\u03a3\u03c5\u03c7\u03bd\u03ad\u03c2 \u03b5\u03c1\u03c9\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2<\/p>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a4\u03b9 \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03bf Direct Preference Optimization;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a4\u03bf DPO \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03ad\u03b8\u03bf\u03b4\u03bf\u03c2 \u03c0\u03c1\u03bf\u03c3\u03b1\u03c1\u03bc\u03bf\u03b3\u03ae\u03c2 \u03bc\u03b9\u03b1\u03c2 policy \u03c0\u03ac\u03bd\u03c9 \u03c3\u03b5 \u03b6\u03b5\u03cd\u03b3\u03b7 \u03c0\u03c1\u03bf\u03c4\u03b9\u03bc\u03ce\u03bc\u03b5\u03bd\u03c9\u03bd \u03ba\u03b1\u03b9 \u03b1\u03c0\u03bf\u03c1\u03c1\u03b9\u03c6\u03b8\u03b5\u03b9\u03c3\u03ce\u03bd \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03c9\u03bd \u03c9\u03c2 \u03c0\u03c1\u03bf\u03c2 \u03bc\u03b9\u03b1 reference policy, \u03c7\u03c9\u03c1\u03af\u03c2 \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03b5\u03ba\u03c0\u03b1\u03af\u03b4\u03b5\u03c5\u03c3\u03b7 reward model \u03ba\u03b1\u03b9 \u03c7\u03c9\u03c1\u03af\u03c2 \u03c4\u03bf\u03bd \u03ba\u03bb\u03b1\u03c3\u03b9\u03ba\u03cc \u03b2\u03c1\u03cc\u03c7\u03bf reinforcement learning \u03c4\u03bf\u03c5 PPO.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf preference data audit \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c0\u03c1\u03bf\u03b7\u03b3\u03b5\u03af\u03c4\u03b1\u03b9 \u03c4\u03bf\u03c5 DPO;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u0395\u03c0\u03b5\u03b9\u03b4\u03ae \u03c4\u03bf DPO \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03c4\u03bf \u03c3\u03ae\u03bc\u03b1 \u03c0\u03bf\u03c5 \u03c5\u03c0\u03ac\u03c1\u03c7\u03b5\u03b9 \u03c3\u03c4\u03b1 pairs. \u0391\u03bd \u03b7 \u03b5\u03c4\u03b9\u03ba\u03ad\u03c4\u03b1 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c0\u03b5\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03bc\u03ae\u03ba\u03bf\u03c2, \u03bb\u03b5\u03be\u03b9\u03bb\u03cc\u03b3\u03b9\u03bf, \u03bb\u03ac\u03b8\u03bf\u03c2 conversational prefix \u03ae \u03ac\u03bd\u03b9\u03c3\u03b7 \u03ba\u03ac\u03bb\u03c5\u03c8\u03b7 use cases, \u03b7 policy \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03bc\u03ac\u03b8\u03b5\u03b9 shortcut \u03b1\u03bd\u03c4\u03af \u03b3\u03b9\u03b1 \u03c4\u03b7\u03bd \u03b5\u03c0\u03b9\u03b8\u03c5\u03bc\u03b7\u03c4\u03ae \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0393\u03b9\u03b1\u03c4\u03af chosen \u03ba\u03b1\u03b9 rejected \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03ad\u03c7\u03bf\u03c5\u03bd \u03c4\u03bf \u03af\u03b4\u03b9\u03bf prompt;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u0393\u03b9\u03b1 \u03bd\u03b1 \u03b1\u03c0\u03bf\u03bc\u03bf\u03bd\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03c4\u03b7\u03c2 \u03c4\u03b5\u03bb\u03b9\u03ba\u03ae\u03c2 \u03b1\u03c0\u03ac\u03bd\u03c4\u03b7\u03c3\u03b7\u03c2. \u0391\u03bd \u03b1\u03bb\u03bb\u03ac\u03b6\u03b5\u03b9 \u03ba\u03b1\u03b9 \u03c4\u03bf prompt \u03ae \u03c4\u03bf \u03c0\u03c1\u03bf\u03b7\u03b3\u03bf\u03cd\u03bc\u03b5\u03bd\u03bf context, \u03bf optimizer \u03b4\u03b5\u03bd \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b3\u03bd\u03c9\u03c1\u03af\u03b6\u03b5\u03b9 \u03b1\u03bd \u03b7 \u03c0\u03c1\u03bf\u03c4\u03af\u03bc\u03b7\u03c3\u03b7 \u03b1\u03c6\u03bf\u03c1\u03ac \u03c4\u03b7\u03bd completion \u03ae \u03c4\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ae \u03c3\u03c5\u03b6\u03ae\u03c4\u03b7\u03c3\u03b7.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a0\u03ce\u03c2 \u03b5\u03bd\u03c4\u03bf\u03c0\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c4\u03bf length bias;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u039c\u03b5 token deltas \u03b1\u03bd\u03ac pair, \u03c0\u03bf\u03c3\u03bf\u03c3\u03c4\u03cc \u03c0\u03b5\u03c1\u03b9\u03c0\u03c4\u03ce\u03c3\u03b5\u03c9\u03bd \u03cc\u03c0\u03bf\u03c5 \u03c4\u03bf chosen \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf, \u03ba\u03b1\u03c4\u03b1\u03bd\u03bf\u03bc\u03ad\u03c2 \u03b1\u03bd\u03ac subset \u03ba\u03b1\u03b9 \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7 \u03c3\u03b5 length-controlled buckets. \u0397 \u03b2\u03b5\u03bb\u03c4\u03af\u03c9\u03c3\u03b7 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b1\u03bd \u03c4\u03bf \u03bc\u03ae\u03ba\u03bf\u03c2 \u03c0\u03b1\u03cd\u03b5\u03b9 \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b5\u03cd\u03ba\u03bf\u03bb\u03bf proxy.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a4\u03b9 \u03b4\u03b5\u03af\u03c7\u03bd\u03b5\u03b9 \u03ad\u03bd\u03b1 TF-IDF lexical diagnostic;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u0394\u03b5\u03af\u03c7\u03bd\u03b5\u03b9 \u03b1\u03bd \u03b1\u03c0\u03bb\u03ad\u03c2 \u03bb\u03ad\u03be\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 \u03c6\u03c1\u03ac\u03c3\u03b5\u03b9\u03c2 \u03bc\u03c0\u03bf\u03c1\u03bf\u03cd\u03bd \u03bd\u03b1 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c8\u03bf\u03c5\u03bd chosen \u03ae rejected. \u0395\u03c0\u03af\u03b4\u03bf\u03c3\u03b7 \u03c0\u03ac\u03bd\u03c9 \u03b1\u03c0\u03cc \u03ad\u03bd\u03b1 pair-safe permutation baseline \u03b5\u03af\u03bd\u03b1\u03b9 \u03ad\u03bd\u03b4\u03b5\u03b9\u03be\u03b7 \u03b5\u03c0\u03b9\u03c6\u03b1\u03bd\u03b5\u03b9\u03b1\u03ba\u03bf\u03cd \u03c3\u03ae\u03bc\u03b1\u03c4\u03bf\u03c2 \u03c0\u03bf\u03c5 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03b4\u03b9\u03b5\u03c1\u03b5\u03cd\u03bd\u03b7\u03c3\u03b7, \u03cc\u03c7\u03b9 \u03b1\u03c0\u03cc\u03b4\u03b5\u03b9\u03be\u03b7 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2 \u03c4\u03bf\u03c5 LLM.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0393\u03b9\u03b1\u03c4\u03af \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 split \u03b1\u03bd\u03ac pair ID;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u0393\u03b9\u03b1 \u03bd\u03b1 \u03bc\u03b7\u03bd \u03c0\u03b5\u03c1\u03ac\u03c3\u03bf\u03c5\u03bd \u03bf\u03b9 \u03b4\u03cd\u03bf \u03c0\u03bb\u03b5\u03c5\u03c1\u03ad\u03c2 \u03c4\u03bf\u03c5 \u03af\u03b4\u03b9\u03bf\u03c5 preference pair \u03c3\u03b5 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac train \u03ba\u03b1\u03b9 test sets. \u0391\u03c5\u03c4\u03ae \u03b7 \u03b4\u03b9\u03b1\u03c1\u03c1\u03bf\u03ae \u03b8\u03b1 \u03ad\u03b4\u03b9\u03bd\u03b5 \u03c5\u03c0\u03b5\u03c1\u03b1\u03b9\u03c3\u03b9\u03cc\u03b4\u03bf\u03be\u03b7 \u03b5\u03b9\u03ba\u03cc\u03bd\u03b1 \u03b3\u03b9\u03b1 \u03c4\u03bf \u03c0\u03cc\u03c3\u03bf \u03ba\u03b1\u03bb\u03ac \u03b3\u03b5\u03bd\u03b9\u03ba\u03b5\u03cd\u03b5\u03b9 \u03c4\u03bf lexical diagnostic.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0395\u03af\u03bd\u03b1\u03b9 \u03b1\u03c1\u03ba\u03b5\u03c4\u03ac \u03c4\u03b1 30 training steps;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u038c\u03c7\u03b9 \u03b3\u03b9\u03b1 \u03c3\u03c5\u03bc\u03c0\u03ad\u03c1\u03b1\u03c3\u03bc\u03b1 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2. \u03a3\u03b5 \u03b1\u03c5\u03c4\u03cc \u03c4\u03bf workflow \u03bb\u03b5\u03b9\u03c4\u03bf\u03c5\u03c1\u03b3\u03bf\u03cd\u03bd \u03c9\u03c2 smoke test \u03c0\u03bf\u03c5 \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03b9\u03ce\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03c4\u03bf pipeline \u03b5\u03ba\u03c4\u03b5\u03bb\u03b5\u03af\u03c4\u03b1\u03b9. Production \u03b1\u03c0\u03cc\u03c6\u03b1\u03c3\u03b7 \u03b1\u03c0\u03b1\u03b9\u03c4\u03b5\u03af \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b1 runs, \u03b5\u03c0\u03b1\u03bd\u03b1\u03bb\u03ae\u03c8\u03b5\u03b9\u03c2, slice-level metrics \u03ba\u03b1\u03b9 \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a0\u03bf\u03b9\u03b1 \u03c3\u03c4\u03bf\u03b9\u03c7\u03b5\u03af\u03b1 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03ad\u03bd\u03b1 production DPO run;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 versioned dataset \u03ba\u03b1\u03b9 chat template, model \u03ba\u03b1\u03b9 tokenizer revisions, hashes, reason-coded filtering, resolved training configuration, metrics \u03b1\u03bd\u03ac use case, safety evals, \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03ad\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03ba\u03b1\u03b9 monitoring \u03bc\u03b5\u03c4\u03ac \u03c4\u03b7 \u03b4\u03b9\u03ac\u03b8\u03b5\u03c3\u03b7.<\/p>\n<\/div>\n<\/details>\n<\/div>\n<\/section>\n<div class=\"td-source-list\">\n<p id=\"piges\" class=\"td-source-list-title\">\u03a0\u03b7\u03b3\u03ad\u03c2<\/p>\n<ul>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2305.18290\" target=\"_blank\" rel=\"noopener\">Rafailov et al. \u2014 Direct Preference Optimization: Your Language Model is Secretly a Reward Model<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/anthropics\/hh-rlhf\" target=\"_blank\" rel=\"noopener\">Anthropic \u2014 HH-RLHF preference dataset repository<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/docs\/trl\/dataset_formats\" target=\"_blank\" rel=\"noopener\">Hugging Face TRL \u2014 Dataset formats and preference data<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/docs\/trl\/dpo_trainer\" target=\"_blank\" rel=\"noopener\">Hugging Face TRL \u2014 DPO Trainer documentation<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen2.5-0.5B-Instruct\" target=\"_blank\" rel=\"noopener\">Qwen \u2014 Qwen2.5-0.5B-Instruct model card<\/a><\/li>\n<li><a href=\"https:\/\/www.nist.gov\/publications\/artificial-intelligence-risk-management-framework-generative-artificial-intelligence\" target=\"_blank\" rel=\"noopener\">NIST \u2014 AI Risk Management Framework: Generative AI Profile<\/a><\/li>\n<li><a href=\"https:\/\/guslok.com\/en\/train-qwen2-5-dpo-hh-rlhf\/\" target=\"_blank\" rel=\"noopener\">Guslok \u2014 Tutorial on auditing HH-RLHF pairs and training Qwen2.5 with DPO<\/a><\/li>\n<\/ul>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>\u03a4\u03bf DPO \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03c4\u03bf \u03c3\u03ae\u03bc\u03b1 \u03c4\u03c9\u03bd preference data. \u0394\u03b5\u03af\u03c4\u03b5 \u03c0\u03ce\u03c2 \u03b5\u03bb\u03ad\u03b3\u03c7\u03b5\u03c4\u03b5 pairs, length bias, lexical shortcuts, tokenizer limits \u03ba\u03b1\u03b9 \u03ba\u03c1\u03af\u03c3\u03b9\u03bc\u03b1 slices \u03c0\u03c1\u03b9\u03bd \u03b1\u03c0\u03cc fine-tuning.<\/p>","protected":false},"author":1,"featured_media":89441,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_gspb_post_css":"","content-type":"","footnotes":""},"categories":[199],"tags":[18888,19461,19462,19275,19463],"class_list":["post-89372","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-techniti-noimosyni","tag-ai-evaluation","tag-dpo","tag-llm-fine-tuning","tag-lora","tag-preference-data"],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":7}},"_links":{"self":[{"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/posts\/89372","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/comments?post=89372"}],"version-history":[{"count":0,"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/posts\/89372\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/media\/89441"}],"wp:attachment":[{"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/media?parent=89372"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/categories?post=89372"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/twodots.gr\/hr\/wp-json\/wp\/v2\/tags?post=89372"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}