{"id":98378,"date":"2026-10-11T18:54:20","date_gmt":"2026-10-11T15:54:20","guid":{"rendered":"https:\/\/twodots.gr\/?p=98378"},"modified":"2026-10-11T18:54:20","modified_gmt":"2026-10-11T15:54:20","slug":"masked-diffusion-llms-bottleneck-cpu-gpu","status":"publish","type":"post","link":"https:\/\/twodots.gr\/bg\/masked-diffusion-llms-bottleneck-cpu-gpu\/","title":{"rendered":"Masked diffusion LLMs: \u03b3\u03b9\u03b1\u03c4\u03af \u03c4\u03bf bottleneck \u03b4\u03b5\u03bd \u03ae\u03c4\u03b1\u03bd \u03b7 GPU"},"content":{"rendered":"<div class=\"td-article-lede\">\n<p><strong>\u041e\u0442\u0433\u043e\u0432\u043e\u0440 \u043f\u044a\u0440\u0432\u043e:<\/strong> \u03c3\u03c4\u03bf \u03bc\u03b5\u03c4\u03c1\u03b7\u03bc\u03ad\u03bd\u03bf masked diffusion LLM serving, \u03b7 H200 \u03b4\u03b5\u03bd \u03ae\u03c4\u03b1\u03bd \u03c4\u03bf \u03ba\u03cd\u03c1\u03b9\u03bf \u03cc\u03c1\u03b9\u03bf. \u039c\u03cc\u03bb\u03b9\u03c2 \u03c4\u03bf 24,4% \u03c4\u03bf\u03c5 \u03c3\u03c5\u03bd\u03bf\u03bb\u03b9\u03ba\u03bf\u03cd \u03c7\u03c1\u03cc\u03bd\u03bf\u03c5 \u03ae\u03c4\u03b1\u03bd GPU kernel execution, \u03b5\u03bd\u03ce \u03c4\u03bf 75,6% \u03ba\u03b1\u03c4\u03b1\u03bd\u03b1\u03bb\u03ce\u03b8\u03b7\u03ba\u03b5 \u03c3\u03c4\u03bf\u03bd CPU-side control loop \u03ba\u03b1\u03b9 \u03c3\u03c4\u03bf dispatch.<\/p>\n<\/div>\n<p>\u03a4\u03b1 masked diffusion LLMs \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03bf\u03c5\u03bd \u03c0\u03bf\u03bb\u03bb\u03ac tokens \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03b1, \u03b1\u03bb\u03bb\u03ac \u03c4\u03bf serving \u03c4\u03bf\u03c5\u03c2 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 profiling CPU, GPU, batching \u03ba\u03b1\u03b9 generation budgets.<\/p>\n<p>\u0397 \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03b7 \u03b1\u03c0\u03bf\u03ba\u03ac\u03bb\u03c5\u03c8\u03b7 tokens \u03b4\u03b5\u03bd \u03b5\u03b3\u03b3\u03c5\u03ac\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03bc\u03cc\u03bd\u03b7 \u03c4\u03b7\u03c2 \u03c5\u03c8\u03b7\u03bb\u03cc throughput: \u03b1\u03bd \u03bf host \u03b4\u03b5\u03bd \u03c0\u03c1\u03bf\u03bb\u03b1\u03b2\u03b1\u03af\u03bd\u03b5\u03b9 \u03bd\u03b1 \u03bf\u03c1\u03b3\u03b1\u03bd\u03ce\u03c3\u03b5\u03b9 \u03c4\u03b1 denoising steps, \u03b7 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ae GPU \u03c0\u03b5\u03c1\u03b9\u03bc\u03ad\u03bd\u03b5\u03b9.<\/p>\n<p>\u0420\u0430\u0431\u043e\u0442\u0430\u0442\u0430 <em>Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware<\/em> \u03c4\u03c9\u03bd Farhana Amin, Sabiha Afroz, Mona Moghadampanah \u03ba\u03b1\u03b9 Dimitrios S. Nikolopoulos \u03b5\u03be\u03b5\u03c4\u03ac\u03b6\u03b5\u03b9 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 \u03b1\u03c5\u03c4\u03cc \u03c4\u03bf serving workload. \u03a7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af LLaDA-8B-Instruct \u03bc\u03b5 D2F LoRA adapter, \u03c4\u03b1 benchmarks GSM8K \u03ba\u03b1\u03b9 HumanEval \u03ba\u03b1\u03b9 \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c3\u03b5 \u03bc\u03af\u03b1 NVIDIA H200.<\/p>\n<p>\u03a4\u03bf \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b7\u03c3\u03b9\u03b1\u03ba\u03cc \u03c3\u03c5\u03bc\u03c0\u03ad\u03c1\u03b1\u03c3\u03bc\u03b1 \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03b7 GPU \u03c0\u03b1\u03cd\u03b5\u03b9 \u03bd\u03b1 \u03ad\u03c7\u03b5\u03b9 \u03c3\u03b7\u03bc\u03b1\u03c3\u03af\u03b1. \u0395\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03ad\u03bd\u03b1 capacity plan \u03b3\u03b9\u03b1 \u03bd\u03ad\u03b1 \u03b1\u03c1\u03c7\u03b9\u03c4\u03b5\u03ba\u03c4\u03bf\u03bd\u03b9\u03ba\u03ae \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae\u03c2 \u03b4\u03b5\u03bd \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b2\u03b1\u03c3\u03b9\u03c3\u03c4\u03b5\u03af \u03bc\u03cc\u03bd\u03bf \u03c3\u03b5 FLOPS, VRAM \u03ae \u03bc\u03ad\u03c3\u03bf latency. \u03a7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 profiling \u03bf\u03bb\u03cc\u03ba\u03bb\u03b7\u03c1\u03b7\u03c2 \u03c4\u03b7\u03c2 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae\u03c2, generation budgets \u03c0\u03bf\u03c5 \u03b4\u03b5\u03bd \u03ba\u03cc\u03b2\u03bf\u03c5\u03bd \u03c4\u03b5\u03c7\u03bd\u03b7\u03c4\u03ac \u03c4\u03b9\u03c2 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 batching policy \u03c0\u03c1\u03bf\u03c3\u03b1\u03c1\u03bc\u03bf\u03c3\u03bc\u03ad\u03bd\u03b7 \u03c3\u03c4\u03bf computational pattern \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5.<\/p>\n<div class=\"td-article-toc\">\n<div class=\"td-toc-title\">\u0421\u044a\u0434\u044a\u0440\u0436\u0430\u043d\u0438\u0435<\/div>\n<ul>\n<li><a href=\"#parallilo-denoising\">\u0391\u03c0\u03cc \u03c4\u03bf token-by-token \u03c3\u03c4\u03bf \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03bf denoising<\/a><\/li>\n<li><a href=\"#difficulty-tiers\">\u0397 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 11 \u03b4\u03b9\u03b1\u03ba\u03c1\u03b9\u03c4\u03ac \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1<\/a><\/li>\n<li><a href=\"#prompt-routing\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf prompt length \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03b3\u03b9\u03b1 \u03ad\u03be\u03c5\u03c0\u03bd\u03bf routing<\/a><\/li>\n<li><a href=\"#generation-budgets\">\u03a4\u03b1 \u03bc\u03b9\u03ba\u03c1\u03ac generation budgets \u03ba\u03c1\u03cd\u03b2\u03bf\u03c5\u03bd \u03c4\u03bf variance<\/a><\/li>\n<li><a href=\"#humaneval\">\u03a4\u03b9 \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03af\u03c9\u03c3\u03b5 \u03c4\u03bf HumanEval<\/a><\/li>\n<li><a href=\"#synchronized-batching\">Synchronized batching: 16\u00d7 throughput \u03bc\u03b5 \u03ba\u03bf\u03b9\u03bd\u03cc forward pass<\/a><\/li>\n<li><a href=\"#cpu-bottleneck\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf H200 \u03c0\u03b5\u03c1\u03af\u03bc\u03b5\u03bd\u03b5 \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7\u03bd CPU<\/a><\/li>\n<li><a href=\"#batching-quality\">\u0397 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c5\u03c0\u03cc batching \u03ad\u03c7\u03b5\u03b9 \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2<\/a><\/li>\n<li><a href=\"#batch-timeout\">\u03a4\u03bf timeout \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03c4\u03b1\u03c4\u03b5\u03cd\u03b5\u03b9 \u03c4\u03b7 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03b7\u03c2 \u03bf\u03c5\u03c1\u03ac\u03c2<\/a><\/li>\n<li><a href=\"#business-infrastructure\">\u03a4\u03b9 \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03b3\u03b9\u03b1 \u03bc\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 AI infrastructure<\/a><\/li>\n<li><a href=\"#limitations\">\u03a0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03bf\u03af \u03c0\u03bf\u03c5 \u03b1\u03bb\u03bb\u03ac\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf \u03b2\u03ac\u03c1\u03bf\u03c2 \u03c4\u03c9\u03bd \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b1\u03c3\u03bc\u03ac\u03c4\u03c9\u03bd<\/a><\/li>\n<\/ul>\n<\/div>\n<h2 id=\"parallilo-denoising\">\u0391\u03c0\u03cc \u03c4\u03bf token-by-token \u03c3\u03c4\u03bf \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03bf denoising<\/h2>\n<p>\u0388\u03bd\u03b1 autoregressive \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03c0\u03b1\u03c1\u03ac\u03b3\u03b5\u03b9 \u03c4\u03bf \u03b5\u03c0\u03cc\u03bc\u03b5\u03bd\u03bf token \u03bc\u03b5 \u03b2\u03ac\u03c3\u03b7 \u03cc\u03c3\u03b1 \u03ad\u03c7\u03bf\u03c5\u03bd \u03c0\u03c1\u03bf\u03b7\u03b3\u03b7\u03b8\u03b5\u03af. \u0388\u03bd\u03b1 masked diffusion LLM \u03be\u03b5\u03ba\u03b9\u03bd\u03ac \u03bc\u03b5 \u03ba\u03b1\u03bb\u03c5\u03bc\u03bc\u03ad\u03bd\u03b5\u03c2 \u03b8\u03ad\u03c3\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 \u03c4\u03b9\u03c2 \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03b5\u03b9 \u03c3\u03c4\u03b1\u03b4\u03b9\u03b1\u03ba\u03ac \u03bc\u03ad\u03c3\u03b1 \u03b1\u03c0\u03cc denoising steps. \u03a3\u03c4\u03b7\u03bd \u03c5\u03bb\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7 D2F \u03c4\u03b7\u03c2 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7\u03c2, \u03b7 \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae \u03bf\u03c1\u03b3\u03b1\u03bd\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 blocks. \u039d\u03ad\u03bf block \u03b1\u03c1\u03c7\u03af\u03b6\u03b5\u03b9 \u03bc\u03cc\u03bd\u03bf \u03cc\u03c4\u03b1\u03bd \u03c4\u03bf \u03c0\u03c1\u03bf\u03b7\u03b3\u03bf\u03cd\u03bc\u03b5\u03bd\u03bf \u03c0\u03b5\u03c1\u03ac\u03c3\u03b5\u03b9 \u03ad\u03bd\u03b1 \u03ba\u03b1\u03b8\u03bf\u03c1\u03b9\u03c3\u03bc\u03ad\u03bd\u03bf \u03cc\u03c1\u03b9\u03bf \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b7\u03c2.<\/p>\n<p>\u0397 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03b1\u03c5\u03c4\u03ae \u03b1\u03bb\u03bb\u03ac\u03b6\u03b5\u03b9 \u03c4\u03bf \u03c3\u03c7\u03ae\u03bc\u03b1 \u03c4\u03bf\u03c5 computational work \u03ba\u03b1\u03b9, \u03b5\u03c0\u03bf\u03bc\u03ad\u03bd\u03c9\u03c2, \u03c4\u03bf \u03c0\u03ce\u03c2 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b3\u03af\u03bd\u03bf\u03bd\u03c4\u03b1\u03b9 batching, admission \u03ba\u03b1\u03b9 eviction. \u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b4\u03b5\u03bd \u03c0\u03b1\u03c1\u03bf\u03c5\u03c3\u03b9\u03ac\u03b6\u03bf\u03c5\u03bd \u03ad\u03bd\u03b1\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03c9\u03bc\u03ad\u03bd\u03bf \u03b1\u03bd\u03c4\u03b1\u03b3\u03c9\u03bd\u03b9\u03c3\u03c4\u03b9\u03ba\u03cc inference engine. \u03a0\u03c1\u03ce\u03c4\u03b1 \u03c7\u03b1\u03c1\u03c4\u03bf\u03b3\u03c1\u03b1\u03c6\u03bf\u03cd\u03bd \u03c0\u03cc\u03c3\u03bf \u03bc\u03b5\u03c4\u03b1\u03b2\u03ac\u03bb\u03bb\u03b5\u03c4\u03b1\u03b9 \u03c4\u03bf \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03b1\u03bd\u03ac request, \u03b1\u03bd \u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c0\u03b5\u03c4\u03b1\u03b9 \u03c0\u03c1\u03b9\u03bd \u03b1\u03c1\u03c7\u03af\u03c3\u03b5\u03b9 \u03b7 \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae, \u03c0\u03cc\u03c3\u03bf \u03c0\u03b1\u03c1\u03b1\u03bc\u03bf\u03c1\u03c6\u03ce\u03bd\u03b5\u03b9 \u03c4\u03bf generation budget \u03c4\u03b9\u03c2 \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 \u03c0\u03bf\u03cd \u03ba\u03b1\u03c4\u03b1\u03bd\u03b1\u03bb\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ac \u03bf \u03c7\u03c1\u03cc\u03bd\u03bf\u03c2.<\/p>\n<aside class=\"td-article-note\"><strong>\u0397 \u03c3\u03b5\u03b9\u03c1\u03ac \u03c4\u03b7\u03c2 \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7\u03c2 \u03ad\u03c7\u03b5\u03b9 \u03c3\u03b7\u03bc\u03b1\u03c3\u03af\u03b1.<\/strong> \u0391\u03bd \u03bc\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03b1\u03b3\u03bf\u03c1\u03ac\u03c3\u03b5\u03b9 \u03c0\u03b5\u03c1\u03b9\u03c3\u03c3\u03cc\u03c4\u03b5\u03c1\u03b5\u03c2 GPUs \u03c0\u03c1\u03b9\u03bd \u03b4\u03b9\u03b1\u03c7\u03c9\u03c1\u03af\u03c3\u03b5\u03b9 kernel time, dispatch time \u03ba\u03b1\u03b9 control-loop time, \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b1\u03c5\u03be\u03ae\u03c3\u03b5\u03b9 \u03c4\u03bf \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c7\u03c9\u03c1\u03af\u03c2 \u03bd\u03b1 \u03b1\u03c6\u03b1\u03b9\u03c1\u03ad\u03c3\u03b5\u03b9 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc bottleneck.<\/aside>\n<p>\u0397 \u03b1\u03c1\u03c7\u03ae \u03b9\u03c3\u03c7\u03cd\u03b5\u03b9 \u03c0\u03ad\u03c1\u03b1 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf paper. \u038c\u03c0\u03c9\u03c2 \u03b4\u03b5\u03af\u03c7\u03bd\u03b5\u03b9 \u03ba\u03b1\u03b9 \u03b7 \u03b1\u03bd\u03ac\u03bb\u03c5\u03c3\u03b7 \u03b3\u03b9\u03b1 \u03c4\u03bf <a href=\"https:\/\/twodots.gr\/llm4llm-gpu-kernel-dokimi-pragmatiko-montelo\/\">\u03b3\u03c1\u03ae\u03b3\u03bf\u03c1\u03bf GPU kernel \u03bc\u03ad\u03c3\u03b1 \u03c3\u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf<\/a>, \u03ad\u03bd\u03b1 \u03c4\u03bf\u03c0\u03b9\u03ba\u03cc microbenchmark \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03cc\u03c4\u03b1\u03bd \u03bf end-to-end \u03c7\u03c1\u03cc\u03bd\u03bf\u03c2 \u03ba\u03b1\u03b8\u03bf\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03bf\u03bb\u03cc\u03ba\u03bb\u03b7\u03c1\u03bf \u03c4\u03bf serving stack.<\/p>\n<h2 id=\"difficulty-tiers\">\u0397 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 11 \u03b4\u03b9\u03b1\u03ba\u03c1\u03b9\u03c4\u03ac \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1<\/h2>\n<p>\u03a3\u03b5 100 requests \u03c4\u03bf\u03c5 GSM8K, \u03bc\u03b5 thresholds \u03c4<sub>add<\/sub>=0,5 \u03ba\u03b1\u03b9 \u03c4<sub>decode<\/sub>=0,9, \u03bf \u03b1\u03c1\u03b9\u03b8\u03bc\u03cc\u03c2 \u03c4\u03c9\u03bd denoising steps \u03c0\u03ae\u03c1\u03b5 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 11 \u03c4\u03b9\u03bc\u03ad\u03c2: \u03b1\u03c0\u03cc 178 \u03ad\u03c9\u03c2 468, \u03bc\u03b5 \u03b1\u03c0\u03cc\u03c3\u03c4\u03b1\u03c3\u03b7 29 steps \u03bc\u03b5\u03c4\u03b1\u03be\u03cd \u03b4\u03b9\u03b1\u03b4\u03bf\u03c7\u03b9\u03ba\u03ce\u03bd \u03b5\u03c0\u03b9\u03c0\u03ad\u03b4\u03c9\u03bd. \u03a4\u03bf \u03bc\u03bf\u03c4\u03af\u03b2\u03bf \u03c0\u03c1\u03bf\u03ba\u03cd\u03c0\u03c4\u03b5\u03b9 \u03b1\u03c0\u03cc \u03c4\u03bf\u03bd \u03ba\u03b1\u03bd\u03cc\u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03b8\u03ae\u03ba\u03b7\u03c2 blocks \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03b1\u03c0\u03cc \u03c4\u03c5\u03c7\u03b1\u03af\u03bf \u03b8\u03cc\u03c1\u03c5\u03b2\u03bf.<\/p>\n<p>\u03a4\u03bf step count \u03c0\u03c1\u03bf\u03ad\u03b2\u03bb\u03b5\u03c8\u03b5 \u03c3\u03c7\u03b5\u03b4\u03cc\u03bd \u03c4\u03ad\u03bb\u03b5\u03b9\u03b1 \u03c4\u03bf end-to-end latency \u03c3\u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf \u03c0\u03b5\u03af\u03c1\u03b1\u03bc\u03b1: r=0,9997 \u03ba\u03b1\u03b9 R\u00b2=0,9994. \u0398\u03b5\u03c9\u03c1\u03b7\u03c4\u03b9\u03ba\u03ac, \u03b1\u03bd \u03c4\u03bf \u03c3\u03cd\u03c3\u03c4\u03b7\u03bc\u03b1 \u03b3\u03bd\u03ce\u03c1\u03b9\u03b6\u03b5 \u03b1\u03c0\u03cc \u03c4\u03b7\u03bd \u03b5\u03af\u03c3\u03bf\u03b4\u03bf \u03c3\u03b5 \u03c0\u03bf\u03b9\u03bf \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03bf \u03b8\u03b1 \u03ba\u03b1\u03c4\u03b1\u03bb\u03ae\u03be\u03b5\u03b9 \u03ad\u03bd\u03b1 request, \u03bf scheduler \u03b8\u03b1 \u03b5\u03c0\u03ad\u03bb\u03b5\u03b3\u03b5 \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03b5 11 \u03b3\u03bd\u03c9\u03c3\u03c4\u03ad\u03c2 \u03b5\u03ba\u03b2\u03ac\u03c3\u03b5\u03b9\u03c2 \u03b1\u03bd\u03c4\u03af \u03bd\u03b1 \u03b5\u03ba\u03c4\u03b9\u03bc\u03ac \u03ad\u03bd\u03b1\u03bd \u03c3\u03c5\u03bd\u03b5\u03c7\u03cc\u03bc\u03b5\u03bd\u03bf \u03c7\u03c1\u03cc\u03bd\u03bf \u03c5\u03c0\u03b7\u03c1\u03b5\u03c3\u03af\u03b1\u03c2. \u0397 \u03c0\u03bb\u03b7\u03c1\u03bf\u03c6\u03bf\u03c1\u03af\u03b1 \u03b1\u03c5\u03c4\u03ae, \u03cc\u03bc\u03c9\u03c2, \u03b4\u03b5\u03bd \u03ae\u03c4\u03b1\u03bd \u03b4\u03b9\u03b1\u03b8\u03ad\u03c3\u03b9\u03bc\u03b7 \u03c0\u03c1\u03b9\u03bd \u03be\u03b5\u03b4\u03b9\u03c0\u03bb\u03c9\u03b8\u03b5\u03af \u03b7 \u03b4\u03b9\u03b1\u03b4\u03b9\u03ba\u03b1\u03c3\u03af\u03b1 denoising.<\/p>\n<p>\u0397 \u03b4\u03b9\u03b1\u03ba\u03cd\u03bc\u03b1\u03bd\u03c3\u03b7 \u03ae\u03c4\u03b1\u03bd \u03bf\u03c5\u03c3\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03ae. \u03a3\u03c4\u03bf GSM8K \u03bc\u03b5 ceiling 512 tokens, \u03b7 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b5 coefficient of variation 0,272 \u03ba\u03b1\u03b9 \u03bb\u03cc\u03b3\u03bf \u03bc\u03ad\u03b3\u03b9\u03c3\u03c4\u03bf\u03c5 \u03c0\u03c1\u03bf\u03c2 \u03b5\u03bb\u03ac\u03c7\u03b9\u03c3\u03c4\u03bf latency 2,64\u00d7. \u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b1\u03bd\u03b1\u03c6\u03ad\u03c1\u03bf\u03c5\u03bd CV \u03b1\u03c0\u03cc 0,256 \u03ad\u03c9\u03c2 0,283 \u03c3\u03b5 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac measurement passes \u03bc\u03b5 n=100 \u03ba\u03b1\u03b9 \u03c4\u03b9\u03c2 \u03ba\u03c1\u03b1\u03c4\u03bf\u03cd\u03bd \u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ad\u03c2 \u03b1\u03bd\u03c4\u03af \u03bd\u03b1 \u03c4\u03b9\u03c2 \u03c3\u03c5\u03b3\u03c7\u03c9\u03bd\u03b5\u03cd\u03c3\u03bf\u03c5\u03bd \u03c4\u03b5\u03c7\u03bd\u03b7\u03c4\u03ac.<\/p>\n<div class=\"td-chart td-chart--metrics\">\n<div class=\"td-chart-head\">\n<p class=\"td-chart-kicker\">LLaDA-8B-Instruct + D2F \u00b7 \u03bc\u03af\u03b1 H200<\/p>\n<p class=\"td-chart-title\">\u03a4\u03b1 \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 \u03bc\u03b5\u03b3\u03ad\u03b8\u03b7 \u03c0\u03bf\u03c5 \u03b1\u03bb\u03bb\u03ac\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf capacity plan<\/p>\n<p class=\"td-chart-intro\">\u039a\u03ac\u03b8\u03b5 \u03c4\u03b9\u03bc\u03ae \u03b1\u03c6\u03bf\u03c1\u03ac \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf \u03c0\u03b5\u03b9\u03c1\u03b1\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc setup \u03ba\u03b1\u03b9 \u03b4\u03b5\u03bd \u03b1\u03c0\u03bf\u03c4\u03b5\u03bb\u03b5\u03af \u03b3\u03b5\u03bd\u03b9\u03ba\u03cc benchmark \u03b3\u03b9\u03b1 \u03cc\u03bb\u03b1 \u03c4\u03b1 diffusion LLMs.<\/p>\n<\/div>\n<div class=\"td-metric-grid\">\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">11 tiers<\/span><span class=\"td-metric-label\">\u03b4\u03b9\u03b1\u03ba\u03c1\u03b9\u03c4\u03ac \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1 \u03b1\u03c0\u03cc 178 \u03ad\u03c9\u03c2 468 denoising steps<\/span><\/div>\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">R\u00b2 0,150<\/span><span class=\"td-metric-label\">\u03c4\u03bf \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf admission-time signal \u03c3\u03c4\u03bf GSM8K<\/span><\/div>\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">16,0\u00d7<\/span><span class=\"td-metric-label\">throughput \u03c3\u03c4\u03bf batch size 16 \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 single-request baseline<\/span><\/div>\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">75,6%<\/span><span class=\"td-metric-label\">CPU-side overhead \u03c3\u03c4\u03bf profiled request<\/span><\/div>\n<\/div>\n<\/div>\n<h2 id=\"prompt-routing\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf prompt length \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03b3\u03b9\u03b1 \u03ad\u03be\u03c5\u03c0\u03bd\u03bf routing<\/h2>\n<p>\u0397 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03b4\u03bf\u03ba\u03af\u03bc\u03b1\u03c3\u03b5 \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 \u03c6\u03b8\u03b7\u03bd\u03ac signals \u03c0\u03bf\u03c5 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b4\u03b9\u03b1\u03b8\u03ad\u03c3\u03b9\u03bc\u03b1 \u03bc\u03b5\u03c4\u03ac \u03c4\u03bf \u03c0\u03c1\u03ce\u03c4\u03bf denoising step: \u03c4\u03bf \u03c0\u03bf\u03c3\u03bf\u03c3\u03c4\u03cc tokens \u03c0\u03ac\u03bd\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf confidence threshold, \u03c4\u03bf \u03bc\u03ad\u03c3\u03bf token confidence, \u03c4\u03bf output entropy \u03ba\u03b1\u03b9 \u03c4\u03bf prompt length. \u03a4\u03b1 R\u00b2 \u03c9\u03c2 \u03c0\u03c1\u03bf\u03c2 \u03c4\u03bf \u03c4\u03b5\u03bb\u03b9\u03ba\u03cc difficulty level \u03ae\u03c4\u03b1\u03bd \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b1 0,011, 0,004, 0,023 \u03ba\u03b1\u03b9 0,150. \u038c\u03bb\u03b1 \u03ad\u03bc\u03b5\u03b9\u03bd\u03b1\u03bd \u03c0\u03bf\u03bb\u03cd \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03c1\u03b1\u03ba\u03c4\u03b9\u03ba\u03cc \u03ba\u03b1\u03c4\u03ce\u03c6\u03bb\u03b9 0,5 \u03c0\u03bf\u03c5 \u03ad\u03b8\u03b5\u03c3\u03b1\u03bd \u03bf\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2.<\/p>\n<p>\u0397 \u03b5\u03be\u03ae\u03b3\u03b7\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b4\u03bf\u03bc\u03b9\u03ba\u03ae. \u039c\u03b5\u03c4\u03ac \u03c4\u03bf \u03c0\u03c1\u03ce\u03c4\u03bf step, \u03ba\u03b1\u03b9 \u03bf\u03b9 32 \u03b8\u03ad\u03c3\u03b5\u03b9\u03c2 \u03c4\u03bf\u03c5 \u03c0\u03c1\u03ce\u03c4\u03bf\u03c5 block \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03bf\u03c5\u03bd masked \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b1 \u03b1\u03c0\u03cc \u03c4\u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03c4\u03bf\u03c5 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ae\u03bc\u03b1\u03c4\u03bf\u03c2. \u0388\u03bd\u03b1 \u03c0\u03bf\u03bb\u03cd \u03c0\u03c1\u03ce\u03b9\u03bc\u03bf signal \u03b1\u03bd\u03c4\u03b1\u03bd\u03b1\u03ba\u03bb\u03ac \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7 \u03b3\u03b5\u03bd\u03b9\u03ba\u03ae prior \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5 \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03c4\u03b7 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03b7 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae \u03c0\u03bf\u03c5 \u03b8\u03b1 \u03b1\u03ba\u03bf\u03bb\u03bf\u03c5\u03b8\u03ae\u03c3\u03b5\u03b9 \u03c4\u03bf request. \u0397 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b1\u03bd\u03b1\u03b4\u03cd\u03b5\u03c4\u03b1\u03b9 \u03bc\u03ad\u03c3\u03b1 \u03c3\u03c4\u03b7 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae denoising.<\/p>\n<p>\u0393\u03b9\u03b1 product \u03ba\u03b1\u03b9 operations teams \u03b1\u03c5\u03c4\u03cc \u03b1\u03c0\u03bf\u03ba\u03bb\u03b5\u03af\u03b5\u03b9 \u03bc\u03b9\u03b1 \u03b4\u03b5\u03bb\u03b5\u03b1\u03c3\u03c4\u03b9\u03ba\u03ae \u03b1\u03bb\u03bb\u03ac \u03bc\u03b7 \u03c4\u03b5\u03ba\u03bc\u03b7\u03c1\u03b9\u03c9\u03bc\u03ad\u03bd\u03b7 \u03c3\u03c5\u03bd\u03c4\u03cc\u03bc\u03b5\u03c5\u03c3\u03b7: \u03ad\u03bd\u03b1 \u03c3\u03cd\u03bd\u03c4\u03bf\u03bc\u03bf prompt \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03ba\u03b1\u03c4\u2019 \u03b1\u03bd\u03ac\u03b3\u03ba\u03b7 \u03c6\u03b8\u03b7\u03bd\u03cc \u03ba\u03b1\u03b9 \u03ad\u03bd\u03b1 \u03b3\u03c1\u03ae\u03b3\u03bf\u03c1\u03bf confidence check \u03b4\u03b5\u03bd \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c0\u03b5\u03b9 \u03b1\u03be\u03b9\u03cc\u03c0\u03b9\u03c3\u03c4\u03b1 \u03c4\u03bf\u03bd \u03c7\u03c1\u03cc\u03bd\u03bf \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b7\u03c2. \u03a4\u03bf routing \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 evidence \u03b1\u03c0\u03cc \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf model, task \u03ba\u03b1\u03b9 engine, \u03cc\u03c0\u03c9\u03c2 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03c4\u03bf <a href=\"https:\/\/twodots.gr\/llm-routing-latency-accuracy-cost\/\">LLM routing \u03bc\u03b5 \u03b5\u03c0\u03af\u03b3\u03bd\u03c9\u03c3\u03b7 latency, accuracy \u03ba\u03b1\u03b9 \u03ba\u03cc\u03c3\u03c4\u03bf\u03c5\u03c2<\/a>.<\/p>\n<h2 id=\"generation-budgets\">\u03a4\u03b1 \u03bc\u03b9\u03ba\u03c1\u03ac generation budgets \u03ba\u03c1\u03cd\u03b2\u03bf\u03c5\u03bd \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc serving variance<\/h2>\n<p>\u0397 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03b5\u03c0\u03b1\u03bd\u03ad\u03bb\u03b1\u03b2\u03b5 \u03c4\u03bf GSM8K \u03bc\u03b5 budgets 128, 256, 512 \u03ba\u03b1\u03b9 1024 tokens, \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03ce\u03bd\u03c4\u03b1\u03c2 100 requests \u03b1\u03bd\u03ac budget \u03c3\u03b5 single-request inference. \u03a3\u03c4\u03b1 128 tokens, \u03c4\u03bf 100% \u03c4\u03c9\u03bd \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03c9\u03bd \u03ba\u03cc\u03c0\u03b7\u03ba\u03b5 \u03b1\u03c0\u03cc \u03c4\u03bf \u03cc\u03c1\u03b9\u03bf \u03ba\u03b1\u03b9 \u03b7 accuracy \u03ae\u03c4\u03b1\u03bd 6%. \u03a3\u03c4\u03b1 256 tokens, \u03c4\u03bf truncation \u03c0\u03b1\u03c1\u03ad\u03bc\u03b5\u03bd\u03b5 83% \u03ba\u03b1\u03b9 \u03b7 accuracy \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 48%. \u03a3\u03c4\u03b1 512 tokens, \u03c4\u03bf truncation \u03c5\u03c0\u03bf\u03c7\u03ce\u03c1\u03b7\u03c3\u03b5 \u03c3\u03b5 3% \u03ba\u03b1\u03b9 \u03b7 accuracy \u03ae\u03c4\u03b1\u03bd 76%, \u03b5\u03bd\u03ce \u03c3\u03c4\u03b1 1024 tokens \u03bf\u03b9 \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b5\u03c2 \u03c4\u03b9\u03bc\u03ad\u03c2 \u03ae\u03c4\u03b1\u03bd 1% \u03ba\u03b1\u03b9 73%.<\/p>\n<p>\u03a4\u03bf workload \u03c7\u03c1\u03b5\u03b9\u03b1\u03b6\u03cc\u03c4\u03b1\u03bd \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 320 tokens \u03ba\u03b1\u03c4\u03ac \u03bc\u03ad\u03c3\u03bf \u03cc\u03c1\u03bf \u03b3\u03b9\u03b1 \u03bd\u03b1 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03c3\u03b5\u03b9 \u03ad\u03bd\u03b1 GSM8K \u03c0\u03c1\u03cc\u03b2\u03bb\u03b7\u03bc\u03b1: 317,4 tokens \u03c3\u03c4\u03b7 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae \u03bc\u03b5 \u03cc\u03c1\u03b9\u03bf 512 \u03ba\u03b1\u03b9 323,5 \u03bc\u03b5 \u03cc\u03c1\u03b9\u03bf 1024. \u03a0\u03b1\u03c1\u2019 \u03cc\u03bb\u03b1 \u03b1\u03c5\u03c4\u03ac, \u03c4\u03bf latency CV \u03c3\u03c5\u03bd\u03ad\u03c7\u03b9\u03c3\u03b5 \u03bd\u03b1 \u03b1\u03bd\u03b5\u03b2\u03b1\u03af\u03bd\u03b5\u03b9 \u03b1\u03c0\u03cc 0,283 \u03c3\u03c4\u03b1 512 tokens \u03c3\u03b5 0,360 \u03c3\u03c4\u03b1 1024. \u03a4\u03b1 P50\/P90\/P99 \u03ae\u03c4\u03b1\u03bd 6,46\/9,72\/11,48 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03c3\u03c4\u03b1 512 \u03ba\u03b1\u03b9 6,41\/9,05\/10,66 \u03c3\u03c4\u03b1 1024.<\/p>\n<p>\u03a4\u03bf \u03c3\u03c5\u03bc\u03c0\u03ad\u03c1\u03b1\u03c3\u03bc\u03b1 \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf budget \u03b5\u03af\u03bd\u03b1\u03b9 \u03c0\u03ac\u03bd\u03c4\u03b1 \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf. \u0395\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03ad\u03bd\u03b1 benchmark \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c6\u03c5\u03c3\u03b9\u03ba\u03cc completion length \u03bc\u03b5\u03c4\u03c1\u03ac \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7 \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac truncation \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc serving workload. \u039c\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03c0\u03bf\u03c5 \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03b9 providers \u03ae architectures \u03bc\u03b5 \u03bc\u03b9\u03ba\u03c1\u03ac caps \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03c3\u03b5\u03b9 \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc variance \u03b1\u03c0\u03bb\u03ce\u03c2 \u03b5\u03c0\u03b5\u03b9\u03b4\u03ae \u03ba\u03cc\u03b2\u03b5\u03b9 \u03c4\u03b9\u03c2 \u03b4\u03cd\u03c3\u03ba\u03bf\u03bb\u03b5\u03c2 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c0\u03c1\u03b9\u03bd \u03c6\u03b1\u03bd\u03b5\u03af \u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac.<\/p>\n<h2 id=\"humaneval\">\u03a4\u03b9 \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03af\u03c9\u03c3\u03b5 \u03c4\u03bf HumanEval<\/h2>\n<p>\u0393\u03b9\u03b1 \u03bd\u03b1 \u03b5\u03bb\u03ad\u03b3\u03be\u03bf\u03c5\u03bd \u03b1\u03bd \u03c4\u03bf \u03b5\u03cd\u03c1\u03b7\u03bc\u03b1 \u03c0\u03b5\u03c1\u03b9\u03bf\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03c4\u03b1 \u03bc\u03b1\u03b8\u03b7\u03bc\u03b1\u03c4\u03b9\u03ba\u03ac \u03c0\u03c1\u03bf\u03b2\u03bb\u03ae\u03bc\u03b1\u03c4\u03b1, \u03bf\u03b9 \u03b5\u03c1\u03b5\u03c5\u03bd\u03b7\u03c4\u03ad\u03c2 \u03b5\u03c0\u03b1\u03bd\u03ad\u03bb\u03b1\u03b2\u03b1\u03bd block-size sweep \u03ba\u03b1\u03b9 variance characterization \u03c3\u03b5 64 HumanEval code-completion problems. \u039c\u03ad\u03c3\u03b1 \u03c3\u03c4\u03bf \u03af\u03b4\u03b9\u03bf request, \u03c4\u03bf latency \u03ad\u03bc\u03b5\u03b9\u03bd\u03b5 \u03c3\u03c7\u03b5\u03b4\u03cc\u03bd \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc \u03b3\u03b9\u03b1 block sizes 16, 32, 64 \u03ba\u03b1\u03b9 128, \u03bc\u03b5 CV 0,0096. \u039c\u03b5\u03c4\u03b1\u03be\u03cd \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ce\u03bd requests, \u03c4\u03bf CV \u03ae\u03c4\u03b1\u03bd 0,345. \u0397 \u03b1\u03c0\u03cc\u03c3\u03c4\u03b1\u03c3\u03b7 \u03ae\u03c4\u03b1\u03bd \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 36\u00d7, \u03c0\u03bf\u03bb\u03cd \u03ba\u03bf\u03bd\u03c4\u03ac \u03c3\u03c4\u03bf \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03bf 36\u00d7 \u03c4\u03bf\u03c5 GSM8K, \u03cc\u03c0\u03bf\u03c5 \u03bf\u03b9 \u03c4\u03b9\u03bc\u03ad\u03c2 \u03ae\u03c4\u03b1\u03bd 0,0073 \u03ba\u03b1\u03b9 0,260.<\/p>\n<p>\u03a4\u03bf HumanEval \u03b5\u03af\u03c7\u03b5 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b7 \u03b4\u03b9\u03b1\u03ba\u03cd\u03bc\u03b1\u03bd\u03c3\u03b7: CV 0,343-0,346 \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 0,256-0,263 \u03c3\u03c4\u03bf GSM8K, \u03ba\u03b1\u03b9 max-to-min ratio 3,99-4,09\u00d7 \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 2,60-2,70\u00d7. \u03a4\u03bf prompt length \u03b5\u03af\u03c7\u03b5 \u03b9\u03c3\u03c7\u03c5\u03c1\u03cc\u03c4\u03b5\u03c1\u03b7 \u03c3\u03c5\u03c3\u03c7\u03ad\u03c4\u03b9\u03c3\u03b7 \u03bc\u03b5 \u03c4\u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03c3\u03c4\u03bf HumanEval, r=0,648 \u03ba\u03b1\u03b9 R\u00b2=0,42, \u03b1\u03bb\u03bb\u03ac \u03ba\u03b1\u03b9 \u03c0\u03ac\u03bb\u03b9 \u03b4\u03b5\u03bd \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 \u03c4\u03bf \u03c0\u03c1\u03b1\u03ba\u03c4\u03b9\u03ba\u03cc \u03ba\u03b1\u03c4\u03ce\u03c6\u03bb\u03b9 \u03c0\u03c1\u03cc\u03b2\u03bb\u03b5\u03c8\u03b7\u03c2 \u03c4\u03b7\u03c2 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7\u03c2.<\/p>\n<p>\u039f\u03b9 \u03af\u03b4\u03b9\u03bf\u03b9 \u03bf\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03c7\u03b1\u03c1\u03b1\u03ba\u03c4\u03b7\u03c1\u03af\u03b6\u03bf\u03c5\u03bd \u03c4\u03b7 \u03c3\u03c5\u03bc\u03c6\u03c9\u03bd\u03af\u03b1 \u03c3\u03c5\u03bd\u03b5\u03c0\u03ae \u03b1\u03bb\u03bb\u03ac \u03c0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03ad\u03bd\u03b7 \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03ba\u03b1\u03b8\u03bf\u03bb\u03b9\u03ba\u03ae \u03b1\u03c0\u03cc\u03b4\u03b5\u03b9\u03be\u03b7. \u0397 \u03c3\u03c9\u03c3\u03c4\u03ae \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b7\u03bc\u03b1\u03c4\u03b9\u03ba\u03ae \u03b1\u03bd\u03ac\u03b3\u03bd\u03c9\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b1 \u03c0\u03c1\u03bf\u03c3\u03b5\u03ba\u03c4\u03b9\u03ba\u03ae: \u03c4\u03bf pattern \u03b1\u03be\u03af\u03b6\u03b5\u03b9 \u03bd\u03b1 \u03b5\u03c0\u03b1\u03bb\u03b7\u03b8\u03b5\u03c5\u03c4\u03b5\u03af \u03c3\u03b5 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac tasks, models \u03ba\u03b1\u03b9 hardware \u03c0\u03c1\u03b9\u03bd \u03bc\u03b5\u03c4\u03b1\u03c4\u03c1\u03b1\u03c0\u03b5\u03af \u03c3\u03b5 production policy.<\/p>\n<h2 id=\"synchronized-batching\">Synchronized batching: 16\u00d7 throughput \u03bc\u03b5 \u03ba\u03bf\u03b9\u03bd\u03cc forward pass<\/h2>\n<p>\u03a3\u03c4\u03bf synchronized batching, \u03cc\u03bb\u03b1 \u03c4\u03b1 requests \u03b5\u03bd\u03cc\u03c2 batch \u03c0\u03c1\u03bf\u03c3\u03b1\u03c1\u03bc\u03cc\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03c3\u03b5 \u03ba\u03bf\u03b9\u03bd\u03cc shape \u03ba\u03b1\u03b9 \u03c0\u03c1\u03bf\u03c7\u03c9\u03c1\u03bf\u03cd\u03bd \u03bc\u03b5 \u03ad\u03bd\u03b1 shared forward pass \u03c3\u03b5 \u03ba\u03ac\u03b8\u03b5 denoising step. \u039c\u03b5\u03c4\u03ac \u03c4\u03bf pass, \u03ba\u03ac\u03b8\u03b5 request \u03c3\u03c5\u03bd\u03b5\u03c7\u03af\u03b6\u03b5\u03b9 \u03bd\u03b1 \u03ba\u03ac\u03bd\u03b5\u03b9 sampling \u03bc\u03b5 \u03c4\u03bf \u03b4\u03b9\u03ba\u03cc \u03c4\u03bf\u03c5 stream. \u0397 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03b9 \u03bc\u03b5 per-request dispatch, \u03cc\u03c0\u03bf\u03c5 \u03ba\u03ac\u03b8\u03b5 request \u03b5\u03ba\u03c4\u03b5\u03bb\u03b5\u03af \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03bf forward pass \u03c3\u03b5 \u03ba\u03ac\u03b8\u03b5 step.<\/p>\n<p>\u03a3\u03c4\u03bf batch size 16, \u03c4\u03bf synchronized batching \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 1,939 requests \u03b1\u03bd\u03ac \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03bf \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 0,121 requests \u03b1\u03bd\u03ac \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03bf \u03c3\u03c4\u03bf single-request throughput: \u03b2\u03b5\u03bb\u03c4\u03af\u03c9\u03c3\u03b7 16,0\u00d7. \u03a4\u03bf \u03c0\u03c1\u03cc\u03c3\u03b8\u03b5\u03c4\u03bf overhead \u03c4\u03bf\u03c5 per-request dispatch \u03b1\u03ba\u03bf\u03bb\u03bf\u03cd\u03b8\u03b7\u03c3\u03b5 \u03c3\u03c4\u03b5\u03bd\u03ac \u03c4\u03bf batch size: 2,4\u00d7 \u03c3\u03c4\u03bf B=2, 5,1\u00d7 \u03c3\u03c4\u03bf B=4 \u03ba\u03b1\u03b9 8,1\u00d7 \u03c3\u03c4\u03bf B=8. \u0393\u03b9\u03b1 B requests, \u03b7 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b7 \u03b5\u03ba\u03c4\u03ad\u03bb\u03b5\u03c3\u03b7 \u03c0\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03b9 B forward passes \u03b1\u03bd\u03ac denoising step, \u03b5\u03bd\u03ce \u03b7 \u03ba\u03bf\u03b9\u03bd\u03ae \u03b5\u03ba\u03c4\u03ad\u03bb\u03b5\u03c3\u03b7 \u03c0\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03b9 \u03ad\u03bd\u03b1.<\/p>\n<div class=\"td-comparison td-comparison-cards--horizontal\">\n<div class=\"td-chart-head\">\n<p class=\"td-chart-kicker\">\u03a4\u03bf \u03af\u03b4\u03b9\u03bf batch, \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ae \u03bc\u03bf\u03bd\u03ac\u03b4\u03b1 \u03c0\u03b1\u03c1\u03b1\u03bb\u03bb\u03b7\u03bb\u03b9\u03c3\u03bc\u03bf\u03cd<\/p>\n<p class=\"td-chart-title\">\u03a4\u03bf \u03ba\u03ad\u03c1\u03b4\u03bf\u03c2 \u03ad\u03c1\u03c7\u03b5\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03c4\u03b7\u03bd \u03ba\u03bf\u03b9\u03bd\u03ae \u03b5\u03c1\u03b3\u03b1\u03c3\u03af\u03b1 \u03b1\u03bd\u03ac denoising step<\/p>\n<p class=\"td-chart-intro\">\u0397 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03b1\u03c0\u03bf\u03bc\u03bf\u03bd\u03ce\u03bd\u03b5\u03b9 \u03c4\u03bf\u03bd dispatch \u03bc\u03b7\u03c7\u03b1\u03bd\u03b9\u03c3\u03bc\u03cc\u00b7 \u03b4\u03b5\u03bd \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03b9 \u03ad\u03bd\u03b1\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03c9\u03bc\u03ad\u03bd\u03bf \u03bd\u03ad\u03bf engine \u03bc\u03b5 \u03cc\u03bb\u03b1 \u03c4\u03b1 production features \u03c4\u03bf\u03c5 vLLM \u03ae \u03c4\u03bf\u03c5 Orca.<\/p>\n<\/div>\n<div class=\"td-comparison-grid\">\n<div class=\"td-platform-card\">\n<p class=\"td-platform-name\">Per-request dispatch<\/p>\n<p>\u039a\u03ac\u03b8\u03b5 \u03b5\u03bd\u03b5\u03c1\u03b3\u03cc request \u03c0\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03b9 \u03b4\u03b9\u03ba\u03cc \u03c4\u03bf\u03c5 forward pass \u03c3\u03b5 \u03ba\u03ac\u03b8\u03b5 step, \u03bf\u03c0\u03cc\u03c4\u03b5 \u03c4\u03bf host overhead \u03b1\u03c5\u03be\u03ac\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c3\u03c7\u03b5\u03b4\u03cc\u03bd \u03bc\u03b1\u03b6\u03af \u03bc\u03b5 \u03c4\u03bf batch size.<\/p>\n<\/div>\n<div class=\"td-platform-card\">\n<p class=\"td-platform-name\">Synchronized batch<\/p>\n<p>\u03a4\u03b1 requests \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03ad\u03bd\u03b1 forward pass \u03b1\u03bd\u03ac step \u03ba\u03b1\u03b9 \u03b1\u03c0\u03bf\u03c3\u03b2\u03ad\u03bd\u03bf\u03c5\u03bd \u03c4\u03bf \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c4\u03bf\u03c5 control loop \u03c3\u03b5 \u03cc\u03bb\u03bf \u03c4\u03bf batch.<\/p>\n<\/div>\n<div class=\"td-platform-card\">\n<p class=\"td-platform-name\">Slot-based design<\/p>\n<p>\u0397 \u03c0\u03c1\u03bf\u03c4\u03b5\u03b9\u03bd\u03cc\u03bc\u03b5\u03bd\u03b7 \u03b5\u03c0\u03cc\u03bc\u03b5\u03bd\u03b7 \u03ba\u03b1\u03c4\u03b5\u03cd\u03b8\u03c5\u03bd\u03c3\u03b7 \u03ba\u03c1\u03b1\u03c4\u03ac \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03ac GPU slots \u03ba\u03b1\u03b9 \u03b3\u03b5\u03bc\u03af\u03b6\u03b5\u03b9 \u03ac\u03bc\u03b5\u03c3\u03b1 \u03ad\u03bd\u03b1 slot \u03cc\u03c4\u03b1\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c4\u03bf request \u03c4\u03bf\u03c5\u00b7 \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03b5\u03b9 future work.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p>\u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b4\u03b9\u03b5\u03c5\u03ba\u03c1\u03b9\u03bd\u03af\u03b6\u03bf\u03c5\u03bd \u03cc\u03c4\u03b9 \u03c4\u03bf baseline \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03bf \u03af\u03b4\u03b9\u03bf \u03bc\u03b5 continuous batching \u03c3\u03b5 vLLM \u03ae Orca, \u03cc\u03c0\u03bf\u03c5 \u03ae\u03b4\u03b7 \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 forward pass \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03c4\u03b1 \u03b5\u03bd\u03b5\u03c1\u03b3\u03ac requests \u03ba\u03ac\u03b8\u03b5 iteration. \u0397 \u03c3\u03cd\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b1\u03c0\u03bf\u03bc\u03bf\u03bd\u03ce\u03bd\u03b5\u03b9 \u03c4\u03bf CPU dispatch overhead. \u0393\u03b9\u03b1 \u03c4\u03b7 \u03c3\u03c9\u03c3\u03c4\u03ae \u03b1\u03bd\u03ac\u03b3\u03bd\u03c9\u03c3\u03b7 \u03c4\u03c9\u03bd inference trade-offs \u03b2\u03bf\u03b7\u03b8\u03ac \u03ba\u03b1\u03b9 \u03b7 \u03b1\u03bd\u03ac\u03bb\u03c5\u03c3\u03b7 \u03b3\u03b9\u03b1 \u03c4\u03bf <a href=\"https:\/\/twodots.gr\/vllm-transformers-native-speed-ai-inference\/\">vLLM \u03ba\u03b1\u03b9 \u03c4\u03bf native-speed AI inference<\/a>.<\/p>\n<h2 id=\"cpu-bottleneck\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf H200 \u03c0\u03b5\u03c1\u03af\u03bc\u03b5\u03bd\u03b5 \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7\u03bd CPU<\/h2>\n<p>\u03a3\u03b5 profiling \u03b5\u03bd\u03cc\u03c2 request \u03bc\u03b5 budget 512 tokens, \u03b7 \u03c0\u03bb\u03ae\u03c1\u03b7\u03c2 \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae \u03b4\u03b9\u03ae\u03c1\u03ba\u03b5\u03c3\u03b5 10,28 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1. \u0391\u03c0\u03cc \u03b1\u03c5\u03c4\u03ac, 2,511 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03ae 24,4% \u03ae\u03c4\u03b1\u03bd GPU kernel execution. \u03a4\u03b1 \u03c5\u03c0\u03cc\u03bb\u03bf\u03b9\u03c0\u03b1 7,77 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03ae 75,6% \u03b1\u03c0\u03bf\u03b4\u03cc\u03b8\u03b7\u03ba\u03b1\u03bd \u03c3\u03c4\u03bf\u03bd CPU-side control loop: \u03c0\u03b1\u03c1\u03b1\u03ba\u03bf\u03bb\u03bf\u03cd\u03b8\u03b7\u03c3\u03b7 \u03ba\u03b1\u03c4\u03ac\u03c3\u03c4\u03b1\u03c3\u03b7\u03c2 blocks, \u03ba\u03b1\u03c4\u03b1\u03c3\u03ba\u03b5\u03c5\u03ae attention masks, \u03ad\u03bb\u03b5\u03b3\u03c7\u03bf\u03c2 thresholds \u03ba\u03b1\u03b9 \u03c3\u03c5\u03b3\u03c7\u03c1\u03bf\u03bd\u03b9\u03c3\u03bc\u03cc\u03c2 \u03bc\u03b5\u03c4\u03b1\u03be\u03cd forward passes.<\/p>\n<p>\u0391\u03c5\u03c4\u03cc \u03bc\u03b5\u03c4\u03b1\u03c6\u03ad\u03c1\u03b5\u03b9 \u03c4\u03b7\u03bd \u03c0\u03c1\u03bf\u03c4\u03b5\u03c1\u03b1\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7\u03c2. \u03a4\u03bf batching \u03b4\u03b5\u03bd \u03b2\u03bf\u03b7\u03b8\u03ac \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03b5\u03c0\u03b5\u03b9\u03b4\u03ae \u00ab\u03b3\u03b5\u03bc\u03af\u03b6\u03b5\u03b9\u00bb \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b1 \u03c4\u03b7\u03bd GPU \u03bc\u03b5 \u03c4\u03bf\u03bd \u03ba\u03bb\u03b1\u03c3\u03b9\u03ba\u03cc \u03c4\u03c1\u03cc\u03c0\u03bf \u03c4\u03c9\u03bd AR workloads. \u0392\u03bf\u03b7\u03b8\u03ac \u03b5\u03c0\u03b5\u03b9\u03b4\u03ae \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03b5\u03b9 \u03c4\u03bf \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c4\u03bf\u03c5 control loop \u03c3\u03b5 \u03c0\u03bf\u03bb\u03bb\u03ac requests. \u0397 \u03c0\u03b1\u03c1\u03b1\u03c4\u03ae\u03c1\u03b7\u03c3\u03b7 \u03c3\u03c5\u03bd\u03b4\u03ad\u03b5\u03c4\u03b1\u03b9 \u03bc\u03b5 tightly coupled CPU-GPU \u03c5\u03c0\u03bf\u03b4\u03bf\u03bc\u03ad\u03c2, \u03cc\u03c0\u03c9\u03c2 GH200-class superchips, \u03ba\u03b1\u03b9 \u03bc\u03b5 \u03c4\u03b5\u03c7\u03bd\u03b9\u03ba\u03ad\u03c2 dispatch elimination \u03cc\u03c0\u03c9\u03c2 CUDA Graphs. \u03a0\u03c1\u03cc\u03ba\u03b5\u03b9\u03c4\u03b1\u03b9 \u03b3\u03b9\u03b1 \u03ba\u03b1\u03c4\u03b5\u03cd\u03b8\u03c5\u03bd\u03c3\u03b7 \u03c0\u03bf\u03c5 \u03c0\u03c1\u03bf\u03ba\u03cd\u03c0\u03c4\u03b5\u03b9 \u03b1\u03c0\u03cc \u03c4\u03b7 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7, \u03cc\u03c7\u03b9 \u03b3\u03b9\u03b1 \u03bc\u03b5\u03c4\u03c1\u03b7\u03bc\u03ad\u03bd\u03b7 \u03c3\u03cd\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03bf\u03cd hardware \u03c3\u03c4\u03bf paper.<\/p>\n<p>\u03a3\u03b5 micro-experiment, \u03b7 \u03c3\u03cd\u03bb\u03bb\u03b7\u03c8\u03b7 \u03b5\u03bd\u03cc\u03c2 forward-pass call \u03bc\u03b5 torch.cuda.CUDAGraph \u03bc\u03b5\u03af\u03c9\u03c3\u03b5 \u03c4\u03bf CPU dispatch time \u03b1\u03c0\u03cc 19,4 ms \u03c3\u03b5 10,0 ms, \u03b4\u03b7\u03bb\u03b1\u03b4\u03ae \u03ba\u03b1\u03c4\u03ac 48,6%, \u03bc\u03b5 \u03bc\u03ad\u03b3\u03b9\u03c3\u03c4\u03b7 \u03b1\u03c0\u03cc\u03bb\u03c5\u03c4\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac logits 0,0. \u03a4\u03bf profiled request \u03b5\u03af\u03c7\u03b5 235 forward-pass calls. \u0391\u03bd \u03c0\u03b1\u03c1\u03cc\u03bc\u03bf\u03b9\u03b1 \u03b5\u03be\u03bf\u03b9\u03ba\u03bf\u03bd\u03cc\u03bc\u03b7\u03c3\u03b7 \u03af\u03c3\u03c7\u03c5\u03b5 \u03c3\u03b5 \u03cc\u03bb\u03b1 \u03c4\u03b1 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac KV-cache shapes, \u03b7 \u03c0\u03c1\u03bf\u03b2\u03bf\u03bb\u03ae \u03c4\u03b7\u03c2 \u03bf\u03bc\u03ac\u03b4\u03b1\u03c2 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03c6\u03b1\u03af\u03c1\u03b5\u03c3\u03b7 2,22 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03bf\u03bb\u03ad\u03c0\u03c4\u03c9\u03bd, \u03b4\u03b7\u03bb\u03b1\u03b4\u03ae 28,5% \u03c4\u03bf\u03c5 \u03c3\u03c5\u03bd\u03bf\u03bb\u03b9\u03ba\u03bf\u03cd dispatch overhead.<\/p>\n<p>\u0397 \u03b4\u03b5\u03cd\u03c4\u03b5\u03c1\u03b7 \u03b5\u03ba\u03c4\u03af\u03bc\u03b7\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 extrapolation \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 end-to-end CUDA Graph measurement. \u03a4\u03bf \u03c5\u03c0\u03cc\u03bb\u03bf\u03b9\u03c0\u03bf 71,5% \u03b2\u03c1\u03af\u03c3\u03ba\u03b5\u03c4\u03b1\u03b9 \u03ad\u03be\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf forward call, \u03c3\u03c4\u03bf\u03bd Python control flow. \u0397 \u03b5\u03c0\u03af\u03c3\u03b7\u03bc\u03b7 \u03c4\u03b5\u03ba\u03bc\u03b7\u03c1\u03af\u03c9\u03c3\u03b7 \u03c4\u03b7\u03c2 NVIDIA \u03b5\u03be\u03b7\u03b3\u03b5\u03af \u03b3\u03b9\u03b1\u03c4\u03af \u03c4\u03b1 CUDA Graphs \u03bc\u03b5\u03b9\u03ce\u03bd\u03bf\u03c5\u03bd \u03b5\u03c0\u03b1\u03bd\u03b1\u03bb\u03b1\u03bc\u03b2\u03b1\u03bd\u03cc\u03bc\u03b5\u03bd\u03bf CPU launch cost, \u03b1\u03bb\u03bb\u03ac \u03ba\u03b1\u03b9 \u03b3\u03b9\u03b1\u03c4\u03af \u03c9\u03c6\u03b5\u03bb\u03bf\u03cd\u03bd \u03bc\u03cc\u03bd\u03bf \u03cc\u03c4\u03b1\u03bd \u03c3\u03c5\u03bb\u03bb\u03b1\u03bc\u03b2\u03ac\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc CPU-bound \u03c4\u03bc\u03ae\u03bc\u03b1.<\/p>\n<h2 id=\"batching-quality\">\u0397 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c5\u03c0\u03cc batching \u03ad\u03c7\u03b5\u03b9 \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2<\/h2>\n<p>\u03a3\u03c4\u03bf single-request GSM8K \u03bc\u03b5 generation length 512, \u03bc\u03af\u03b1 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae \u03ad\u03b4\u03c9\u03c3\u03b5 exact-match accuracy 74% \u03ba\u03b1\u03b9 \u03bc\u03b9\u03b1 \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae 76%. \u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b1\u03c0\u03bf\u03b4\u03af\u03b4\u03bf\u03c5\u03bd \u03c4\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03c3\u03c4\u03bf stochastic sampling \u03bc\u03b5 temperature \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf \u03b1\u03c0\u03cc \u03bc\u03b7\u03b4\u03ad\u03bd. \u0391\u03c0\u03cc \u03c4\u03b9\u03c2 24 \u03bb\u03b1\u03bd\u03b8\u03b1\u03c3\u03bc\u03ad\u03bd\u03b5\u03c2 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c3\u03c4\u03b7 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae 74%, \u03bc\u03cc\u03bd\u03bf \u03b4\u03cd\u03bf \u03ad\u03c6\u03c4\u03b1\u03c3\u03b1\u03bd \u03c3\u03c4\u03bf \u03cc\u03c1\u03b9\u03bf \u03c4\u03c9\u03bd 512 tokens, \u03ac\u03c1\u03b1 \u03c4\u03bf truncation \u03ae\u03c4\u03b1\u03bd \u03bc\u03b9\u03ba\u03c1\u03cc\u03c2 \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03bf \u03ba\u03cd\u03c1\u03b9\u03bf\u03c2 \u03c0\u03b1\u03c1\u03ac\u03b3\u03bf\u03bd\u03c4\u03b1\u03c2 \u03c3\u03c6\u03ac\u03bb\u03bc\u03b1\u03c4\u03bf\u03c2 \u03c3\u03b5 \u03b5\u03ba\u03b5\u03af\u03bd\u03bf \u03c4\u03bf run.<\/p>\n<p>\u0393\u03b9\u03b1 \u03c4\u03bf synchronized batching, \u03c4\u03bf paper \u03b4\u03b5\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b5 \u03ac\u03bc\u03b5\u03c3\u03b7 empirical quality evaluation \u03c3\u03c4\u03b1 batch sizes 8 \u03ba\u03b1\u03b9 16. \u0391\u03bd\u03c4\u03af \u03b3\u03b9\u2019 \u03b1\u03c5\u03c4\u03cc \u03b4\u03af\u03bd\u03b5\u03b9 \u03b4\u03bf\u03bc\u03b9\u03ba\u03cc \u03b5\u03c0\u03b9\u03c7\u03b5\u03af\u03c1\u03b7\u03bc\u03b1 \u03c0\u03bf\u03c5 \u03b1\u03c0\u03b1\u03b9\u03c4\u03b5\u03af \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2: attention mask \u03ba\u03b1\u03b9 positional encoding \u03ba\u03ac\u03b8\u03b5 request \u03bd\u03b1 \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03bf\u03c5\u03bd \u03af\u03b4\u03b9\u03b1 \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc padding, \u03ba\u03ac\u03b8\u03b5 request \u03bd\u03b1 \u03ad\u03c7\u03b5\u03b9 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03bf RNG stream \u03ba\u03b1\u03b9 \u03bf\u03b9 batched attention\/sampling kernels \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03c1\u03b9\u03b8\u03bc\u03b7\u03c4\u03b9\u03ba\u03ac \u03b9\u03c3\u03bf\u03b4\u03cd\u03bd\u03b1\u03bc\u03bf\u03b9 \u03bc\u03b5 \u03c4\u03b9\u03c2 single-request \u03b5\u03ba\u03b4\u03cc\u03c3\u03b5\u03b9\u03c2.<\/p>\n<p>\u0395\u03c0\u03b5\u03b9\u03b4\u03ae \u03b7 \u03b5\u03bc\u03c0\u03b5\u03b9\u03c1\u03b9\u03ba\u03ae \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03af\u03c9\u03c3\u03b7 \u03bb\u03b5\u03af\u03c0\u03b5\u03b9, \u03bc\u03b9\u03b1 production \u03bf\u03bc\u03ac\u03b4\u03b1 \u03b4\u03b5\u03bd \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03bc\u03b5\u03c4\u03b1\u03c4\u03c1\u03ad\u03c8\u03b5\u03b9 \u03b1\u03c5\u03c4\u03ae \u03c4\u03b7\u03bd \u03c5\u03c0\u03cc\u03b8\u03b5\u03c3\u03b7 \u03c3\u03b5 \u03b5\u03b3\u03b3\u03cd\u03b7\u03c3\u03b7. Throughput regression tests \u03ba\u03b1\u03b9 quality regression tests \u03b5\u03af\u03bd\u03b1\u03b9 \u03b4\u03cd\u03bf \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ad\u03c2 \u03c0\u03cd\u03bb\u03b5\u03c2 \u03ad\u03ba\u03b4\u03bf\u03c3\u03b7\u03c2.<\/p>\n<h2 id=\"batch-timeout\">\u03a4\u03bf timeout \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03c4\u03b1\u03c4\u03b5\u03cd\u03b5\u03b9 \u03c4\u03b7 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03b7\u03c2 \u03bf\u03c5\u03c1\u03ac\u03c2<\/h2>\n<p>\u03a4\u03bf fixed-fill synchronized batching \u03b4\u03b7\u03bc\u03b9\u03bf\u03c5\u03c1\u03b3\u03b5\u03af \u03ad\u03bd\u03b1 trade-off: \u03bc\u03b5\u03b3\u03ac\u03bb\u03b7 \u03b1\u03bd\u03b1\u03bc\u03bf\u03bd\u03ae \u03b3\u03b5\u03bc\u03af\u03b6\u03b5\u03b9 \u03c4\u03bf batch \u03b1\u03bb\u03bb\u03ac \u03b1\u03c5\u03be\u03ac\u03bd\u03b5\u03b9 \u03c4\u03bf latency, \u03b5\u03bd\u03ce \u03c0\u03c1\u03cc\u03c9\u03c1\u03b7 \u03b1\u03c0\u03bf\u03c3\u03c4\u03bf\u03bb\u03ae \u03b4\u03b7\u03bc\u03b9\u03bf\u03c5\u03c1\u03b3\u03b5\u03af underfull batches \u03c0\u03bf\u03c5 \u03ba\u03bf\u03c3\u03c4\u03af\u03b6\u03bf\u03c5\u03bd \u03ad\u03bd\u03b1 \u03c0\u03bb\u03ae\u03c1\u03b5\u03c2 forward pass. \u0397 \u03b1\u03bd\u03ac\u03bb\u03c5\u03c3\u03b7 \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af B=8, \u03bc\u03ad\u03c3\u03bf measured service time S=8,74 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03b1\u03c0\u03cc \u03bf\u03ba\u03c4\u03ce batches \u03ba\u03b1\u03b9 Poisson arrivals. \u0397 full-batch capacity \u03b5\u03af\u03bd\u03b1\u03b9 \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 0,915 requests \u03b1\u03bd\u03ac \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03bf.<\/p>\n<p>\u0397 \u03c0\u03c1\u03bf\u03c3\u03bf\u03bc\u03bf\u03af\u03c9\u03c3\u03b7 20.000 requests \u03ad\u03b4\u03b5\u03b9\u03be\u03b5 U-shaped mean latency \u03c9\u03c2 \u03c0\u03c1\u03bf\u03c2 \u03c4\u03bf utilization \u03c1, \u03bc\u03b5 \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc\u03c4\u03b5\u03c1\u03bf \u03bc\u03ad\u03c3\u03bf latency \u03ba\u03bf\u03bd\u03c4\u03ac \u03c3\u03c4\u03bf \u03c1\u22480,70. \u03a3\u03b5 \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc load \u03ba\u03c5\u03c1\u03b9\u03b1\u03c1\u03c7\u03b5\u03af \u03c4\u03bf batch-fill delay, \u03bc\u03b5 \u03bc\u03ad\u03c3\u03b7 \u03b1\u03bd\u03b1\u03bc\u03bf\u03bd\u03ae (B-1)\/(2\u03bb), \u03b4\u03b7\u03bb\u03b1\u03b4\u03ae 3,5\/\u03bb \u03cc\u03c4\u03b1\u03bd B=8. \u03a3\u03b5 \u03c5\u03c8\u03b7\u03bb\u03cc load \u03ba\u03c5\u03c1\u03b9\u03b1\u03c1\u03c7\u03b5\u03af \u03b7 \u03b1\u03bd\u03b1\u03bc\u03bf\u03bd\u03ae \u03c0\u03af\u03c3\u03c9 \u03b1\u03c0\u03cc \u03c0\u03c1\u03bf\u03b7\u03b3\u03bf\u03cd\u03bc\u03b5\u03bd\u03b1 batches.<\/p>\n<p>\u0393\u03b9\u03b1 sparse arrivals, \u03bf\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03c0\u03c1\u03bf\u03c3\u03b5\u03b3\u03b3\u03af\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf expected batch size \u03c9\u03c2 1+\u03bbT. \u0391\u03c0\u03cc \u03c4\u03b7 \u03c3\u03c5\u03bd\u03b8\u03ae\u03ba\u03b7 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2 \u03c0\u03c1\u03bf\u03ba\u03cd\u03c0\u03c4\u03b5\u03b9 \u03bf \u03ba\u03b1\u03bd\u03cc\u03bd\u03b1\u03c2 T<sub>min<\/sub>=S-1\/\u03bb. \u0394\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 universal constant: \u03b5\u03be\u03b1\u03c1\u03c4\u03ac\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc arrival rate \u03ba\u03b1\u03b9 measured service time \u03c4\u03bf\u03c5 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c3\u03c5\u03c3\u03c4\u03ae\u03bc\u03b1\u03c4\u03bf\u03c2. \u039c\u03b5\u03c4\u03b1\u03b3\u03b5\u03bd\u03ad\u03c3\u03c4\u03b5\u03c1\u03bf sample 32 batches \u03ac\u03bb\u03bb\u03b1\u03be\u03b5 \u03c4\u03b7\u03bd \u03b5\u03ba\u03c4\u03b9\u03bc\u03ce\u03bc\u03b5\u03bd\u03b7 capacity \u03ba\u03b1\u03c4\u03ac \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 24%, \u03ac\u03c1\u03b1 \u03c4\u03b1 percentiles \u03ba\u03b1\u03b9 \u03b7 \u03b2\u03b1\u03b8\u03bc\u03bf\u03bd\u03cc\u03bc\u03b7\u03c3\u03b7 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf \u03b4\u03b5\u03af\u03b3\u03bc\u03b1 \u03c0\u03c1\u03b9\u03bd \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b7\u03b8\u03bf\u03cd\u03bd \u03c9\u03c2 \u03c4\u03b5\u03bb\u03b9\u03ba\u03ae production \u03c0\u03bf\u03bb\u03b9\u03c4\u03b9\u03ba\u03ae.<\/p>\n<h2 id=\"business-infrastructure\">\u03a4\u03b9 \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03b3\u03b9\u03b1 \u03bc\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 AI infrastructure<\/h2>\n<p>\u03a0\u03c1\u03ce\u03c4\u03bf\u03bd, \u03c4\u03bf benchmark \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b5\u03c0\u03b9\u03c4\u03c1\u03ad\u03c0\u03b5\u03b9 \u03c3\u03c4\u03bf task \u03bd\u03b1 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c6\u03c5\u03c3\u03b9\u03ba\u03ac. \u0394\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac, \u03c4\u03bf \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc latency \u03ba\u03b1\u03b9 \u03c4\u03bf \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc variance \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03c0\u03bf\u03c4\u03ad\u03bb\u03b5\u03c3\u03bc\u03b1 truncation. \u0394\u03b5\u03cd\u03c4\u03b5\u03c1\u03bf\u03bd, \u03b7 \u03c0\u03b1\u03c1\u03b1\u03ba\u03bf\u03bb\u03bf\u03cd\u03b8\u03b7\u03c3\u03b7 \u03bc\u03cc\u03bd\u03bf GPU utilization \u03b4\u03b5\u03bd \u03b5\u03be\u03b7\u03b3\u03b5\u03af \u03ad\u03bd\u03b1 host-side bottleneck. \u03a7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03b4\u03b9\u03ac\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03b5 wall-clock time, kernel time, control-loop time \u03ba\u03b1\u03b9 dispatch overhead.<\/p>\n<p>\u03a4\u03c1\u03af\u03c4\u03bf\u03bd, \u03c4\u03bf batching policy \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03cc\u03bd\u03bf \u03c0\u03b1\u03c1\u03ac\u03bc\u03b5\u03c4\u03c1\u03bf\u03c2 throughput. \u0395\u03c0\u03b7\u03c1\u03b5\u03ac\u03b6\u03b5\u03b9 queue stability \u03ba\u03b1\u03b9 tail latency, \u03b5\u03bd\u03ce \u03b7 \u03bf\u03c1\u03b8\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03b7\u03c2 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2 \u03b1\u03c0\u03b1\u03b9\u03c4\u03b5\u03af \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03b5\u03c0\u03b1\u03bb\u03ae\u03b8\u03b5\u03c5\u03c3\u03b7. \u03a4\u03ad\u03c4\u03b1\u03c1\u03c4\u03bf\u03bd, schedulers \u03b2\u03b1\u03c3\u03b9\u03c3\u03bc\u03ad\u03bd\u03bf\u03b9 \u03c3\u03b5 prompt length \u03ae \u03c3\u03b5 \u03c0\u03bf\u03bb\u03cd \u03c0\u03c1\u03ce\u03b9\u03bc\u03bf confidence \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03c3\u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf model \u03ba\u03b1\u03b9 task. \u03a0\u03ad\u03bc\u03c0\u03c4\u03bf\u03bd, \u03bf\u03b9 \u03c3\u03c7\u03b5\u03b4\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03ad\u03c2 \u03b5\u03c0\u03b9\u03bb\u03bf\u03b3\u03ad\u03c2 \u03c4\u03c9\u03bd AR inference engines \u03b4\u03b5\u03bd \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b8\u03b5\u03c9\u03c1\u03bf\u03cd\u03bd\u03c4\u03b1\u03b9 \u03b1\u03c5\u03c4\u03cc\u03bc\u03b1\u03c4\u03b1 \u03b2\u03ad\u03bb\u03c4\u03b9\u03c3\u03c4\u03b5\u03c2 \u03b3\u03b9\u03b1 masked diffusion.<\/p>\n<p>\u0393\u03b9\u03b1 e-commerce, marketing automation \u03ae customer-experience workloads, \u03b7 \u03b5\u03c0\u03af\u03c0\u03c4\u03c9\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 \u03bb\u03b5\u03b9\u03c4\u03bf\u03c5\u03c1\u03b3\u03b9\u03ba\u03ae. \u0391\u03bd \u03ad\u03bd\u03b1 \u03c3\u03cd\u03c3\u03c4\u03b7\u03bc\u03b1 \u03b4\u03b7\u03bc\u03b9\u03bf\u03c5\u03c1\u03b3\u03b5\u03af \u03c0\u03bf\u03bb\u03bb\u03ad\u03c2 \u03c0\u03b5\u03c1\u03b9\u03b3\u03c1\u03b1\u03c6\u03ad\u03c2 \u03c0\u03c1\u03bf\u03ca\u03cc\u03bd\u03c4\u03c9\u03bd, \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c5\u03c0\u03bf\u03c3\u03c4\u03ae\u03c1\u03b9\u03be\u03b7\u03c2 \u03ae \u03c0\u03b1\u03c1\u03b1\u03bb\u03bb\u03b1\u03b3\u03ad\u03c2 \u03c0\u03b5\u03c1\u03b9\u03b5\u03c7\u03bf\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c4\u03b1\u03c5\u03c4\u03cc\u03c7\u03c1\u03bf\u03bd\u03b1, \u03b7 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ae \u03c7\u03c9\u03c1\u03b7\u03c4\u03b9\u03ba\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b8\u03b1 \u03b5\u03be\u03b1\u03c1\u03c4\u03b7\u03b8\u03b5\u03af \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03ce\u03c2 \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03c4\u03b1 steps \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03bc\u03cc\u03bd\u03bf \u03b1\u03c0\u03cc \u03c4\u03bf \u03bc\u03ad\u03b3\u03b5\u03b8\u03bf\u03c2 \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5. \u038c\u03c0\u03c9\u03c2 \u03ba\u03b1\u03b9 \u03c3\u03c4\u03bf <a href=\"https:\/\/twodots.gr\/perissoteres-gpus-mikrotero-kv-cache-kostos-llm-serving\/\">\u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c4\u03bf\u03c5 LLM serving \u03bc\u03b5 GPUs \u03ba\u03b1\u03b9 KV cache<\/a>, \u03c4\u03bf \u03c3\u03c9\u03c3\u03c4\u03cc capacity plan \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c4\u03b9\u03c2 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ad\u03c2 distributions \u03c4\u03c9\u03bd requests \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03ad\u03bd\u03b1 demo average.<\/p>\n<div class=\"td-decision-band\">\n<p class=\"td-decision-kicker\">Production gate<\/p>\n<p class=\"td-decision-title\">\u0391\u03b3\u03bf\u03c1\u03ac\u03c3\u03c4\u03b5 compute \u03bc\u03cc\u03bd\u03bf \u03b1\u03c6\u03bf\u03cd \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b5\u03c4\u03b5 \u03c4\u03bf bottleneck \u03c4\u03b7\u03c2 \u03c0\u03bb\u03ae\u03c1\u03bf\u03c5\u03c2 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae\u03c2<\/p>\n<p>\u0391\u03bd \u03bf host control loop \u03ba\u03c1\u03b1\u03c4\u03ac \u03c4\u03b7\u03bd GPU \u03b1\u03bd\u03b5\u03bd\u03b5\u03c1\u03b3\u03ae, \u03c0\u03b5\u03c1\u03b9\u03c3\u03c3\u03cc\u03c4\u03b5\u03c1\u03b7 accelerator \u03c7\u03c9\u03c1\u03b7\u03c4\u03b9\u03ba\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b4\u03b5\u03bd \u03bb\u03cd\u03bd\u03b5\u03b9 \u03bc\u03cc\u03bd\u03b7 \u03c4\u03b7\u03c2 \u03c4\u03bf serving \u03c0\u03c1\u03cc\u03b2\u03bb\u03b7\u03bc\u03b1. \u03a0\u03c1\u03bf\u03b7\u03b3\u03bf\u03cd\u03bd\u03c4\u03b1\u03b9 trace, representative load, quality checks \u03ba\u03b1\u03b9 queue model.<\/p>\n<\/div>\n<div class=\"td-step-list\">\n<p class=\"td-step-list-title\">\u0395\u03c0\u03c4\u03ac \u03ad\u03bb\u03b5\u03b3\u03c7\u03bf\u03b9 \u03c0\u03c1\u03b9\u03bd \u03b1\u03c0\u03cc production rollout<\/p>\n<ol>\n<li><span class=\"td-step-kicker\">\u0421\u0442\u044a\u043f\u043a\u0430 1<\/span><strong>\u039f\u03c1\u03af\u03c3\u03c4\u03b5 representative requests<\/strong>\n<p>\u03a7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03ae\u03c3\u03c4\u03b5 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ad\u03c2 \u03ba\u03b1\u03c4\u03b1\u03bd\u03bf\u03bc\u03ad\u03c2 prompt, task \u03ba\u03b1\u03b9 output length \u03b1\u03bd\u03c4\u03af \u03b3\u03b9\u03b1 \u03bb\u03af\u03b3\u03b1 \u03bf\u03bc\u03bf\u03b9\u03cc\u03bc\u03bf\u03c1\u03c6\u03b1 demo prompts.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0421\u0442\u044a\u043f\u043a\u0430 2<\/span><strong>\u0391\u03c6\u03b1\u03b9\u03c1\u03ad\u03c3\u03c4\u03b5 \u03c4\u03bf truncation bias<\/strong>\n<p>\u0395\u03c0\u03b9\u03bb\u03ad\u03be\u03c4\u03b5 generation budget \u03c0\u03bf\u03c5 \u03b5\u03c0\u03b9\u03c4\u03c1\u03ad\u03c0\u03b5\u03b9 \u03c3\u03c4\u03bf task \u03bd\u03b1 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03ba\u03b1\u03b9 \u03ba\u03b1\u03c4\u03b1\u03b3\u03c1\u03ac\u03c8\u03c4\u03b5 \u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ac \u03c4\u03bf \u03c0\u03bf\u03c3\u03bf\u03c3\u03c4\u03cc \u03ba\u03bf\u03bc\u03bc\u03ad\u03bd\u03c9\u03bd \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03c9\u03bd.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0421\u0442\u044a\u043f\u043a\u0430 3<\/span><strong>\u0394\u03b9\u03b1\u03c7\u03c9\u03c1\u03af\u03c3\u03c4\u03b5 host \u03ba\u03b1\u03b9 device time<\/strong>\n<p>\u039c\u03b5\u03c4\u03c1\u03ae\u03c3\u03c4\u03b5 wall-clock, kernel execution, dispatch, mask construction, synchronization \u03ba\u03b1\u03b9 control-loop time \u03c3\u03c4\u03bf \u03af\u03b4\u03b9\u03bf trace.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0421\u0442\u044a\u043f\u043a\u0430 4<\/span><strong>\u03a3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03c4\u03b5 batching policies<\/strong>\n<p>\u0395\u03bb\u03ad\u03b3\u03be\u03c4\u03b5 single-request, per-request dispatch \u03ba\u03b1\u03b9 shared-step batching \u03bc\u03b5 \u03af\u03b4\u03b9\u03bf model, thresholds \u03ba\u03b1\u03b9 quality protocol.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0421\u0442\u044a\u043f\u043a\u0430 5<\/span><strong>\u0395\u03c0\u03b1\u03bb\u03b7\u03b8\u03b5\u03cd\u03c3\u03c4\u03b5 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b1\u03bd\u03ac batch size<\/strong>\n<p>\u03a4\u03c1\u03ad\u03be\u03c4\u03b5 repeated trials \u03bc\u03b5 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b1 RNG streams \u03ba\u03b1\u03b9 \u03b5\u03bb\u03ad\u03b3\u03be\u03c4\u03b5 numerical equivalence \u03c0\u03c1\u03b9\u03bd \u03b8\u03b5\u03c9\u03c1\u03ae\u03c3\u03b5\u03c4\u03b5 \u03b1\u03c3\u03c6\u03b1\u03bb\u03ad\u03c2 \u03c4\u03bf throughput gain.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0421\u0442\u044a\u043f\u043a\u0430 6<\/span><strong>\u0392\u03b1\u03b8\u03bc\u03bf\u03bd\u03bf\u03bc\u03ae\u03c3\u03c4\u03b5 timeout \u03ba\u03b1\u03b9 capacity<\/strong>\n<p>\u03a7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03ae\u03c3\u03c4\u03b5 \u03b1\u03c1\u03ba\u03b5\u03c4\u03ac measured batches, arrival distributions \u03ba\u03b1\u03b9 tail-latency \u03c3\u03c4\u03cc\u03c7\u03bf\u03c5\u03c2 \u03b1\u03bd\u03c4\u03af \u03b3\u03b9\u03b1 \u03ad\u03bd\u03b1\u03bd \u03b3\u03b5\u03bd\u03b9\u03ba\u03cc timeout.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">\u0421\u0442\u044a\u043f\u043a\u0430 7<\/span><strong>\u0398\u03ad\u03c3\u03c4\u03b5 release \u03ba\u03b1\u03b9 fallback \u03cc\u03c1\u03b9\u03b1<\/strong>\n<p>\u039f\u03c1\u03af\u03c3\u03c4\u03b5 \u03c0\u03cc\u03c4\u03b5 \u03c4\u03bf \u03c3\u03cd\u03c3\u03c4\u03b7\u03bc\u03b1 \u03bc\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 batch size, \u03b5\u03c0\u03b9\u03c3\u03c4\u03c1\u03ad\u03c6\u03b5\u03b9 \u03c3\u03b5 \u03b1\u03c3\u03c6\u03b1\u03bb\u03ad\u03c2 baseline \u03ae \u03b6\u03b7\u03c4\u03ac \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03c0\u03b1\u03c1\u03ad\u03bc\u03b2\u03b1\u03c3\u03b7 \u03cc\u03c4\u03b1\u03bd quality \u03ae queue stability \u03b1\u03c0\u03bf\u03ba\u03bb\u03af\u03bd\u03bf\u03c5\u03bd.<\/p>\n<\/li>\n<\/ol>\n<\/div>\n<p>\u0397 \u03bf\u03b9\u03ba\u03bf\u03bd\u03bf\u03bc\u03b9\u03ba\u03ae \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03b5\u03c0\u03af\u03c3\u03b7\u03c2 \u03bd\u03b1 \u03c3\u03c5\u03bd\u03b4\u03ad\u03b5\u03b9 \u03c4\u03bf infrastructure metric \u03bc\u03b5 \u03c4\u03b7\u03bd \u03b5\u03c1\u03b3\u03b1\u03c3\u03af\u03b1 \u03c0\u03bf\u03c5 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9. \u03a4\u03bf <a href=\"https:\/\/twodots.gr\/ai-roi-axia-ergasias\/\">AI ROI \u03b4\u03b5\u03bd \u03ba\u03c1\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 \u03bc\u03cc\u03bd\u03bf \u03b1\u03c0\u03cc tokens<\/a>, \u03b1\u03bb\u03bb\u03ac \u03b1\u03c0\u03cc \u03c4\u03bf \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03ba\u03b1\u03b9 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03ba\u03ac\u03b8\u03b5 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03c9\u03bc\u03ad\u03bd\u03bf\u03c5 business outcome.<\/p>\n<h2 id=\"limitations\">\u03a0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03bf\u03af \u03c0\u03bf\u03c5 \u03b1\u03bb\u03bb\u03ac\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf \u03b2\u03ac\u03c1\u03bf\u03c2 \u03c4\u03c9\u03bd \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b1\u03c3\u03bc\u03ac\u03c4\u03c9\u03bd<\/h2>\n<p>\u0397 \u03ad\u03c1\u03b5\u03c5\u03bd\u03b1 \u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03b5\u03b9 \u03bc\u03af\u03b1 \u03bf\u03b9\u03ba\u03bf\u03b3\u03ad\u03bd\u03b5\u03b9\u03b1 \u03b1\u03c1\u03c7\u03b9\u03c4\u03b5\u03ba\u03c4\u03bf\u03bd\u03b9\u03ba\u03ae\u03c2, LLaDA base \u03ba\u03b1\u03b9 D2F, \u03ba\u03b1\u03b9 \u03bc\u03af\u03b1 GPU, \u03c4\u03b7\u03bd H200. \u03a4\u03bf task generalization \u03ba\u03b1\u03b9 \u03c4\u03bf configuration generalization \u03b5\u03be\u03b5\u03c4\u03ac\u03c3\u03c4\u03b7\u03ba\u03b1\u03bd \u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ac, \u03cc\u03c7\u03b9 \u03bc\u03b1\u03b6\u03af. \u0394\u03b5\u03bd \u03ad\u03b3\u03b9\u03bd\u03b5 \u03ac\u03bc\u03b5\u03c3\u03b7 \u03c3\u03cd\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03bc\u03b5 \u03c5\u03c0\u03ac\u03c1\u03c7\u03bf\u03bd\u03c4\u03b1 dLLM inference engines \u03cc\u03c0\u03c9\u03c2 \u03c4\u03bf dInfer. \u0397 \u03b4\u03bf\u03bc\u03ae \u03c4\u03c9\u03bd 11 tiers \u03bc\u03b5 \u03c4\u03cd\u03c0\u03bf 178+29k \u03bc\u03b5\u03c4\u03c1\u03ae\u03b8\u03b7\u03ba\u03b5 \u03c3\u03b5 \u03ad\u03bd\u03b1 \u03bc\u03cc\u03bd\u03bf \u03b6\u03b5\u03cd\u03b3\u03bf\u03c2 thresholds \u03ba\u03b1\u03b9 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b5\u03b9\u03b4\u03b9\u03ba\u03ae \u03b3\u03b9\u03b1 \u03b1\u03c5\u03c4\u03ae \u03c4\u03b7 \u03c1\u03cd\u03b8\u03bc\u03b9\u03c3\u03b7.<\/p>\n<p>\u0391\u03c1\u03ba\u03b5\u03c4\u03ac \u03b1\u03c0\u03bf\u03c4\u03b5\u03bb\u03ad\u03c3\u03bc\u03b1\u03c4\u03b1 \u03c0\u03c1\u03bf\u03ad\u03c1\u03c7\u03bf\u03bd\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc single runs \u03c7\u03c9\u03c1\u03af\u03c2 repeated trials. \u0397 scheduling simulation \u03be\u03b5\u03ba\u03af\u03bd\u03b7\u03c3\u03b5 \u03b1\u03c0\u03cc \u03bc\u03cc\u03bb\u03b9\u03c2 \u03bf\u03ba\u03c4\u03ce measured batches, \u03b5\u03bd\u03ce \u03ad\u03bd\u03b1 \u03bc\u03b5\u03c4\u03b1\u03b3\u03b5\u03bd\u03ad\u03c3\u03c4\u03b5\u03c1\u03bf run 32 batches \u03bc\u03b5\u03c4\u03b1\u03ba\u03af\u03bd\u03b7\u03c3\u03b5 \u03c4\u03b7\u03bd capacity \u03ba\u03b1\u03c4\u03ac 24%. \u03a4\u03bf slot-based design \u03b4\u03b5\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03b8\u03b7\u03ba\u03b5 \u03c3\u03b5 \u03c0\u03bb\u03ae\u03c1\u03c9\u03c2 \u03c3\u03c9\u03c3\u03c4\u03ae \u03c5\u03bb\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03b7 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c3\u03b5 batch sizes 8 \u03ba\u03b1\u03b9 16 \u03b4\u03b5\u03bd \u03b5\u03c0\u03b1\u03bb\u03b7\u03b8\u03b5\u03cd\u03c4\u03b7\u03ba\u03b5 \u03b5\u03bc\u03c0\u03b5\u03b9\u03c1\u03b9\u03ba\u03ac.<\/p>\n<p>\u039f\u03b9 \u03c0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03bf\u03af \u03b4\u03b5\u03bd \u03b1\u03ba\u03c5\u03c1\u03ce\u03bd\u03bf\u03c5\u03bd \u03c4\u03b1 \u03b5\u03c5\u03c1\u03ae\u03bc\u03b1\u03c4\u03b1\u00b7 \u03ba\u03b1\u03b8\u03bf\u03c1\u03af\u03b6\u03bf\u03c5\u03bd \u03c0\u03bf\u03cd \u03c4\u03b5\u03bb\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 \u03b7 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03c0\u03bf\u03cd \u03b1\u03c1\u03c7\u03af\u03b6\u03b5\u03b9 \u03b7 \u03c3\u03c7\u03b5\u03b4\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03ae \u03c5\u03c0\u03cc\u03b8\u03b5\u03c3\u03b7. \u03a4\u03bf \u03b9\u03c3\u03c7\u03c5\u03c1\u03cc\u03c4\u03b5\u03c1\u03bf \u03c3\u03c5\u03bc\u03c0\u03ad\u03c1\u03b1\u03c3\u03bc\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03b5\u03b8\u03bf\u03b4\u03bf\u03bb\u03bf\u03b3\u03b9\u03ba\u03cc: \u03c4\u03bf serving \u03b5\u03bd\u03cc\u03c2 \u03bd\u03ad\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c7\u03b1\u03c1\u03b1\u03ba\u03c4\u03b7\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c0\u03ac\u03bd\u03c9 \u03c3\u03b5 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc hardware, \u03bc\u03b5 budgets \u03c0\u03bf\u03c5 \u03b4\u03b5\u03bd \u03c0\u03b1\u03c1\u03b1\u03bc\u03bf\u03c1\u03c6\u03ce\u03bd\u03bf\u03c5\u03bd \u03c4\u03bf task, \u03bc\u03b5 profiling host \u03ba\u03b1\u03b9 accelerator \u03ba\u03b1\u03b9 \u03bc\u03b5 \u03c3\u03b1\u03c6\u03ae \u03b4\u03b9\u03ac\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03b5 measured result, analytical rule \u03ba\u03b1\u03b9 future design.<\/p>\n<section class=\"td-service-cta\">\n<div class=\"td-service-cta-content\">\n<p class=\"td-service-cta-kicker\">AI \u03c5\u03c0\u03bf\u03b4\u03bf\u03bc\u03ae \u03bc\u03b5 \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b9\u03bc\u03bf capacity<\/p>\n<p class=\"td-service-cta-title\">\u03a3\u03c7\u03b5\u03b4\u03b9\u03ac\u03c3\u03c4\u03b5 \u03c4\u03bf serving \u03b3\u03cd\u03c1\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc workload \u03c4\u03b7\u03c2 \u03b5\u03c0\u03b9\u03c7\u03b5\u03af\u03c1\u03b7\u03c3\u03b7\u03c2<\/p>\n<p>\u0397 TWO DOTS \u03c7\u03b1\u03c1\u03c4\u03bf\u03b3\u03c1\u03b1\u03c6\u03b5\u03af requests, model calls, latency, batching, monitoring, \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03ba\u03b1\u03b9 fallback \u03ce\u03c3\u03c4\u03b5 \u03ad\u03bd\u03b1 AI workflow \u03bd\u03b1 \u03b1\u03be\u03b9\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03c3\u03c9\u03c3\u03c4\u03ac CPU, GPU \u03ba\u03b1\u03b9 \u03b4\u03b9\u03b1\u03b8\u03ad\u03c3\u03b9\u03bc\u03bf budget.<\/p>\n<div class=\"td-service-cta-actions\"><a class=\"td-service-cta-button\" href=\"https:\/\/twodots.gr\/aftomatismoi-epicheiriseon-ai\/\">\u0394\u03b5\u03af\u03c4\u03b5 \u0391\u03c5\u03c4\u03bf\u03bc\u03b1\u03c4\u03b9\u03c3\u03bc\u03bf\u03cd\u03c2 \u0395\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03ae\u03c3\u03b5\u03c9\u03bd &amp; AI<\/a><\/div>\n<\/div>\n<\/section>\n<section id=\"sychnes-erotiseis\" class=\"td-faq-section\">\n<div class=\"td-faq\">\n<p class=\"td-faq-heading\">\u0427\u0435\u0441\u0442\u043e \u0437\u0430\u0434\u0430\u0432\u0430\u043d\u0438 \u0432\u044a\u043f\u0440\u043e\u0441\u0438<\/p>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a4\u03b9 \u03b5\u03af\u03bd\u03b1\u03b9 \u03ad\u03bd\u03b1 masked diffusion LLM;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u0395\u03af\u03bd\u03b1\u03b9 \u03b3\u03bb\u03c9\u03c3\u03c3\u03b9\u03ba\u03cc \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03c0\u03bf\u03c5 \u03c0\u03b1\u03c1\u03ac\u03b3\u03b5\u03b9 \u03ba\u03b5\u03af\u03bc\u03b5\u03bd\u03bf \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03bf\u03bd\u03c4\u03b1\u03c2 \u03c3\u03c4\u03b1\u03b4\u03b9\u03b1\u03ba\u03ac masked \u03b8\u03ad\u03c3\u03b5\u03b9\u03c2 \u03bc\u03ad\u03c3\u03c9 denoising steps, \u03b1\u03bd\u03c4\u03af \u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03b8\u03ad\u03c4\u03b5\u03b9 \u03b1\u03c5\u03c3\u03c4\u03b7\u03c1\u03ac \u03ad\u03bd\u03b1 token \u03ba\u03ac\u03b8\u03b5 \u03c6\u03bf\u03c1\u03ac \u03cc\u03c0\u03c9\u03c2 \u03ad\u03bd\u03b1 autoregressive \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0393\u03b9\u03b1\u03c4\u03af \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03c3\u03c4\u03b7\u03ba\u03b1\u03bd \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 11 \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1\u03c2;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a3\u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf D2F setup, \u03bf \u03ba\u03b1\u03bd\u03cc\u03bd\u03b1\u03c2 \u03c0\u03c1\u03bf\u03c3\u03b8\u03ae\u03ba\u03b7\u03c2 blocks \u03ba\u03b1\u03b9 \u03c4\u03b1 thresholds \u03bf\u03b4\u03ae\u03b3\u03b7\u03c3\u03b1\u03bd \u03c3\u03b5 step counts \u03b1\u03c0\u03cc 178 \u03ad\u03c9\u03c2 468 \u03b1\u03bd\u03ac 29 steps. \u039f\u03b9 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03b5\u03c2 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03ad\u03c2 \u03c0\u03b9\u03b8\u03b1\u03bd\u03cc\u03c4\u03b1\u03c4\u03b1 \u03b5\u03be\u03b1\u03c1\u03c4\u03ce\u03bd\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03c4\u03b7 \u03c1\u03cd\u03b8\u03bc\u03b9\u03c3\u03b7.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u039c\u03c0\u03bf\u03c1\u03b5\u03af \u03bf scheduler \u03bd\u03b1 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c8\u03b5\u03b9 \u03c4\u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b1\u03c0\u03cc \u03c4\u03bf prompt;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u038c\u03c7\u03b9 \u03b1\u03be\u03b9\u03cc\u03c0\u03b9\u03c3\u03c4\u03b1 \u03bc\u03b5 \u03c4\u03b1 \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 signals \u03c0\u03bf\u03c5 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ac\u03c3\u03c4\u03b7\u03ba\u03b1\u03bd. \u0397 \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b7 \u03c4\u03b9\u03bc\u03ae \u03ae\u03c4\u03b1\u03bd R\u00b2=0,150 \u03b3\u03b9\u03b1 prompt length \u03c3\u03c4\u03bf GSM8K, \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03c1\u03b1\u03ba\u03c4\u03b9\u03ba\u03cc \u03ba\u03b1\u03c4\u03ce\u03c6\u03bb\u03b9 0,5 \u03c4\u03b7\u03c2 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7\u03c2.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03b1 \u03bc\u03b9\u03ba\u03c1\u03ac generation budgets \u03ae\u03c4\u03b1\u03bd \u03c0\u03b1\u03c1\u03b1\u03c0\u03bb\u03b1\u03bd\u03b7\u03c4\u03b9\u03ba\u03ac;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a3\u03c4\u03b1 128 \u03ba\u03b1\u03b9 256 tokens \u03ba\u03cc\u03c0\u03b7\u03ba\u03b5 \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b1 \u03c4\u03bf 100% \u03ba\u03b1\u03b9 \u03c4\u03bf 83% \u03c4\u03c9\u03bd requests. \u03a4\u03bf benchmark \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b5 \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 truncation \u03ba\u03b1\u03b9 \u03ad\u03ba\u03c1\u03c5\u03c8\u03b5 \u03c4\u03bf \u03c6\u03c5\u03c3\u03b9\u03ba\u03cc latency variance \u03c4\u03bf\u03c5 workload.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0391\u03c0\u03cc \u03c0\u03bf\u03cd \u03c0\u03c1\u03bf\u03ad\u03ba\u03c5\u03c8\u03b5 \u03c4\u03bf 16\u00d7 throughput;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a3\u03c4\u03bf batch size 16, \u03c4\u03bf synchronized batching \u03bc\u03bf\u03b9\u03c1\u03ac\u03c3\u03c4\u03b7\u03ba\u03b5 \u03ad\u03bd\u03b1 forward pass \u03b1\u03bd\u03ac denoising step \u03b3\u03b9\u03b1 \u03cc\u03bb\u03bf \u03c4\u03bf batch \u03ba\u03b1\u03b9 \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 1,939 requests\/s \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 0,121 requests\/s \u03c3\u03c4\u03bf single-request baseline.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03b7 GPU \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03c3\u03b7\u03bc\u03b1\u03bd\u03c4\u03b9\u03ba\u03ae;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u038c\u03c7\u03b9. \u03a3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03c3\u03c4\u03bf \u03bc\u03b5\u03c4\u03c1\u03b7\u03bc\u03ad\u03bd\u03bf setup \u03c4\u03bf \u03ac\u03bc\u03b5\u03c3\u03bf bottleneck \u03ae\u03c4\u03b1\u03bd \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03bf CPU-side control loop. \u0386\u03bb\u03bb\u03b1 models, engines \u03ba\u03b1\u03b9 hardware \u03bc\u03c0\u03bf\u03c1\u03bf\u03cd\u03bd \u03bd\u03b1 \u03ad\u03c7\u03bf\u03c5\u03bd \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03cc breakdown.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0391\u03c0\u03bf\u03b4\u03b5\u03af\u03c7\u03b8\u03b7\u03ba\u03b5 \u03cc\u03c4\u03b9 \u03c4\u03bf batching \u03b4\u03b5\u03bd \u03bc\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u038c\u03c7\u03b9 \u03c0\u03bb\u03ae\u03c1\u03c9\u03c2. \u03a4\u03bf paper \u03c0\u03b1\u03c1\u03ad\u03c7\u03b5\u03b9 \u03b4\u03bf\u03bc\u03b9\u03ba\u03cc \u03b5\u03c0\u03b9\u03c7\u03b5\u03af\u03c1\u03b7\u03bc\u03b1 \u03bc\u03b5 \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2, \u03b1\u03bb\u03bb\u03ac \u03b4\u03b5\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b5 \u03ac\u03bc\u03b5\u03c3\u03b7 quality evaluation \u03c3\u03c4\u03b1 batch sizes 8 \u03ba\u03b1\u03b9 16.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a0\u03bf\u03b9\u03bf \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03bf \u03b2\u03b1\u03c3\u03b9\u03ba\u03cc production takeaway;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u039d\u03b1 \u03bc\u03b5\u03c4\u03c1\u03ac\u03c4\u03b5 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc workload \u03c3\u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc hardware, \u03bd\u03b1 \u03b1\u03c0\u03bf\u03c6\u03b5\u03cd\u03b3\u03b5\u03c4\u03b5 truncation-biased benchmarks \u03ba\u03b1\u03b9 \u03bd\u03b1 \u03c3\u03c7\u03b5\u03b4\u03b9\u03ac\u03b6\u03b5\u03c4\u03b5 batching \u03ba\u03b1\u03b9 scheduling \u03c3\u03cd\u03bc\u03c6\u03c9\u03bd\u03b1 \u03bc\u03b5 \u03c4\u03bf computational pattern \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5.<\/p>\n<\/div>\n<\/details>\n<\/div>\n<\/section>\n<div class=\"td-source-list\">\n<p id=\"piges\" class=\"td-source-list-title\">\u0418\u0437\u0442\u043e\u0447\u043d\u0438\u0446\u0438<\/p>\n<ul>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2608.23807\" target=\"_blank\" rel=\"noopener\">Amin, Afroz, Moghadampanah \u03ba\u03b1\u03b9 Nikolopoulos \u2014 Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware<\/a><\/li>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2508.09192\" target=\"_blank\" rel=\"noopener\">Wang et al. \u2014 Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing<\/a><\/li>\n<li><a href=\"https:\/\/docs.nvidia.com\/cuda\/cuda-programming-guide\/04-special-topics\/cuda-graphs.html\" target=\"_blank\" rel=\"noopener\">NVIDIA CUDA Programming Guide \u2014 CUDA Graphs \u03ba\u03b1\u03b9 \u03bc\u03b5\u03af\u03c9\u03c3\u03b7 \u03c4\u03bf\u03c5 CPU launch overhead<\/a><\/li>\n<\/ul>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>\u03a4\u03b1 masked diffusion LLMs \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03bf\u03c5\u03bd \u03c0\u03bf\u03bb\u03bb\u03ac tokens \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03b1, \u03b1\u03bb\u03bb\u03ac \u03c4\u03bf serving \u03c4\u03bf\u03c5\u03c2 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 profiling CPU, GPU, batching \u03ba\u03b1\u03b9 generation budgets.<\/p>","protected":false},"author":1,"featured_media":106447,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_gspb_post_css":"","footnotes":""},"categories":[199],"tags":[8988,7565,21047,20982,21046],"class_list":["post-98378","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-techniti-noimosyni","tag-ai-infrastructure","tag-artificial-intelligence-2","tag-gpu-computing","tag-llm-serving","tag-masked-diffusion-llm"],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":8}},"_links":{"self":[{"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/posts\/98378","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/comments?post=98378"}],"version-history":[{"count":1,"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/posts\/98378\/revisions"}],"predecessor-version":[{"id":106448,"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/posts\/98378\/revisions\/106448"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/media\/106447"}],"wp:attachment":[{"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/media?parent=98378"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/categories?post=98378"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/twodots.gr\/bg\/wp-json\/wp\/v2\/tags?post=98378"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}