{"id":98378,"date":"2026-10-11T18:54:20","date_gmt":"2026-10-11T15:54:20","guid":{"rendered":"https:\/\/twodots.gr\/?p=98378"},"modified":"2026-10-11T18:54:20","modified_gmt":"2026-10-11T15:54:20","slug":"difuzie-mascata-llm-uri-gatuire-cpu-gpu","status":"publish","type":"post","link":"https:\/\/twodots.gr\/ro\/masked-diffusion-llms-bottleneck-cpu-gpu\/","title":{"rendered":"LLM-uri cu difuzie mascat\u0103: de ce GPU-ul nu a reprezentat un punct de blocaj"},"content":{"rendered":"<div class=\"td-article-lede\">\n<p><strong>R\u0103spunsul mai \u00eent\u00e2i:<\/strong> \u03c3\u03c4\u03bf \u03bc\u03b5\u03c4\u03c1\u03b7\u03bc\u03ad\u03bd\u03bf masked diffusion LLM serving, \u03b7 H200 \u03b4\u03b5\u03bd \u03ae\u03c4\u03b1\u03bd \u03c4\u03bf \u03ba\u03cd\u03c1\u03b9\u03bf \u03cc\u03c1\u03b9\u03bf. \u039c\u03cc\u03bb\u03b9\u03c2 \u03c4\u03bf 24,4% \u03c4\u03bf\u03c5 \u03c3\u03c5\u03bd\u03bf\u03bb\u03b9\u03ba\u03bf\u03cd \u03c7\u03c1\u03cc\u03bd\u03bf\u03c5 \u03ae\u03c4\u03b1\u03bd GPU kernel execution, \u03b5\u03bd\u03ce \u03c4\u03bf 75,6% \u03ba\u03b1\u03c4\u03b1\u03bd\u03b1\u03bb\u03ce\u03b8\u03b7\u03ba\u03b5 \u03c3\u03c4\u03bf\u03bd CPU-side control loop \u03ba\u03b1\u03b9 \u03c3\u03c4\u03bf dispatch.<\/p>\n<\/div>\n<p>Modelele LLM cu difuzie mascat\u0103 dezv\u0103luie mai multe tokenuri simultan, dar punerea lor \u00een func\u021biune necesit\u0103 profilarea resurselor CPU, GPU, precum \u0219i stabilirea bugetelor pentru procesarea \u00een loturi \u0219i generarea de con\u021binut.<\/p>\n<p>\u0397 \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03b7 \u03b1\u03c0\u03bf\u03ba\u03ac\u03bb\u03c5\u03c8\u03b7 tokens \u03b4\u03b5\u03bd \u03b5\u03b3\u03b3\u03c5\u03ac\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03bc\u03cc\u03bd\u03b7 \u03c4\u03b7\u03c2 \u03c5\u03c8\u03b7\u03bb\u03cc throughput: \u03b1\u03bd \u03bf host \u03b4\u03b5\u03bd \u03c0\u03c1\u03bf\u03bb\u03b1\u03b2\u03b1\u03af\u03bd\u03b5\u03b9 \u03bd\u03b1 \u03bf\u03c1\u03b3\u03b1\u03bd\u03ce\u03c3\u03b5\u03b9 \u03c4\u03b1 denoising steps, \u03b7 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ae GPU \u03c0\u03b5\u03c1\u03b9\u03bc\u03ad\u03bd\u03b5\u03b9.<\/p>\n<p>\u0397 \u03b5\u03c1\u03b3\u03b1\u03c3\u03af\u03b1 <em>Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware<\/em> \u03c4\u03c9\u03bd Farhana Amin, Sabiha Afroz, Mona Moghadampanah \u03ba\u03b1\u03b9 Dimitrios S. Nikolopoulos \u03b5\u03be\u03b5\u03c4\u03ac\u03b6\u03b5\u03b9 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 \u03b1\u03c5\u03c4\u03cc \u03c4\u03bf serving workload. \u03a7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af LLaDA-8B-Instruct \u03bc\u03b5 D2F LoRA adapter, \u03c4\u03b1 benchmarks GSM8K \u03ba\u03b1\u03b9 HumanEval \u03ba\u03b1\u03b9 \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c3\u03b5 \u03bc\u03af\u03b1 NVIDIA H200.<\/p>\n<p>\u03a4\u03bf \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b7\u03c3\u03b9\u03b1\u03ba\u03cc \u03c3\u03c5\u03bc\u03c0\u03ad\u03c1\u03b1\u03c3\u03bc\u03b1 \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03b7 GPU \u03c0\u03b1\u03cd\u03b5\u03b9 \u03bd\u03b1 \u03ad\u03c7\u03b5\u03b9 \u03c3\u03b7\u03bc\u03b1\u03c3\u03af\u03b1. \u0395\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03ad\u03bd\u03b1 capacity plan \u03b3\u03b9\u03b1 \u03bd\u03ad\u03b1 \u03b1\u03c1\u03c7\u03b9\u03c4\u03b5\u03ba\u03c4\u03bf\u03bd\u03b9\u03ba\u03ae \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae\u03c2 \u03b4\u03b5\u03bd \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b2\u03b1\u03c3\u03b9\u03c3\u03c4\u03b5\u03af \u03bc\u03cc\u03bd\u03bf \u03c3\u03b5 FLOPS, VRAM \u03ae \u03bc\u03ad\u03c3\u03bf latency. \u03a7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 profiling \u03bf\u03bb\u03cc\u03ba\u03bb\u03b7\u03c1\u03b7\u03c2 \u03c4\u03b7\u03c2 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae\u03c2, generation budgets \u03c0\u03bf\u03c5 \u03b4\u03b5\u03bd \u03ba\u03cc\u03b2\u03bf\u03c5\u03bd \u03c4\u03b5\u03c7\u03bd\u03b7\u03c4\u03ac \u03c4\u03b9\u03c2 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 batching policy \u03c0\u03c1\u03bf\u03c3\u03b1\u03c1\u03bc\u03bf\u03c3\u03bc\u03ad\u03bd\u03b7 \u03c3\u03c4\u03bf computational pattern \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5.<\/p>\n<div class=\"td-article-toc\">\n<div class=\"td-toc-title\">Cuprins<\/div>\n<ul>\n<li><a href=\"#parallilo-denoising\">\u0391\u03c0\u03cc \u03c4\u03bf token-by-token \u03c3\u03c4\u03bf \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03bf denoising<\/a><\/li>\n<li><a href=\"#difficulty-tiers\">\u0397 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 11 \u03b4\u03b9\u03b1\u03ba\u03c1\u03b9\u03c4\u03ac \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1<\/a><\/li>\n<li><a href=\"#prompt-routing\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf prompt length \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03b3\u03b9\u03b1 \u03ad\u03be\u03c5\u03c0\u03bd\u03bf routing<\/a><\/li>\n<li><a href=\"#generation-budgets\">\u03a4\u03b1 \u03bc\u03b9\u03ba\u03c1\u03ac generation budgets \u03ba\u03c1\u03cd\u03b2\u03bf\u03c5\u03bd \u03c4\u03bf variance<\/a><\/li>\n<li><a href=\"#humaneval\">\u03a4\u03b9 \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03af\u03c9\u03c3\u03b5 \u03c4\u03bf HumanEval<\/a><\/li>\n<li><a href=\"#synchronized-batching\">Synchronized batching: 16\u00d7 throughput \u03bc\u03b5 \u03ba\u03bf\u03b9\u03bd\u03cc forward pass<\/a><\/li>\n<li><a href=\"#cpu-bottleneck\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf H200 \u03c0\u03b5\u03c1\u03af\u03bc\u03b5\u03bd\u03b5 \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7\u03bd CPU<\/a><\/li>\n<li><a href=\"#batching-quality\">\u0397 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c5\u03c0\u03cc batching \u03ad\u03c7\u03b5\u03b9 \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2<\/a><\/li>\n<li><a href=\"#batch-timeout\">\u03a4\u03bf timeout \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03c4\u03b1\u03c4\u03b5\u03cd\u03b5\u03b9 \u03c4\u03b7 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03b7\u03c2 \u03bf\u03c5\u03c1\u03ac\u03c2<\/a><\/li>\n<li><a href=\"#business-infrastructure\">\u03a4\u03b9 \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03b3\u03b9\u03b1 \u03bc\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 AI infrastructure<\/a><\/li>\n<li><a href=\"#limitations\">\u03a0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03bf\u03af \u03c0\u03bf\u03c5 \u03b1\u03bb\u03bb\u03ac\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf \u03b2\u03ac\u03c1\u03bf\u03c2 \u03c4\u03c9\u03bd \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b1\u03c3\u03bc\u03ac\u03c4\u03c9\u03bd<\/a><\/li>\n<\/ul>\n<\/div>\n<h2 id=\"parallilo-denoising\">\u0391\u03c0\u03cc \u03c4\u03bf token-by-token \u03c3\u03c4\u03bf \u03c0\u03b1\u03c1\u03ac\u03bb\u03bb\u03b7\u03bb\u03bf denoising<\/h2>\n<p>\u0388\u03bd\u03b1 autoregressive \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03c0\u03b1\u03c1\u03ac\u03b3\u03b5\u03b9 \u03c4\u03bf \u03b5\u03c0\u03cc\u03bc\u03b5\u03bd\u03bf token \u03bc\u03b5 \u03b2\u03ac\u03c3\u03b7 \u03cc\u03c3\u03b1 \u03ad\u03c7\u03bf\u03c5\u03bd \u03c0\u03c1\u03bf\u03b7\u03b3\u03b7\u03b8\u03b5\u03af. \u0388\u03bd\u03b1 masked diffusion LLM \u03be\u03b5\u03ba\u03b9\u03bd\u03ac \u03bc\u03b5 \u03ba\u03b1\u03bb\u03c5\u03bc\u03bc\u03ad\u03bd\u03b5\u03c2 \u03b8\u03ad\u03c3\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 \u03c4\u03b9\u03c2 \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03b5\u03b9 \u03c3\u03c4\u03b1\u03b4\u03b9\u03b1\u03ba\u03ac \u03bc\u03ad\u03c3\u03b1 \u03b1\u03c0\u03cc denoising steps. \u03a3\u03c4\u03b7\u03bd \u03c5\u03bb\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7 D2F \u03c4\u03b7\u03c2 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7\u03c2, \u03b7 \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae \u03bf\u03c1\u03b3\u03b1\u03bd\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 blocks. \u039d\u03ad\u03bf block \u03b1\u03c1\u03c7\u03af\u03b6\u03b5\u03b9 \u03bc\u03cc\u03bd\u03bf \u03cc\u03c4\u03b1\u03bd \u03c4\u03bf \u03c0\u03c1\u03bf\u03b7\u03b3\u03bf\u03cd\u03bc\u03b5\u03bd\u03bf \u03c0\u03b5\u03c1\u03ac\u03c3\u03b5\u03b9 \u03ad\u03bd\u03b1 \u03ba\u03b1\u03b8\u03bf\u03c1\u03b9\u03c3\u03bc\u03ad\u03bd\u03bf \u03cc\u03c1\u03b9\u03bf \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b7\u03c2.<\/p>\n<p>\u0397 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03b1\u03c5\u03c4\u03ae \u03b1\u03bb\u03bb\u03ac\u03b6\u03b5\u03b9 \u03c4\u03bf \u03c3\u03c7\u03ae\u03bc\u03b1 \u03c4\u03bf\u03c5 computational work \u03ba\u03b1\u03b9, \u03b5\u03c0\u03bf\u03bc\u03ad\u03bd\u03c9\u03c2, \u03c4\u03bf \u03c0\u03ce\u03c2 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b3\u03af\u03bd\u03bf\u03bd\u03c4\u03b1\u03b9 batching, admission \u03ba\u03b1\u03b9 eviction. \u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b4\u03b5\u03bd \u03c0\u03b1\u03c1\u03bf\u03c5\u03c3\u03b9\u03ac\u03b6\u03bf\u03c5\u03bd \u03ad\u03bd\u03b1\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03c9\u03bc\u03ad\u03bd\u03bf \u03b1\u03bd\u03c4\u03b1\u03b3\u03c9\u03bd\u03b9\u03c3\u03c4\u03b9\u03ba\u03cc inference engine. \u03a0\u03c1\u03ce\u03c4\u03b1 \u03c7\u03b1\u03c1\u03c4\u03bf\u03b3\u03c1\u03b1\u03c6\u03bf\u03cd\u03bd \u03c0\u03cc\u03c3\u03bf \u03bc\u03b5\u03c4\u03b1\u03b2\u03ac\u03bb\u03bb\u03b5\u03c4\u03b1\u03b9 \u03c4\u03bf \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03b1\u03bd\u03ac request, \u03b1\u03bd \u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c0\u03b5\u03c4\u03b1\u03b9 \u03c0\u03c1\u03b9\u03bd \u03b1\u03c1\u03c7\u03af\u03c3\u03b5\u03b9 \u03b7 \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae, \u03c0\u03cc\u03c3\u03bf \u03c0\u03b1\u03c1\u03b1\u03bc\u03bf\u03c1\u03c6\u03ce\u03bd\u03b5\u03b9 \u03c4\u03bf generation budget \u03c4\u03b9\u03c2 \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b5\u03b9\u03c2 \u03ba\u03b1\u03b9 \u03c0\u03bf\u03cd \u03ba\u03b1\u03c4\u03b1\u03bd\u03b1\u03bb\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ac \u03bf \u03c7\u03c1\u03cc\u03bd\u03bf\u03c2.<\/p>\n<aside class=\"td-article-note\"><strong>\u0397 \u03c3\u03b5\u03b9\u03c1\u03ac \u03c4\u03b7\u03c2 \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7\u03c2 \u03ad\u03c7\u03b5\u03b9 \u03c3\u03b7\u03bc\u03b1\u03c3\u03af\u03b1.<\/strong> \u0391\u03bd \u03bc\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03b1\u03b3\u03bf\u03c1\u03ac\u03c3\u03b5\u03b9 \u03c0\u03b5\u03c1\u03b9\u03c3\u03c3\u03cc\u03c4\u03b5\u03c1\u03b5\u03c2 GPUs \u03c0\u03c1\u03b9\u03bd \u03b4\u03b9\u03b1\u03c7\u03c9\u03c1\u03af\u03c3\u03b5\u03b9 kernel time, dispatch time \u03ba\u03b1\u03b9 control-loop time, \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b1\u03c5\u03be\u03ae\u03c3\u03b5\u03b9 \u03c4\u03bf \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c7\u03c9\u03c1\u03af\u03c2 \u03bd\u03b1 \u03b1\u03c6\u03b1\u03b9\u03c1\u03ad\u03c3\u03b5\u03b9 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc bottleneck.<\/aside>\n<p>\u0397 \u03b1\u03c1\u03c7\u03ae \u03b9\u03c3\u03c7\u03cd\u03b5\u03b9 \u03c0\u03ad\u03c1\u03b1 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf paper. \u038c\u03c0\u03c9\u03c2 \u03b4\u03b5\u03af\u03c7\u03bd\u03b5\u03b9 \u03ba\u03b1\u03b9 \u03b7 \u03b1\u03bd\u03ac\u03bb\u03c5\u03c3\u03b7 \u03b3\u03b9\u03b1 \u03c4\u03bf <a href=\"https:\/\/twodots.gr\/llm4llm-gpu-kernel-dokimi-pragmatiko-montelo\/\">\u03b3\u03c1\u03ae\u03b3\u03bf\u03c1\u03bf GPU kernel \u03bc\u03ad\u03c3\u03b1 \u03c3\u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf<\/a>, \u03ad\u03bd\u03b1 \u03c4\u03bf\u03c0\u03b9\u03ba\u03cc microbenchmark \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03cc\u03c4\u03b1\u03bd \u03bf end-to-end \u03c7\u03c1\u03cc\u03bd\u03bf\u03c2 \u03ba\u03b1\u03b8\u03bf\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03bf\u03bb\u03cc\u03ba\u03bb\u03b7\u03c1\u03bf \u03c4\u03bf serving stack.<\/p>\n<h2 id=\"difficulty-tiers\">\u0397 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03b5 11 \u03b4\u03b9\u03b1\u03ba\u03c1\u03b9\u03c4\u03ac \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1<\/h2>\n<p>\u03a3\u03b5 100 requests \u03c4\u03bf\u03c5 GSM8K, \u03bc\u03b5 thresholds \u03c4<sub>add<\/sub>=0,5 \u03ba\u03b1\u03b9 \u03c4<sub>decode<\/sub>=0,9, \u03bf \u03b1\u03c1\u03b9\u03b8\u03bc\u03cc\u03c2 \u03c4\u03c9\u03bd denoising steps \u03c0\u03ae\u03c1\u03b5 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 11 \u03c4\u03b9\u03bc\u03ad\u03c2: \u03b1\u03c0\u03cc 178 \u03ad\u03c9\u03c2 468, \u03bc\u03b5 \u03b1\u03c0\u03cc\u03c3\u03c4\u03b1\u03c3\u03b7 29 steps \u03bc\u03b5\u03c4\u03b1\u03be\u03cd \u03b4\u03b9\u03b1\u03b4\u03bf\u03c7\u03b9\u03ba\u03ce\u03bd \u03b5\u03c0\u03b9\u03c0\u03ad\u03b4\u03c9\u03bd. \u03a4\u03bf \u03bc\u03bf\u03c4\u03af\u03b2\u03bf \u03c0\u03c1\u03bf\u03ba\u03cd\u03c0\u03c4\u03b5\u03b9 \u03b1\u03c0\u03cc \u03c4\u03bf\u03bd \u03ba\u03b1\u03bd\u03cc\u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03b8\u03ae\u03ba\u03b7\u03c2 blocks \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03b1\u03c0\u03cc \u03c4\u03c5\u03c7\u03b1\u03af\u03bf \u03b8\u03cc\u03c1\u03c5\u03b2\u03bf.<\/p>\n<p>\u03a4\u03bf step count \u03c0\u03c1\u03bf\u03ad\u03b2\u03bb\u03b5\u03c8\u03b5 \u03c3\u03c7\u03b5\u03b4\u03cc\u03bd \u03c4\u03ad\u03bb\u03b5\u03b9\u03b1 \u03c4\u03bf end-to-end latency \u03c3\u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf \u03c0\u03b5\u03af\u03c1\u03b1\u03bc\u03b1: r=0,9997 \u03ba\u03b1\u03b9 R\u00b2=0,9994. \u0398\u03b5\u03c9\u03c1\u03b7\u03c4\u03b9\u03ba\u03ac, \u03b1\u03bd \u03c4\u03bf \u03c3\u03cd\u03c3\u03c4\u03b7\u03bc\u03b1 \u03b3\u03bd\u03ce\u03c1\u03b9\u03b6\u03b5 \u03b1\u03c0\u03cc \u03c4\u03b7\u03bd \u03b5\u03af\u03c3\u03bf\u03b4\u03bf \u03c3\u03b5 \u03c0\u03bf\u03b9\u03bf \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03bf \u03b8\u03b1 \u03ba\u03b1\u03c4\u03b1\u03bb\u03ae\u03be\u03b5\u03b9 \u03ad\u03bd\u03b1 request, \u03bf scheduler \u03b8\u03b1 \u03b5\u03c0\u03ad\u03bb\u03b5\u03b3\u03b5 \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03b5 11 \u03b3\u03bd\u03c9\u03c3\u03c4\u03ad\u03c2 \u03b5\u03ba\u03b2\u03ac\u03c3\u03b5\u03b9\u03c2 \u03b1\u03bd\u03c4\u03af \u03bd\u03b1 \u03b5\u03ba\u03c4\u03b9\u03bc\u03ac \u03ad\u03bd\u03b1\u03bd \u03c3\u03c5\u03bd\u03b5\u03c7\u03cc\u03bc\u03b5\u03bd\u03bf \u03c7\u03c1\u03cc\u03bd\u03bf \u03c5\u03c0\u03b7\u03c1\u03b5\u03c3\u03af\u03b1\u03c2. \u0397 \u03c0\u03bb\u03b7\u03c1\u03bf\u03c6\u03bf\u03c1\u03af\u03b1 \u03b1\u03c5\u03c4\u03ae, \u03cc\u03bc\u03c9\u03c2, \u03b4\u03b5\u03bd \u03ae\u03c4\u03b1\u03bd \u03b4\u03b9\u03b1\u03b8\u03ad\u03c3\u03b9\u03bc\u03b7 \u03c0\u03c1\u03b9\u03bd \u03be\u03b5\u03b4\u03b9\u03c0\u03bb\u03c9\u03b8\u03b5\u03af \u03b7 \u03b4\u03b9\u03b1\u03b4\u03b9\u03ba\u03b1\u03c3\u03af\u03b1 denoising.<\/p>\n<p>\u0397 \u03b4\u03b9\u03b1\u03ba\u03cd\u03bc\u03b1\u03bd\u03c3\u03b7 \u03ae\u03c4\u03b1\u03bd \u03bf\u03c5\u03c3\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03ae. \u03a3\u03c4\u03bf GSM8K \u03bc\u03b5 ceiling 512 tokens, \u03b7 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b5 coefficient of variation 0,272 \u03ba\u03b1\u03b9 \u03bb\u03cc\u03b3\u03bf \u03bc\u03ad\u03b3\u03b9\u03c3\u03c4\u03bf\u03c5 \u03c0\u03c1\u03bf\u03c2 \u03b5\u03bb\u03ac\u03c7\u03b9\u03c3\u03c4\u03bf latency 2,64\u00d7. \u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b1\u03bd\u03b1\u03c6\u03ad\u03c1\u03bf\u03c5\u03bd CV \u03b1\u03c0\u03cc 0,256 \u03ad\u03c9\u03c2 0,283 \u03c3\u03b5 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac measurement passes \u03bc\u03b5 n=100 \u03ba\u03b1\u03b9 \u03c4\u03b9\u03c2 \u03ba\u03c1\u03b1\u03c4\u03bf\u03cd\u03bd \u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ad\u03c2 \u03b1\u03bd\u03c4\u03af \u03bd\u03b1 \u03c4\u03b9\u03c2 \u03c3\u03c5\u03b3\u03c7\u03c9\u03bd\u03b5\u03cd\u03c3\u03bf\u03c5\u03bd \u03c4\u03b5\u03c7\u03bd\u03b7\u03c4\u03ac.<\/p>\n<div class=\"td-chart td-chart--metrics\">\n<div class=\"td-chart-head\">\n<p class=\"td-chart-kicker\">LLaDA-8B-Instruct + D2F \u00b7 \u03bc\u03af\u03b1 H200<\/p>\n<p class=\"td-chart-title\">\u03a4\u03b1 \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 \u03bc\u03b5\u03b3\u03ad\u03b8\u03b7 \u03c0\u03bf\u03c5 \u03b1\u03bb\u03bb\u03ac\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf capacity plan<\/p>\n<p class=\"td-chart-intro\">\u039a\u03ac\u03b8\u03b5 \u03c4\u03b9\u03bc\u03ae \u03b1\u03c6\u03bf\u03c1\u03ac \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf \u03c0\u03b5\u03b9\u03c1\u03b1\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc setup \u03ba\u03b1\u03b9 \u03b4\u03b5\u03bd \u03b1\u03c0\u03bf\u03c4\u03b5\u03bb\u03b5\u03af \u03b3\u03b5\u03bd\u03b9\u03ba\u03cc benchmark \u03b3\u03b9\u03b1 \u03cc\u03bb\u03b1 \u03c4\u03b1 diffusion LLMs.<\/p>\n<\/div>\n<div class=\"td-metric-grid\">\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">11 tiers<\/span><span class=\"td-metric-label\">\u03b4\u03b9\u03b1\u03ba\u03c1\u03b9\u03c4\u03ac \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1 \u03b1\u03c0\u03cc 178 \u03ad\u03c9\u03c2 468 denoising steps<\/span><\/div>\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">R\u00b2 0,150<\/span><span class=\"td-metric-label\">\u03c4\u03bf \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf admission-time signal \u03c3\u03c4\u03bf GSM8K<\/span><\/div>\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">16,0\u00d7<\/span><span class=\"td-metric-label\">throughput \u03c3\u03c4\u03bf batch size 16 \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 single-request baseline<\/span><\/div>\n<div class=\"td-metric-card\"><span class=\"td-metric-value\">75,6%<\/span><span class=\"td-metric-label\">CPU-side overhead \u03c3\u03c4\u03bf profiled request<\/span><\/div>\n<\/div>\n<\/div>\n<h2 id=\"prompt-routing\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf prompt length \u03b4\u03b5\u03bd \u03b1\u03c1\u03ba\u03b5\u03af \u03b3\u03b9\u03b1 \u03ad\u03be\u03c5\u03c0\u03bd\u03bf routing<\/h2>\n<p>\u0397 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03b4\u03bf\u03ba\u03af\u03bc\u03b1\u03c3\u03b5 \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 \u03c6\u03b8\u03b7\u03bd\u03ac signals \u03c0\u03bf\u03c5 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b4\u03b9\u03b1\u03b8\u03ad\u03c3\u03b9\u03bc\u03b1 \u03bc\u03b5\u03c4\u03ac \u03c4\u03bf \u03c0\u03c1\u03ce\u03c4\u03bf denoising step: \u03c4\u03bf \u03c0\u03bf\u03c3\u03bf\u03c3\u03c4\u03cc tokens \u03c0\u03ac\u03bd\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf confidence threshold, \u03c4\u03bf \u03bc\u03ad\u03c3\u03bf token confidence, \u03c4\u03bf output entropy \u03ba\u03b1\u03b9 \u03c4\u03bf prompt length. \u03a4\u03b1 R\u00b2 \u03c9\u03c2 \u03c0\u03c1\u03bf\u03c2 \u03c4\u03bf \u03c4\u03b5\u03bb\u03b9\u03ba\u03cc difficulty level \u03ae\u03c4\u03b1\u03bd \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b1 0,011, 0,004, 0,023 \u03ba\u03b1\u03b9 0,150. \u038c\u03bb\u03b1 \u03ad\u03bc\u03b5\u03b9\u03bd\u03b1\u03bd \u03c0\u03bf\u03bb\u03cd \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03c1\u03b1\u03ba\u03c4\u03b9\u03ba\u03cc \u03ba\u03b1\u03c4\u03ce\u03c6\u03bb\u03b9 0,5 \u03c0\u03bf\u03c5 \u03ad\u03b8\u03b5\u03c3\u03b1\u03bd \u03bf\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2.<\/p>\n<p>\u0397 \u03b5\u03be\u03ae\u03b3\u03b7\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b4\u03bf\u03bc\u03b9\u03ba\u03ae. \u039c\u03b5\u03c4\u03ac \u03c4\u03bf \u03c0\u03c1\u03ce\u03c4\u03bf step, \u03ba\u03b1\u03b9 \u03bf\u03b9 32 \u03b8\u03ad\u03c3\u03b5\u03b9\u03c2 \u03c4\u03bf\u03c5 \u03c0\u03c1\u03ce\u03c4\u03bf\u03c5 block \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03bf\u03c5\u03bd masked \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b1 \u03b1\u03c0\u03cc \u03c4\u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03c4\u03bf\u03c5 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ae\u03bc\u03b1\u03c4\u03bf\u03c2. \u0388\u03bd\u03b1 \u03c0\u03bf\u03bb\u03cd \u03c0\u03c1\u03ce\u03b9\u03bc\u03bf signal \u03b1\u03bd\u03c4\u03b1\u03bd\u03b1\u03ba\u03bb\u03ac \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7 \u03b3\u03b5\u03bd\u03b9\u03ba\u03ae prior \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5 \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03c4\u03b7 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03b7 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae \u03c0\u03bf\u03c5 \u03b8\u03b1 \u03b1\u03ba\u03bf\u03bb\u03bf\u03c5\u03b8\u03ae\u03c3\u03b5\u03b9 \u03c4\u03bf request. \u0397 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b1\u03bd\u03b1\u03b4\u03cd\u03b5\u03c4\u03b1\u03b9 \u03bc\u03ad\u03c3\u03b1 \u03c3\u03c4\u03b7 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae denoising.<\/p>\n<p>\u0393\u03b9\u03b1 product \u03ba\u03b1\u03b9 operations teams \u03b1\u03c5\u03c4\u03cc \u03b1\u03c0\u03bf\u03ba\u03bb\u03b5\u03af\u03b5\u03b9 \u03bc\u03b9\u03b1 \u03b4\u03b5\u03bb\u03b5\u03b1\u03c3\u03c4\u03b9\u03ba\u03ae \u03b1\u03bb\u03bb\u03ac \u03bc\u03b7 \u03c4\u03b5\u03ba\u03bc\u03b7\u03c1\u03b9\u03c9\u03bc\u03ad\u03bd\u03b7 \u03c3\u03c5\u03bd\u03c4\u03cc\u03bc\u03b5\u03c5\u03c3\u03b7: \u03ad\u03bd\u03b1 \u03c3\u03cd\u03bd\u03c4\u03bf\u03bc\u03bf prompt \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03ba\u03b1\u03c4\u2019 \u03b1\u03bd\u03ac\u03b3\u03ba\u03b7 \u03c6\u03b8\u03b7\u03bd\u03cc \u03ba\u03b1\u03b9 \u03ad\u03bd\u03b1 \u03b3\u03c1\u03ae\u03b3\u03bf\u03c1\u03bf confidence check \u03b4\u03b5\u03bd \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c0\u03b5\u03b9 \u03b1\u03be\u03b9\u03cc\u03c0\u03b9\u03c3\u03c4\u03b1 \u03c4\u03bf\u03bd \u03c7\u03c1\u03cc\u03bd\u03bf \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b7\u03c2. \u03a4\u03bf routing \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 evidence \u03b1\u03c0\u03cc \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf model, task \u03ba\u03b1\u03b9 engine, \u03cc\u03c0\u03c9\u03c2 \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03c4\u03bf <a href=\"https:\/\/twodots.gr\/llm-routing-latency-accuracy-cost\/\">LLM routing \u03bc\u03b5 \u03b5\u03c0\u03af\u03b3\u03bd\u03c9\u03c3\u03b7 latency, accuracy \u03ba\u03b1\u03b9 \u03ba\u03cc\u03c3\u03c4\u03bf\u03c5\u03c2<\/a>.<\/p>\n<h2 id=\"generation-budgets\">\u03a4\u03b1 \u03bc\u03b9\u03ba\u03c1\u03ac generation budgets \u03ba\u03c1\u03cd\u03b2\u03bf\u03c5\u03bd \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc serving variance<\/h2>\n<p>\u0397 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03b5\u03c0\u03b1\u03bd\u03ad\u03bb\u03b1\u03b2\u03b5 \u03c4\u03bf GSM8K \u03bc\u03b5 budgets 128, 256, 512 \u03ba\u03b1\u03b9 1024 tokens, \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03ce\u03bd\u03c4\u03b1\u03c2 100 requests \u03b1\u03bd\u03ac budget \u03c3\u03b5 single-request inference. \u03a3\u03c4\u03b1 128 tokens, \u03c4\u03bf 100% \u03c4\u03c9\u03bd \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03c9\u03bd \u03ba\u03cc\u03c0\u03b7\u03ba\u03b5 \u03b1\u03c0\u03cc \u03c4\u03bf \u03cc\u03c1\u03b9\u03bf \u03ba\u03b1\u03b9 \u03b7 accuracy \u03ae\u03c4\u03b1\u03bd 6%. \u03a3\u03c4\u03b1 256 tokens, \u03c4\u03bf truncation \u03c0\u03b1\u03c1\u03ad\u03bc\u03b5\u03bd\u03b5 83% \u03ba\u03b1\u03b9 \u03b7 accuracy \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 48%. \u03a3\u03c4\u03b1 512 tokens, \u03c4\u03bf truncation \u03c5\u03c0\u03bf\u03c7\u03ce\u03c1\u03b7\u03c3\u03b5 \u03c3\u03b5 3% \u03ba\u03b1\u03b9 \u03b7 accuracy \u03ae\u03c4\u03b1\u03bd 76%, \u03b5\u03bd\u03ce \u03c3\u03c4\u03b1 1024 tokens \u03bf\u03b9 \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b5\u03c2 \u03c4\u03b9\u03bc\u03ad\u03c2 \u03ae\u03c4\u03b1\u03bd 1% \u03ba\u03b1\u03b9 73%.<\/p>\n<p>\u03a4\u03bf workload \u03c7\u03c1\u03b5\u03b9\u03b1\u03b6\u03cc\u03c4\u03b1\u03bd \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 320 tokens \u03ba\u03b1\u03c4\u03ac \u03bc\u03ad\u03c3\u03bf \u03cc\u03c1\u03bf \u03b3\u03b9\u03b1 \u03bd\u03b1 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03c3\u03b5\u03b9 \u03ad\u03bd\u03b1 GSM8K \u03c0\u03c1\u03cc\u03b2\u03bb\u03b7\u03bc\u03b1: 317,4 tokens \u03c3\u03c4\u03b7 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae \u03bc\u03b5 \u03cc\u03c1\u03b9\u03bf 512 \u03ba\u03b1\u03b9 323,5 \u03bc\u03b5 \u03cc\u03c1\u03b9\u03bf 1024. \u03a0\u03b1\u03c1\u2019 \u03cc\u03bb\u03b1 \u03b1\u03c5\u03c4\u03ac, \u03c4\u03bf latency CV \u03c3\u03c5\u03bd\u03ad\u03c7\u03b9\u03c3\u03b5 \u03bd\u03b1 \u03b1\u03bd\u03b5\u03b2\u03b1\u03af\u03bd\u03b5\u03b9 \u03b1\u03c0\u03cc 0,283 \u03c3\u03c4\u03b1 512 tokens \u03c3\u03b5 0,360 \u03c3\u03c4\u03b1 1024. \u03a4\u03b1 P50\/P90\/P99 \u03ae\u03c4\u03b1\u03bd 6,46\/9,72\/11,48 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03c3\u03c4\u03b1 512 \u03ba\u03b1\u03b9 6,41\/9,05\/10,66 \u03c3\u03c4\u03b1 1024.<\/p>\n<p>\u03a4\u03bf \u03c3\u03c5\u03bc\u03c0\u03ad\u03c1\u03b1\u03c3\u03bc\u03b1 \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf budget \u03b5\u03af\u03bd\u03b1\u03b9 \u03c0\u03ac\u03bd\u03c4\u03b1 \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf. \u0395\u03af\u03bd\u03b1\u03b9 \u03cc\u03c4\u03b9 \u03ad\u03bd\u03b1 benchmark \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c6\u03c5\u03c3\u03b9\u03ba\u03cc completion length \u03bc\u03b5\u03c4\u03c1\u03ac \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7 \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b9\u03c6\u03bf\u03c1\u03ac truncation \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc serving workload. \u039c\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 \u03c0\u03bf\u03c5 \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03b9 providers \u03ae architectures \u03bc\u03b5 \u03bc\u03b9\u03ba\u03c1\u03ac caps \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03c3\u03b5\u03b9 \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc variance \u03b1\u03c0\u03bb\u03ce\u03c2 \u03b5\u03c0\u03b5\u03b9\u03b4\u03ae \u03ba\u03cc\u03b2\u03b5\u03b9 \u03c4\u03b9\u03c2 \u03b4\u03cd\u03c3\u03ba\u03bf\u03bb\u03b5\u03c2 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c0\u03c1\u03b9\u03bd \u03c6\u03b1\u03bd\u03b5\u03af \u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac.<\/p>\n<h2 id=\"humaneval\">\u03a4\u03b9 \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03af\u03c9\u03c3\u03b5 \u03c4\u03bf HumanEval<\/h2>\n<p>\u0393\u03b9\u03b1 \u03bd\u03b1 \u03b5\u03bb\u03ad\u03b3\u03be\u03bf\u03c5\u03bd \u03b1\u03bd \u03c4\u03bf \u03b5\u03cd\u03c1\u03b7\u03bc\u03b1 \u03c0\u03b5\u03c1\u03b9\u03bf\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c3\u03c4\u03b1 \u03bc\u03b1\u03b8\u03b7\u03bc\u03b1\u03c4\u03b9\u03ba\u03ac \u03c0\u03c1\u03bf\u03b2\u03bb\u03ae\u03bc\u03b1\u03c4\u03b1, \u03bf\u03b9 \u03b5\u03c1\u03b5\u03c5\u03bd\u03b7\u03c4\u03ad\u03c2 \u03b5\u03c0\u03b1\u03bd\u03ad\u03bb\u03b1\u03b2\u03b1\u03bd block-size sweep \u03ba\u03b1\u03b9 variance characterization \u03c3\u03b5 64 HumanEval code-completion problems. \u039c\u03ad\u03c3\u03b1 \u03c3\u03c4\u03bf \u03af\u03b4\u03b9\u03bf request, \u03c4\u03bf latency \u03ad\u03bc\u03b5\u03b9\u03bd\u03b5 \u03c3\u03c7\u03b5\u03b4\u03cc\u03bd \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc \u03b3\u03b9\u03b1 block sizes 16, 32, 64 \u03ba\u03b1\u03b9 128, \u03bc\u03b5 CV 0,0096. \u039c\u03b5\u03c4\u03b1\u03be\u03cd \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ce\u03bd requests, \u03c4\u03bf CV \u03ae\u03c4\u03b1\u03bd 0,345. \u0397 \u03b1\u03c0\u03cc\u03c3\u03c4\u03b1\u03c3\u03b7 \u03ae\u03c4\u03b1\u03bd \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 36\u00d7, \u03c0\u03bf\u03bb\u03cd \u03ba\u03bf\u03bd\u03c4\u03ac \u03c3\u03c4\u03bf \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03bf 36\u00d7 \u03c4\u03bf\u03c5 GSM8K, \u03cc\u03c0\u03bf\u03c5 \u03bf\u03b9 \u03c4\u03b9\u03bc\u03ad\u03c2 \u03ae\u03c4\u03b1\u03bd 0,0073 \u03ba\u03b1\u03b9 0,260.<\/p>\n<p>\u03a4\u03bf HumanEval \u03b5\u03af\u03c7\u03b5 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b7 \u03b4\u03b9\u03b1\u03ba\u03cd\u03bc\u03b1\u03bd\u03c3\u03b7: CV 0,343-0,346 \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 0,256-0,263 \u03c3\u03c4\u03bf GSM8K, \u03ba\u03b1\u03b9 max-to-min ratio 3,99-4,09\u00d7 \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 2,60-2,70\u00d7. \u03a4\u03bf prompt length \u03b5\u03af\u03c7\u03b5 \u03b9\u03c3\u03c7\u03c5\u03c1\u03cc\u03c4\u03b5\u03c1\u03b7 \u03c3\u03c5\u03c3\u03c7\u03ad\u03c4\u03b9\u03c3\u03b7 \u03bc\u03b5 \u03c4\u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03c3\u03c4\u03bf HumanEval, r=0,648 \u03ba\u03b1\u03b9 R\u00b2=0,42, \u03b1\u03bb\u03bb\u03ac \u03ba\u03b1\u03b9 \u03c0\u03ac\u03bb\u03b9 \u03b4\u03b5\u03bd \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 \u03c4\u03bf \u03c0\u03c1\u03b1\u03ba\u03c4\u03b9\u03ba\u03cc \u03ba\u03b1\u03c4\u03ce\u03c6\u03bb\u03b9 \u03c0\u03c1\u03cc\u03b2\u03bb\u03b5\u03c8\u03b7\u03c2 \u03c4\u03b7\u03c2 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7\u03c2.<\/p>\n<p>\u039f\u03b9 \u03af\u03b4\u03b9\u03bf\u03b9 \u03bf\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03c7\u03b1\u03c1\u03b1\u03ba\u03c4\u03b7\u03c1\u03af\u03b6\u03bf\u03c5\u03bd \u03c4\u03b7 \u03c3\u03c5\u03bc\u03c6\u03c9\u03bd\u03af\u03b1 \u03c3\u03c5\u03bd\u03b5\u03c0\u03ae \u03b1\u03bb\u03bb\u03ac \u03c0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03ad\u03bd\u03b7 \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03ba\u03b1\u03b8\u03bf\u03bb\u03b9\u03ba\u03ae \u03b1\u03c0\u03cc\u03b4\u03b5\u03b9\u03be\u03b7. \u0397 \u03c3\u03c9\u03c3\u03c4\u03ae \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b7\u03bc\u03b1\u03c4\u03b9\u03ba\u03ae \u03b1\u03bd\u03ac\u03b3\u03bd\u03c9\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b1 \u03c0\u03c1\u03bf\u03c3\u03b5\u03ba\u03c4\u03b9\u03ba\u03ae: \u03c4\u03bf pattern \u03b1\u03be\u03af\u03b6\u03b5\u03b9 \u03bd\u03b1 \u03b5\u03c0\u03b1\u03bb\u03b7\u03b8\u03b5\u03c5\u03c4\u03b5\u03af \u03c3\u03b5 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac tasks, models \u03ba\u03b1\u03b9 hardware \u03c0\u03c1\u03b9\u03bd \u03bc\u03b5\u03c4\u03b1\u03c4\u03c1\u03b1\u03c0\u03b5\u03af \u03c3\u03b5 production policy.<\/p>\n<h2 id=\"synchronized-batching\">Synchronized batching: 16\u00d7 throughput \u03bc\u03b5 \u03ba\u03bf\u03b9\u03bd\u03cc forward pass<\/h2>\n<p>\u03a3\u03c4\u03bf synchronized batching, \u03cc\u03bb\u03b1 \u03c4\u03b1 requests \u03b5\u03bd\u03cc\u03c2 batch \u03c0\u03c1\u03bf\u03c3\u03b1\u03c1\u03bc\u03cc\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03c3\u03b5 \u03ba\u03bf\u03b9\u03bd\u03cc shape \u03ba\u03b1\u03b9 \u03c0\u03c1\u03bf\u03c7\u03c9\u03c1\u03bf\u03cd\u03bd \u03bc\u03b5 \u03ad\u03bd\u03b1 shared forward pass \u03c3\u03b5 \u03ba\u03ac\u03b8\u03b5 denoising step. \u039c\u03b5\u03c4\u03ac \u03c4\u03bf pass, \u03ba\u03ac\u03b8\u03b5 request \u03c3\u03c5\u03bd\u03b5\u03c7\u03af\u03b6\u03b5\u03b9 \u03bd\u03b1 \u03ba\u03ac\u03bd\u03b5\u03b9 sampling \u03bc\u03b5 \u03c4\u03bf \u03b4\u03b9\u03ba\u03cc \u03c4\u03bf\u03c5 stream. \u0397 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03b9 \u03bc\u03b5 per-request dispatch, \u03cc\u03c0\u03bf\u03c5 \u03ba\u03ac\u03b8\u03b5 request \u03b5\u03ba\u03c4\u03b5\u03bb\u03b5\u03af \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03bf forward pass \u03c3\u03b5 \u03ba\u03ac\u03b8\u03b5 step.<\/p>\n<p>\u03a3\u03c4\u03bf batch size 16, \u03c4\u03bf synchronized batching \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 1,939 requests \u03b1\u03bd\u03ac \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03bf \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 0,121 requests \u03b1\u03bd\u03ac \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03bf \u03c3\u03c4\u03bf single-request throughput: \u03b2\u03b5\u03bb\u03c4\u03af\u03c9\u03c3\u03b7 16,0\u00d7. \u03a4\u03bf \u03c0\u03c1\u03cc\u03c3\u03b8\u03b5\u03c4\u03bf overhead \u03c4\u03bf\u03c5 per-request dispatch \u03b1\u03ba\u03bf\u03bb\u03bf\u03cd\u03b8\u03b7\u03c3\u03b5 \u03c3\u03c4\u03b5\u03bd\u03ac \u03c4\u03bf batch size: 2,4\u00d7 \u03c3\u03c4\u03bf B=2, 5,1\u00d7 \u03c3\u03c4\u03bf B=4 \u03ba\u03b1\u03b9 8,1\u00d7 \u03c3\u03c4\u03bf B=8. \u0393\u03b9\u03b1 B requests, \u03b7 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b7 \u03b5\u03ba\u03c4\u03ad\u03bb\u03b5\u03c3\u03b7 \u03c0\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03b9 B forward passes \u03b1\u03bd\u03ac denoising step, \u03b5\u03bd\u03ce \u03b7 \u03ba\u03bf\u03b9\u03bd\u03ae \u03b5\u03ba\u03c4\u03ad\u03bb\u03b5\u03c3\u03b7 \u03c0\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03b9 \u03ad\u03bd\u03b1.<\/p>\n<div class=\"td-comparison td-comparison-cards--horizontal\">\n<div class=\"td-chart-head\">\n<p class=\"td-chart-kicker\">\u03a4\u03bf \u03af\u03b4\u03b9\u03bf batch, \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ae \u03bc\u03bf\u03bd\u03ac\u03b4\u03b1 \u03c0\u03b1\u03c1\u03b1\u03bb\u03bb\u03b7\u03bb\u03b9\u03c3\u03bc\u03bf\u03cd<\/p>\n<p class=\"td-chart-title\">\u03a4\u03bf \u03ba\u03ad\u03c1\u03b4\u03bf\u03c2 \u03ad\u03c1\u03c7\u03b5\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03c4\u03b7\u03bd \u03ba\u03bf\u03b9\u03bd\u03ae \u03b5\u03c1\u03b3\u03b1\u03c3\u03af\u03b1 \u03b1\u03bd\u03ac denoising step<\/p>\n<p class=\"td-chart-intro\">\u0397 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7 \u03b1\u03c0\u03bf\u03bc\u03bf\u03bd\u03ce\u03bd\u03b5\u03b9 \u03c4\u03bf\u03bd dispatch \u03bc\u03b7\u03c7\u03b1\u03bd\u03b9\u03c3\u03bc\u03cc\u00b7 \u03b4\u03b5\u03bd \u03c3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03b9 \u03ad\u03bd\u03b1\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03c9\u03bc\u03ad\u03bd\u03bf \u03bd\u03ad\u03bf engine \u03bc\u03b5 \u03cc\u03bb\u03b1 \u03c4\u03b1 production features \u03c4\u03bf\u03c5 vLLM \u03ae \u03c4\u03bf\u03c5 Orca.<\/p>\n<\/div>\n<div class=\"td-comparison-grid\">\n<div class=\"td-platform-card\">\n<p class=\"td-platform-name\">Per-request dispatch<\/p>\n<p>\u039a\u03ac\u03b8\u03b5 \u03b5\u03bd\u03b5\u03c1\u03b3\u03cc request \u03c0\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03b9 \u03b4\u03b9\u03ba\u03cc \u03c4\u03bf\u03c5 forward pass \u03c3\u03b5 \u03ba\u03ac\u03b8\u03b5 step, \u03bf\u03c0\u03cc\u03c4\u03b5 \u03c4\u03bf host overhead \u03b1\u03c5\u03be\u03ac\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c3\u03c7\u03b5\u03b4\u03cc\u03bd \u03bc\u03b1\u03b6\u03af \u03bc\u03b5 \u03c4\u03bf batch size.<\/p>\n<\/div>\n<div class=\"td-platform-card\">\n<p class=\"td-platform-name\">Synchronized batch<\/p>\n<p>\u03a4\u03b1 requests \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03ad\u03bd\u03b1 forward pass \u03b1\u03bd\u03ac step \u03ba\u03b1\u03b9 \u03b1\u03c0\u03bf\u03c3\u03b2\u03ad\u03bd\u03bf\u03c5\u03bd \u03c4\u03bf \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c4\u03bf\u03c5 control loop \u03c3\u03b5 \u03cc\u03bb\u03bf \u03c4\u03bf batch.<\/p>\n<\/div>\n<div class=\"td-platform-card\">\n<p class=\"td-platform-name\">Slot-based design<\/p>\n<p>\u0397 \u03c0\u03c1\u03bf\u03c4\u03b5\u03b9\u03bd\u03cc\u03bc\u03b5\u03bd\u03b7 \u03b5\u03c0\u03cc\u03bc\u03b5\u03bd\u03b7 \u03ba\u03b1\u03c4\u03b5\u03cd\u03b8\u03c5\u03bd\u03c3\u03b7 \u03ba\u03c1\u03b1\u03c4\u03ac \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03ac GPU slots \u03ba\u03b1\u03b9 \u03b3\u03b5\u03bc\u03af\u03b6\u03b5\u03b9 \u03ac\u03bc\u03b5\u03c3\u03b1 \u03ad\u03bd\u03b1 slot \u03cc\u03c4\u03b1\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c4\u03bf request \u03c4\u03bf\u03c5\u00b7 \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03b5\u03b9 future work.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p>\u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b4\u03b9\u03b5\u03c5\u03ba\u03c1\u03b9\u03bd\u03af\u03b6\u03bf\u03c5\u03bd \u03cc\u03c4\u03b9 \u03c4\u03bf baseline \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03bf \u03af\u03b4\u03b9\u03bf \u03bc\u03b5 continuous batching \u03c3\u03b5 vLLM \u03ae Orca, \u03cc\u03c0\u03bf\u03c5 \u03ae\u03b4\u03b7 \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 forward pass \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03c4\u03b1 \u03b5\u03bd\u03b5\u03c1\u03b3\u03ac requests \u03ba\u03ac\u03b8\u03b5 iteration. \u0397 \u03c3\u03cd\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b1\u03c0\u03bf\u03bc\u03bf\u03bd\u03ce\u03bd\u03b5\u03b9 \u03c4\u03bf CPU dispatch overhead. \u0393\u03b9\u03b1 \u03c4\u03b7 \u03c3\u03c9\u03c3\u03c4\u03ae \u03b1\u03bd\u03ac\u03b3\u03bd\u03c9\u03c3\u03b7 \u03c4\u03c9\u03bd inference trade-offs \u03b2\u03bf\u03b7\u03b8\u03ac \u03ba\u03b1\u03b9 \u03b7 \u03b1\u03bd\u03ac\u03bb\u03c5\u03c3\u03b7 \u03b3\u03b9\u03b1 \u03c4\u03bf <a href=\"https:\/\/twodots.gr\/vllm-transformers-native-speed-ai-inference\/\">vLLM \u03ba\u03b1\u03b9 \u03c4\u03bf native-speed AI inference<\/a>.<\/p>\n<h2 id=\"cpu-bottleneck\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03bf H200 \u03c0\u03b5\u03c1\u03af\u03bc\u03b5\u03bd\u03b5 \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03c4\u03b7\u03bd CPU<\/h2>\n<p>\u03a3\u03b5 profiling \u03b5\u03bd\u03cc\u03c2 request \u03bc\u03b5 budget 512 tokens, \u03b7 \u03c0\u03bb\u03ae\u03c1\u03b7\u03c2 \u03c0\u03b1\u03c1\u03b1\u03b3\u03c9\u03b3\u03ae \u03b4\u03b9\u03ae\u03c1\u03ba\u03b5\u03c3\u03b5 10,28 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1. \u0391\u03c0\u03cc \u03b1\u03c5\u03c4\u03ac, 2,511 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03ae 24,4% \u03ae\u03c4\u03b1\u03bd GPU kernel execution. \u03a4\u03b1 \u03c5\u03c0\u03cc\u03bb\u03bf\u03b9\u03c0\u03b1 7,77 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03ae 75,6% \u03b1\u03c0\u03bf\u03b4\u03cc\u03b8\u03b7\u03ba\u03b1\u03bd \u03c3\u03c4\u03bf\u03bd CPU-side control loop: \u03c0\u03b1\u03c1\u03b1\u03ba\u03bf\u03bb\u03bf\u03cd\u03b8\u03b7\u03c3\u03b7 \u03ba\u03b1\u03c4\u03ac\u03c3\u03c4\u03b1\u03c3\u03b7\u03c2 blocks, \u03ba\u03b1\u03c4\u03b1\u03c3\u03ba\u03b5\u03c5\u03ae attention masks, \u03ad\u03bb\u03b5\u03b3\u03c7\u03bf\u03c2 thresholds \u03ba\u03b1\u03b9 \u03c3\u03c5\u03b3\u03c7\u03c1\u03bf\u03bd\u03b9\u03c3\u03bc\u03cc\u03c2 \u03bc\u03b5\u03c4\u03b1\u03be\u03cd forward passes.<\/p>\n<p>\u0391\u03c5\u03c4\u03cc \u03bc\u03b5\u03c4\u03b1\u03c6\u03ad\u03c1\u03b5\u03b9 \u03c4\u03b7\u03bd \u03c0\u03c1\u03bf\u03c4\u03b5\u03c1\u03b1\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b2\u03b5\u03bb\u03c4\u03b9\u03c3\u03c4\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7\u03c2. \u03a4\u03bf batching \u03b4\u03b5\u03bd \u03b2\u03bf\u03b7\u03b8\u03ac \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03b5\u03c0\u03b5\u03b9\u03b4\u03ae \u00ab\u03b3\u03b5\u03bc\u03af\u03b6\u03b5\u03b9\u00bb \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b1 \u03c4\u03b7\u03bd GPU \u03bc\u03b5 \u03c4\u03bf\u03bd \u03ba\u03bb\u03b1\u03c3\u03b9\u03ba\u03cc \u03c4\u03c1\u03cc\u03c0\u03bf \u03c4\u03c9\u03bd AR workloads. \u0392\u03bf\u03b7\u03b8\u03ac \u03b5\u03c0\u03b5\u03b9\u03b4\u03ae \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03b5\u03b9 \u03c4\u03bf \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c4\u03bf\u03c5 control loop \u03c3\u03b5 \u03c0\u03bf\u03bb\u03bb\u03ac requests. \u0397 \u03c0\u03b1\u03c1\u03b1\u03c4\u03ae\u03c1\u03b7\u03c3\u03b7 \u03c3\u03c5\u03bd\u03b4\u03ad\u03b5\u03c4\u03b1\u03b9 \u03bc\u03b5 tightly coupled CPU-GPU \u03c5\u03c0\u03bf\u03b4\u03bf\u03bc\u03ad\u03c2, \u03cc\u03c0\u03c9\u03c2 GH200-class superchips, \u03ba\u03b1\u03b9 \u03bc\u03b5 \u03c4\u03b5\u03c7\u03bd\u03b9\u03ba\u03ad\u03c2 dispatch elimination \u03cc\u03c0\u03c9\u03c2 CUDA Graphs. \u03a0\u03c1\u03cc\u03ba\u03b5\u03b9\u03c4\u03b1\u03b9 \u03b3\u03b9\u03b1 \u03ba\u03b1\u03c4\u03b5\u03cd\u03b8\u03c5\u03bd\u03c3\u03b7 \u03c0\u03bf\u03c5 \u03c0\u03c1\u03bf\u03ba\u03cd\u03c0\u03c4\u03b5\u03b9 \u03b1\u03c0\u03cc \u03c4\u03b7 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7, \u03cc\u03c7\u03b9 \u03b3\u03b9\u03b1 \u03bc\u03b5\u03c4\u03c1\u03b7\u03bc\u03ad\u03bd\u03b7 \u03c3\u03cd\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03bf\u03cd hardware \u03c3\u03c4\u03bf paper.<\/p>\n<p>\u03a3\u03b5 micro-experiment, \u03b7 \u03c3\u03cd\u03bb\u03bb\u03b7\u03c8\u03b7 \u03b5\u03bd\u03cc\u03c2 forward-pass call \u03bc\u03b5 torch.cuda.CUDAGraph \u03bc\u03b5\u03af\u03c9\u03c3\u03b5 \u03c4\u03bf CPU dispatch time \u03b1\u03c0\u03cc 19,4 ms \u03c3\u03b5 10,0 ms, \u03b4\u03b7\u03bb\u03b1\u03b4\u03ae \u03ba\u03b1\u03c4\u03ac 48,6%, \u03bc\u03b5 \u03bc\u03ad\u03b3\u03b9\u03c3\u03c4\u03b7 \u03b1\u03c0\u03cc\u03bb\u03c5\u03c4\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac logits 0,0. \u03a4\u03bf profiled request \u03b5\u03af\u03c7\u03b5 235 forward-pass calls. \u0391\u03bd \u03c0\u03b1\u03c1\u03cc\u03bc\u03bf\u03b9\u03b1 \u03b5\u03be\u03bf\u03b9\u03ba\u03bf\u03bd\u03cc\u03bc\u03b7\u03c3\u03b7 \u03af\u03c3\u03c7\u03c5\u03b5 \u03c3\u03b5 \u03cc\u03bb\u03b1 \u03c4\u03b1 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac KV-cache shapes, \u03b7 \u03c0\u03c1\u03bf\u03b2\u03bf\u03bb\u03ae \u03c4\u03b7\u03c2 \u03bf\u03bc\u03ac\u03b4\u03b1\u03c2 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03c6\u03b1\u03af\u03c1\u03b5\u03c3\u03b7 2,22 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03bf\u03bb\u03ad\u03c0\u03c4\u03c9\u03bd, \u03b4\u03b7\u03bb\u03b1\u03b4\u03ae 28,5% \u03c4\u03bf\u03c5 \u03c3\u03c5\u03bd\u03bf\u03bb\u03b9\u03ba\u03bf\u03cd dispatch overhead.<\/p>\n<p>\u0397 \u03b4\u03b5\u03cd\u03c4\u03b5\u03c1\u03b7 \u03b5\u03ba\u03c4\u03af\u03bc\u03b7\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 extrapolation \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 end-to-end CUDA Graph measurement. \u03a4\u03bf \u03c5\u03c0\u03cc\u03bb\u03bf\u03b9\u03c0\u03bf 71,5% \u03b2\u03c1\u03af\u03c3\u03ba\u03b5\u03c4\u03b1\u03b9 \u03ad\u03be\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf forward call, \u03c3\u03c4\u03bf\u03bd Python control flow. \u0397 \u03b5\u03c0\u03af\u03c3\u03b7\u03bc\u03b7 \u03c4\u03b5\u03ba\u03bc\u03b7\u03c1\u03af\u03c9\u03c3\u03b7 \u03c4\u03b7\u03c2 NVIDIA \u03b5\u03be\u03b7\u03b3\u03b5\u03af \u03b3\u03b9\u03b1\u03c4\u03af \u03c4\u03b1 CUDA Graphs \u03bc\u03b5\u03b9\u03ce\u03bd\u03bf\u03c5\u03bd \u03b5\u03c0\u03b1\u03bd\u03b1\u03bb\u03b1\u03bc\u03b2\u03b1\u03bd\u03cc\u03bc\u03b5\u03bd\u03bf CPU launch cost, \u03b1\u03bb\u03bb\u03ac \u03ba\u03b1\u03b9 \u03b3\u03b9\u03b1\u03c4\u03af \u03c9\u03c6\u03b5\u03bb\u03bf\u03cd\u03bd \u03bc\u03cc\u03bd\u03bf \u03cc\u03c4\u03b1\u03bd \u03c3\u03c5\u03bb\u03bb\u03b1\u03bc\u03b2\u03ac\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc CPU-bound \u03c4\u03bc\u03ae\u03bc\u03b1.<\/p>\n<h2 id=\"batching-quality\">\u0397 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c5\u03c0\u03cc batching \u03ad\u03c7\u03b5\u03b9 \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2<\/h2>\n<p>\u03a3\u03c4\u03bf single-request GSM8K \u03bc\u03b5 generation length 512, \u03bc\u03af\u03b1 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae \u03ad\u03b4\u03c9\u03c3\u03b5 exact-match accuracy 74% \u03ba\u03b1\u03b9 \u03bc\u03b9\u03b1 \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae 76%. \u039f\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03b1\u03c0\u03bf\u03b4\u03af\u03b4\u03bf\u03c5\u03bd \u03c4\u03b7 \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03ac \u03c3\u03c4\u03bf stochastic sampling \u03bc\u03b5 temperature \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf \u03b1\u03c0\u03cc \u03bc\u03b7\u03b4\u03ad\u03bd. \u0391\u03c0\u03cc \u03c4\u03b9\u03c2 24 \u03bb\u03b1\u03bd\u03b8\u03b1\u03c3\u03bc\u03ad\u03bd\u03b5\u03c2 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c3\u03c4\u03b7 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ae 74%, \u03bc\u03cc\u03bd\u03bf \u03b4\u03cd\u03bf \u03ad\u03c6\u03c4\u03b1\u03c3\u03b1\u03bd \u03c3\u03c4\u03bf \u03cc\u03c1\u03b9\u03bf \u03c4\u03c9\u03bd 512 tokens, \u03ac\u03c1\u03b1 \u03c4\u03bf truncation \u03ae\u03c4\u03b1\u03bd \u03bc\u03b9\u03ba\u03c1\u03cc\u03c2 \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03bf \u03ba\u03cd\u03c1\u03b9\u03bf\u03c2 \u03c0\u03b1\u03c1\u03ac\u03b3\u03bf\u03bd\u03c4\u03b1\u03c2 \u03c3\u03c6\u03ac\u03bb\u03bc\u03b1\u03c4\u03bf\u03c2 \u03c3\u03b5 \u03b5\u03ba\u03b5\u03af\u03bd\u03bf \u03c4\u03bf run.<\/p>\n<p>\u0393\u03b9\u03b1 \u03c4\u03bf synchronized batching, \u03c4\u03bf paper \u03b4\u03b5\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b5 \u03ac\u03bc\u03b5\u03c3\u03b7 empirical quality evaluation \u03c3\u03c4\u03b1 batch sizes 8 \u03ba\u03b1\u03b9 16. \u0391\u03bd\u03c4\u03af \u03b3\u03b9\u2019 \u03b1\u03c5\u03c4\u03cc \u03b4\u03af\u03bd\u03b5\u03b9 \u03b4\u03bf\u03bc\u03b9\u03ba\u03cc \u03b5\u03c0\u03b9\u03c7\u03b5\u03af\u03c1\u03b7\u03bc\u03b1 \u03c0\u03bf\u03c5 \u03b1\u03c0\u03b1\u03b9\u03c4\u03b5\u03af \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2: attention mask \u03ba\u03b1\u03b9 positional encoding \u03ba\u03ac\u03b8\u03b5 request \u03bd\u03b1 \u03c0\u03b1\u03c1\u03b1\u03bc\u03ad\u03bd\u03bf\u03c5\u03bd \u03af\u03b4\u03b9\u03b1 \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc padding, \u03ba\u03ac\u03b8\u03b5 request \u03bd\u03b1 \u03ad\u03c7\u03b5\u03b9 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03bf RNG stream \u03ba\u03b1\u03b9 \u03bf\u03b9 batched attention\/sampling kernels \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03c1\u03b9\u03b8\u03bc\u03b7\u03c4\u03b9\u03ba\u03ac \u03b9\u03c3\u03bf\u03b4\u03cd\u03bd\u03b1\u03bc\u03bf\u03b9 \u03bc\u03b5 \u03c4\u03b9\u03c2 single-request \u03b5\u03ba\u03b4\u03cc\u03c3\u03b5\u03b9\u03c2.<\/p>\n<p>\u0395\u03c0\u03b5\u03b9\u03b4\u03ae \u03b7 \u03b5\u03bc\u03c0\u03b5\u03b9\u03c1\u03b9\u03ba\u03ae \u03b5\u03c0\u03b9\u03b2\u03b5\u03b2\u03b1\u03af\u03c9\u03c3\u03b7 \u03bb\u03b5\u03af\u03c0\u03b5\u03b9, \u03bc\u03b9\u03b1 production \u03bf\u03bc\u03ac\u03b4\u03b1 \u03b4\u03b5\u03bd \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03bc\u03b5\u03c4\u03b1\u03c4\u03c1\u03ad\u03c8\u03b5\u03b9 \u03b1\u03c5\u03c4\u03ae \u03c4\u03b7\u03bd \u03c5\u03c0\u03cc\u03b8\u03b5\u03c3\u03b7 \u03c3\u03b5 \u03b5\u03b3\u03b3\u03cd\u03b7\u03c3\u03b7. Throughput regression tests \u03ba\u03b1\u03b9 quality regression tests \u03b5\u03af\u03bd\u03b1\u03b9 \u03b4\u03cd\u03bf \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ad\u03c2 \u03c0\u03cd\u03bb\u03b5\u03c2 \u03ad\u03ba\u03b4\u03bf\u03c3\u03b7\u03c2.<\/p>\n<h2 id=\"batch-timeout\">\u03a4\u03bf timeout \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03c4\u03b1\u03c4\u03b5\u03cd\u03b5\u03b9 \u03c4\u03b7 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03b7\u03c2 \u03bf\u03c5\u03c1\u03ac\u03c2<\/h2>\n<p>\u03a4\u03bf fixed-fill synchronized batching \u03b4\u03b7\u03bc\u03b9\u03bf\u03c5\u03c1\u03b3\u03b5\u03af \u03ad\u03bd\u03b1 trade-off: \u03bc\u03b5\u03b3\u03ac\u03bb\u03b7 \u03b1\u03bd\u03b1\u03bc\u03bf\u03bd\u03ae \u03b3\u03b5\u03bc\u03af\u03b6\u03b5\u03b9 \u03c4\u03bf batch \u03b1\u03bb\u03bb\u03ac \u03b1\u03c5\u03be\u03ac\u03bd\u03b5\u03b9 \u03c4\u03bf latency, \u03b5\u03bd\u03ce \u03c0\u03c1\u03cc\u03c9\u03c1\u03b7 \u03b1\u03c0\u03bf\u03c3\u03c4\u03bf\u03bb\u03ae \u03b4\u03b7\u03bc\u03b9\u03bf\u03c5\u03c1\u03b3\u03b5\u03af underfull batches \u03c0\u03bf\u03c5 \u03ba\u03bf\u03c3\u03c4\u03af\u03b6\u03bf\u03c5\u03bd \u03ad\u03bd\u03b1 \u03c0\u03bb\u03ae\u03c1\u03b5\u03c2 forward pass. \u0397 \u03b1\u03bd\u03ac\u03bb\u03c5\u03c3\u03b7 \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b5\u03af B=8, \u03bc\u03ad\u03c3\u03bf measured service time S=8,74 \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03b1 \u03b1\u03c0\u03cc \u03bf\u03ba\u03c4\u03ce batches \u03ba\u03b1\u03b9 Poisson arrivals. \u0397 full-batch capacity \u03b5\u03af\u03bd\u03b1\u03b9 \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 0,915 requests \u03b1\u03bd\u03ac \u03b4\u03b5\u03c5\u03c4\u03b5\u03c1\u03cc\u03bb\u03b5\u03c0\u03c4\u03bf.<\/p>\n<p>\u0397 \u03c0\u03c1\u03bf\u03c3\u03bf\u03bc\u03bf\u03af\u03c9\u03c3\u03b7 20.000 requests \u03ad\u03b4\u03b5\u03b9\u03be\u03b5 U-shaped mean latency \u03c9\u03c2 \u03c0\u03c1\u03bf\u03c2 \u03c4\u03bf utilization \u03c1, \u03bc\u03b5 \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc\u03c4\u03b5\u03c1\u03bf \u03bc\u03ad\u03c3\u03bf latency \u03ba\u03bf\u03bd\u03c4\u03ac \u03c3\u03c4\u03bf \u03c1\u22480,70. \u03a3\u03b5 \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc load \u03ba\u03c5\u03c1\u03b9\u03b1\u03c1\u03c7\u03b5\u03af \u03c4\u03bf batch-fill delay, \u03bc\u03b5 \u03bc\u03ad\u03c3\u03b7 \u03b1\u03bd\u03b1\u03bc\u03bf\u03bd\u03ae (B-1)\/(2\u03bb), \u03b4\u03b7\u03bb\u03b1\u03b4\u03ae 3,5\/\u03bb \u03cc\u03c4\u03b1\u03bd B=8. \u03a3\u03b5 \u03c5\u03c8\u03b7\u03bb\u03cc load \u03ba\u03c5\u03c1\u03b9\u03b1\u03c1\u03c7\u03b5\u03af \u03b7 \u03b1\u03bd\u03b1\u03bc\u03bf\u03bd\u03ae \u03c0\u03af\u03c3\u03c9 \u03b1\u03c0\u03cc \u03c0\u03c1\u03bf\u03b7\u03b3\u03bf\u03cd\u03bc\u03b5\u03bd\u03b1 batches.<\/p>\n<p>\u0393\u03b9\u03b1 sparse arrivals, \u03bf\u03b9 \u03c3\u03c5\u03b3\u03b3\u03c1\u03b1\u03c6\u03b5\u03af\u03c2 \u03c0\u03c1\u03bf\u03c3\u03b5\u03b3\u03b3\u03af\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf expected batch size \u03c9\u03c2 1+\u03bbT. \u0391\u03c0\u03cc \u03c4\u03b7 \u03c3\u03c5\u03bd\u03b8\u03ae\u03ba\u03b7 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2 \u03c0\u03c1\u03bf\u03ba\u03cd\u03c0\u03c4\u03b5\u03b9 \u03bf \u03ba\u03b1\u03bd\u03cc\u03bd\u03b1\u03c2 T<sub>min<\/sub>=S-1\/\u03bb. \u0394\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 universal constant: \u03b5\u03be\u03b1\u03c1\u03c4\u03ac\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc arrival rate \u03ba\u03b1\u03b9 measured service time \u03c4\u03bf\u03c5 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c3\u03c5\u03c3\u03c4\u03ae\u03bc\u03b1\u03c4\u03bf\u03c2. \u039c\u03b5\u03c4\u03b1\u03b3\u03b5\u03bd\u03ad\u03c3\u03c4\u03b5\u03c1\u03bf sample 32 batches \u03ac\u03bb\u03bb\u03b1\u03be\u03b5 \u03c4\u03b7\u03bd \u03b5\u03ba\u03c4\u03b9\u03bc\u03ce\u03bc\u03b5\u03bd\u03b7 capacity \u03ba\u03b1\u03c4\u03ac \u03c0\u03b5\u03c1\u03af\u03c0\u03bf\u03c5 24%, \u03ac\u03c1\u03b1 \u03c4\u03b1 percentiles \u03ba\u03b1\u03b9 \u03b7 \u03b2\u03b1\u03b8\u03bc\u03bf\u03bd\u03cc\u03bc\u03b7\u03c3\u03b7 \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03bc\u03b5\u03b3\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03bf \u03b4\u03b5\u03af\u03b3\u03bc\u03b1 \u03c0\u03c1\u03b9\u03bd \u03c7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03b7\u03b8\u03bf\u03cd\u03bd \u03c9\u03c2 \u03c4\u03b5\u03bb\u03b9\u03ba\u03ae production \u03c0\u03bf\u03bb\u03b9\u03c4\u03b9\u03ba\u03ae.<\/p>\n<h2 id=\"business-infrastructure\">\u03a4\u03b9 \u03c3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03b3\u03b9\u03b1 \u03bc\u03b9\u03b1 \u03bf\u03bc\u03ac\u03b4\u03b1 AI infrastructure<\/h2>\n<p>\u03a0\u03c1\u03ce\u03c4\u03bf\u03bd, \u03c4\u03bf benchmark \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b5\u03c0\u03b9\u03c4\u03c1\u03ad\u03c0\u03b5\u03b9 \u03c3\u03c4\u03bf task \u03bd\u03b1 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03c6\u03c5\u03c3\u03b9\u03ba\u03ac. \u0394\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03ac, \u03c4\u03bf \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc latency \u03ba\u03b1\u03b9 \u03c4\u03bf \u03c7\u03b1\u03bc\u03b7\u03bb\u03cc variance \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b1\u03c0\u03bf\u03c4\u03ad\u03bb\u03b5\u03c3\u03bc\u03b1 truncation. \u0394\u03b5\u03cd\u03c4\u03b5\u03c1\u03bf\u03bd, \u03b7 \u03c0\u03b1\u03c1\u03b1\u03ba\u03bf\u03bb\u03bf\u03cd\u03b8\u03b7\u03c3\u03b7 \u03bc\u03cc\u03bd\u03bf GPU utilization \u03b4\u03b5\u03bd \u03b5\u03be\u03b7\u03b3\u03b5\u03af \u03ad\u03bd\u03b1 host-side bottleneck. \u03a7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03b4\u03b9\u03ac\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03b5 wall-clock time, kernel time, control-loop time \u03ba\u03b1\u03b9 dispatch overhead.<\/p>\n<p>\u03a4\u03c1\u03af\u03c4\u03bf\u03bd, \u03c4\u03bf batching policy \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03cc\u03bd\u03bf \u03c0\u03b1\u03c1\u03ac\u03bc\u03b5\u03c4\u03c1\u03bf\u03c2 throughput. \u0395\u03c0\u03b7\u03c1\u03b5\u03ac\u03b6\u03b5\u03b9 queue stability \u03ba\u03b1\u03b9 tail latency, \u03b5\u03bd\u03ce \u03b7 \u03bf\u03c1\u03b8\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c4\u03b7\u03c2 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1\u03c2 \u03b1\u03c0\u03b1\u03b9\u03c4\u03b5\u03af \u03be\u03b5\u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ae \u03b5\u03c0\u03b1\u03bb\u03ae\u03b8\u03b5\u03c5\u03c3\u03b7. \u03a4\u03ad\u03c4\u03b1\u03c1\u03c4\u03bf\u03bd, schedulers \u03b2\u03b1\u03c3\u03b9\u03c3\u03bc\u03ad\u03bd\u03bf\u03b9 \u03c3\u03b5 prompt length \u03ae \u03c3\u03b5 \u03c0\u03bf\u03bb\u03cd \u03c0\u03c1\u03ce\u03b9\u03bc\u03bf confidence \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03c3\u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf model \u03ba\u03b1\u03b9 task. \u03a0\u03ad\u03bc\u03c0\u03c4\u03bf\u03bd, \u03bf\u03b9 \u03c3\u03c7\u03b5\u03b4\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03ad\u03c2 \u03b5\u03c0\u03b9\u03bb\u03bf\u03b3\u03ad\u03c2 \u03c4\u03c9\u03bd AR inference engines \u03b4\u03b5\u03bd \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03b8\u03b5\u03c9\u03c1\u03bf\u03cd\u03bd\u03c4\u03b1\u03b9 \u03b1\u03c5\u03c4\u03cc\u03bc\u03b1\u03c4\u03b1 \u03b2\u03ad\u03bb\u03c4\u03b9\u03c3\u03c4\u03b5\u03c2 \u03b3\u03b9\u03b1 masked diffusion.<\/p>\n<p>\u0393\u03b9\u03b1 e-commerce, marketing automation \u03ae customer-experience workloads, \u03b7 \u03b5\u03c0\u03af\u03c0\u03c4\u03c9\u03c3\u03b7 \u03b5\u03af\u03bd\u03b1\u03b9 \u03bb\u03b5\u03b9\u03c4\u03bf\u03c5\u03c1\u03b3\u03b9\u03ba\u03ae. \u0391\u03bd \u03ad\u03bd\u03b1 \u03c3\u03cd\u03c3\u03c4\u03b7\u03bc\u03b1 \u03b4\u03b7\u03bc\u03b9\u03bf\u03c5\u03c1\u03b3\u03b5\u03af \u03c0\u03bf\u03bb\u03bb\u03ad\u03c2 \u03c0\u03b5\u03c1\u03b9\u03b3\u03c1\u03b1\u03c6\u03ad\u03c2 \u03c0\u03c1\u03bf\u03ca\u03cc\u03bd\u03c4\u03c9\u03bd, \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03b9\u03c2 \u03c5\u03c0\u03bf\u03c3\u03c4\u03ae\u03c1\u03b9\u03be\u03b7\u03c2 \u03ae \u03c0\u03b1\u03c1\u03b1\u03bb\u03bb\u03b1\u03b3\u03ad\u03c2 \u03c0\u03b5\u03c1\u03b9\u03b5\u03c7\u03bf\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c4\u03b1\u03c5\u03c4\u03cc\u03c7\u03c1\u03bf\u03bd\u03b1, \u03b7 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ae \u03c7\u03c9\u03c1\u03b7\u03c4\u03b9\u03ba\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b8\u03b1 \u03b5\u03be\u03b1\u03c1\u03c4\u03b7\u03b8\u03b5\u03af \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03ce\u03c2 \u03bc\u03bf\u03b9\u03c1\u03ac\u03b6\u03bf\u03bd\u03c4\u03b1\u03b9 \u03c4\u03b1 steps \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03bc\u03cc\u03bd\u03bf \u03b1\u03c0\u03cc \u03c4\u03bf \u03bc\u03ad\u03b3\u03b5\u03b8\u03bf\u03c2 \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5. \u038c\u03c0\u03c9\u03c2 \u03ba\u03b1\u03b9 \u03c3\u03c4\u03bf <a href=\"https:\/\/twodots.gr\/perissoteres-gpus-mikrotero-kv-cache-kostos-llm-serving\/\">\u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03c4\u03bf\u03c5 LLM serving \u03bc\u03b5 GPUs \u03ba\u03b1\u03b9 KV cache<\/a>, \u03c4\u03bf \u03c3\u03c9\u03c3\u03c4\u03cc capacity plan \u03c7\u03c1\u03b5\u03b9\u03ac\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c4\u03b9\u03c2 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ad\u03c2 distributions \u03c4\u03c9\u03bd requests \u03ba\u03b1\u03b9 \u03cc\u03c7\u03b9 \u03ad\u03bd\u03b1 demo average.<\/p>\n<div class=\"td-decision-band\">\n<p class=\"td-decision-kicker\">Production gate<\/p>\n<p class=\"td-decision-title\">\u0391\u03b3\u03bf\u03c1\u03ac\u03c3\u03c4\u03b5 compute \u03bc\u03cc\u03bd\u03bf \u03b1\u03c6\u03bf\u03cd \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b5\u03c4\u03b5 \u03c4\u03bf bottleneck \u03c4\u03b7\u03c2 \u03c0\u03bb\u03ae\u03c1\u03bf\u03c5\u03c2 \u03b4\u03b9\u03b1\u03b4\u03c1\u03bf\u03bc\u03ae\u03c2<\/p>\n<p>\u0391\u03bd \u03bf host control loop \u03ba\u03c1\u03b1\u03c4\u03ac \u03c4\u03b7\u03bd GPU \u03b1\u03bd\u03b5\u03bd\u03b5\u03c1\u03b3\u03ae, \u03c0\u03b5\u03c1\u03b9\u03c3\u03c3\u03cc\u03c4\u03b5\u03c1\u03b7 accelerator \u03c7\u03c9\u03c1\u03b7\u03c4\u03b9\u03ba\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b4\u03b5\u03bd \u03bb\u03cd\u03bd\u03b5\u03b9 \u03bc\u03cc\u03bd\u03b7 \u03c4\u03b7\u03c2 \u03c4\u03bf serving \u03c0\u03c1\u03cc\u03b2\u03bb\u03b7\u03bc\u03b1. \u03a0\u03c1\u03bf\u03b7\u03b3\u03bf\u03cd\u03bd\u03c4\u03b1\u03b9 trace, representative load, quality checks \u03ba\u03b1\u03b9 queue model.<\/p>\n<\/div>\n<div class=\"td-step-list\">\n<p class=\"td-step-list-title\">\u0395\u03c0\u03c4\u03ac \u03ad\u03bb\u03b5\u03b3\u03c7\u03bf\u03b9 \u03c0\u03c1\u03b9\u03bd \u03b1\u03c0\u03cc production rollout<\/p>\n<ol>\n<li><span class=\"td-step-kicker\">Pasul 1<\/span><strong>\u039f\u03c1\u03af\u03c3\u03c4\u03b5 representative requests<\/strong>\n<p>\u03a7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03ae\u03c3\u03c4\u03b5 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03ad\u03c2 \u03ba\u03b1\u03c4\u03b1\u03bd\u03bf\u03bc\u03ad\u03c2 prompt, task \u03ba\u03b1\u03b9 output length \u03b1\u03bd\u03c4\u03af \u03b3\u03b9\u03b1 \u03bb\u03af\u03b3\u03b1 \u03bf\u03bc\u03bf\u03b9\u03cc\u03bc\u03bf\u03c1\u03c6\u03b1 demo prompts.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">Pasul 2<\/span><strong>\u0391\u03c6\u03b1\u03b9\u03c1\u03ad\u03c3\u03c4\u03b5 \u03c4\u03bf truncation bias<\/strong>\n<p>\u0395\u03c0\u03b9\u03bb\u03ad\u03be\u03c4\u03b5 generation budget \u03c0\u03bf\u03c5 \u03b5\u03c0\u03b9\u03c4\u03c1\u03ad\u03c0\u03b5\u03b9 \u03c3\u03c4\u03bf task \u03bd\u03b1 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9 \u03ba\u03b1\u03b9 \u03ba\u03b1\u03c4\u03b1\u03b3\u03c1\u03ac\u03c8\u03c4\u03b5 \u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ac \u03c4\u03bf \u03c0\u03bf\u03c3\u03bf\u03c3\u03c4\u03cc \u03ba\u03bf\u03bc\u03bc\u03ad\u03bd\u03c9\u03bd \u03b1\u03c0\u03b1\u03bd\u03c4\u03ae\u03c3\u03b5\u03c9\u03bd.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">Pasul 3<\/span><strong>\u0394\u03b9\u03b1\u03c7\u03c9\u03c1\u03af\u03c3\u03c4\u03b5 host \u03ba\u03b1\u03b9 device time<\/strong>\n<p>\u039c\u03b5\u03c4\u03c1\u03ae\u03c3\u03c4\u03b5 wall-clock, kernel execution, dispatch, mask construction, synchronization \u03ba\u03b1\u03b9 control-loop time \u03c3\u03c4\u03bf \u03af\u03b4\u03b9\u03bf trace.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">Pasul 4<\/span><strong>\u03a3\u03c5\u03b3\u03ba\u03c1\u03af\u03bd\u03b5\u03c4\u03b5 batching policies<\/strong>\n<p>\u0395\u03bb\u03ad\u03b3\u03be\u03c4\u03b5 single-request, per-request dispatch \u03ba\u03b1\u03b9 shared-step batching \u03bc\u03b5 \u03af\u03b4\u03b9\u03bf model, thresholds \u03ba\u03b1\u03b9 quality protocol.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">Pasul 5<\/span><strong>\u0395\u03c0\u03b1\u03bb\u03b7\u03b8\u03b5\u03cd\u03c3\u03c4\u03b5 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03b1\u03bd\u03ac batch size<\/strong>\n<p>\u03a4\u03c1\u03ad\u03be\u03c4\u03b5 repeated trials \u03bc\u03b5 \u03b1\u03bd\u03b5\u03be\u03ac\u03c1\u03c4\u03b7\u03c4\u03b1 RNG streams \u03ba\u03b1\u03b9 \u03b5\u03bb\u03ad\u03b3\u03be\u03c4\u03b5 numerical equivalence \u03c0\u03c1\u03b9\u03bd \u03b8\u03b5\u03c9\u03c1\u03ae\u03c3\u03b5\u03c4\u03b5 \u03b1\u03c3\u03c6\u03b1\u03bb\u03ad\u03c2 \u03c4\u03bf throughput gain.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">Pasul 6<\/span><strong>\u0392\u03b1\u03b8\u03bc\u03bf\u03bd\u03bf\u03bc\u03ae\u03c3\u03c4\u03b5 timeout \u03ba\u03b1\u03b9 capacity<\/strong>\n<p>\u03a7\u03c1\u03b7\u03c3\u03b9\u03bc\u03bf\u03c0\u03bf\u03b9\u03ae\u03c3\u03c4\u03b5 \u03b1\u03c1\u03ba\u03b5\u03c4\u03ac measured batches, arrival distributions \u03ba\u03b1\u03b9 tail-latency \u03c3\u03c4\u03cc\u03c7\u03bf\u03c5\u03c2 \u03b1\u03bd\u03c4\u03af \u03b3\u03b9\u03b1 \u03ad\u03bd\u03b1\u03bd \u03b3\u03b5\u03bd\u03b9\u03ba\u03cc timeout.<\/p>\n<\/li>\n<li><span class=\"td-step-kicker\">Pasul 7<\/span><strong>\u0398\u03ad\u03c3\u03c4\u03b5 release \u03ba\u03b1\u03b9 fallback \u03cc\u03c1\u03b9\u03b1<\/strong>\n<p>\u039f\u03c1\u03af\u03c3\u03c4\u03b5 \u03c0\u03cc\u03c4\u03b5 \u03c4\u03bf \u03c3\u03cd\u03c3\u03c4\u03b7\u03bc\u03b1 \u03bc\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 batch size, \u03b5\u03c0\u03b9\u03c3\u03c4\u03c1\u03ad\u03c6\u03b5\u03b9 \u03c3\u03b5 \u03b1\u03c3\u03c6\u03b1\u03bb\u03ad\u03c2 baseline \u03ae \u03b6\u03b7\u03c4\u03ac \u03b1\u03bd\u03b8\u03c1\u03ce\u03c0\u03b9\u03bd\u03b7 \u03c0\u03b1\u03c1\u03ad\u03bc\u03b2\u03b1\u03c3\u03b7 \u03cc\u03c4\u03b1\u03bd quality \u03ae queue stability \u03b1\u03c0\u03bf\u03ba\u03bb\u03af\u03bd\u03bf\u03c5\u03bd.<\/p>\n<\/li>\n<\/ol>\n<\/div>\n<p>\u0397 \u03bf\u03b9\u03ba\u03bf\u03bd\u03bf\u03bc\u03b9\u03ba\u03ae \u03b1\u03be\u03b9\u03bf\u03bb\u03cc\u03b3\u03b7\u03c3\u03b7 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03b5\u03c0\u03af\u03c3\u03b7\u03c2 \u03bd\u03b1 \u03c3\u03c5\u03bd\u03b4\u03ad\u03b5\u03b9 \u03c4\u03bf infrastructure metric \u03bc\u03b5 \u03c4\u03b7\u03bd \u03b5\u03c1\u03b3\u03b1\u03c3\u03af\u03b1 \u03c0\u03bf\u03c5 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03bd\u03b5\u03c4\u03b1\u03b9. \u03a4\u03bf <a href=\"https:\/\/twodots.gr\/ai-roi-axia-ergasias\/\">AI ROI \u03b4\u03b5\u03bd \u03ba\u03c1\u03af\u03bd\u03b5\u03c4\u03b1\u03b9 \u03bc\u03cc\u03bd\u03bf \u03b1\u03c0\u03cc tokens<\/a>, \u03b1\u03bb\u03bb\u03ac \u03b1\u03c0\u03cc \u03c4\u03bf \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03ba\u03b1\u03b9 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03ba\u03ac\u03b8\u03b5 \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03c9\u03bc\u03ad\u03bd\u03bf\u03c5 business outcome.<\/p>\n<h2 id=\"limitations\">\u03a0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03bf\u03af \u03c0\u03bf\u03c5 \u03b1\u03bb\u03bb\u03ac\u03b6\u03bf\u03c5\u03bd \u03c4\u03bf \u03b2\u03ac\u03c1\u03bf\u03c2 \u03c4\u03c9\u03bd \u03c3\u03c5\u03bc\u03c0\u03b5\u03c1\u03b1\u03c3\u03bc\u03ac\u03c4\u03c9\u03bd<\/h2>\n<p>\u0397 \u03ad\u03c1\u03b5\u03c5\u03bd\u03b1 \u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03b5\u03b9 \u03bc\u03af\u03b1 \u03bf\u03b9\u03ba\u03bf\u03b3\u03ad\u03bd\u03b5\u03b9\u03b1 \u03b1\u03c1\u03c7\u03b9\u03c4\u03b5\u03ba\u03c4\u03bf\u03bd\u03b9\u03ba\u03ae\u03c2, LLaDA base \u03ba\u03b1\u03b9 D2F, \u03ba\u03b1\u03b9 \u03bc\u03af\u03b1 GPU, \u03c4\u03b7\u03bd H200. \u03a4\u03bf task generalization \u03ba\u03b1\u03b9 \u03c4\u03bf configuration generalization \u03b5\u03be\u03b5\u03c4\u03ac\u03c3\u03c4\u03b7\u03ba\u03b1\u03bd \u03c7\u03c9\u03c1\u03b9\u03c3\u03c4\u03ac, \u03cc\u03c7\u03b9 \u03bc\u03b1\u03b6\u03af. \u0394\u03b5\u03bd \u03ad\u03b3\u03b9\u03bd\u03b5 \u03ac\u03bc\u03b5\u03c3\u03b7 \u03c3\u03cd\u03b3\u03ba\u03c1\u03b9\u03c3\u03b7 \u03bc\u03b5 \u03c5\u03c0\u03ac\u03c1\u03c7\u03bf\u03bd\u03c4\u03b1 dLLM inference engines \u03cc\u03c0\u03c9\u03c2 \u03c4\u03bf dInfer. \u0397 \u03b4\u03bf\u03bc\u03ae \u03c4\u03c9\u03bd 11 tiers \u03bc\u03b5 \u03c4\u03cd\u03c0\u03bf 178+29k \u03bc\u03b5\u03c4\u03c1\u03ae\u03b8\u03b7\u03ba\u03b5 \u03c3\u03b5 \u03ad\u03bd\u03b1 \u03bc\u03cc\u03bd\u03bf \u03b6\u03b5\u03cd\u03b3\u03bf\u03c2 thresholds \u03ba\u03b1\u03b9 \u03bc\u03c0\u03bf\u03c1\u03b5\u03af \u03bd\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03b5\u03b9\u03b4\u03b9\u03ba\u03ae \u03b3\u03b9\u03b1 \u03b1\u03c5\u03c4\u03ae \u03c4\u03b7 \u03c1\u03cd\u03b8\u03bc\u03b9\u03c3\u03b7.<\/p>\n<p>\u0391\u03c1\u03ba\u03b5\u03c4\u03ac \u03b1\u03c0\u03bf\u03c4\u03b5\u03bb\u03ad\u03c3\u03bc\u03b1\u03c4\u03b1 \u03c0\u03c1\u03bf\u03ad\u03c1\u03c7\u03bf\u03bd\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc single runs \u03c7\u03c9\u03c1\u03af\u03c2 repeated trials. \u0397 scheduling simulation \u03be\u03b5\u03ba\u03af\u03bd\u03b7\u03c3\u03b5 \u03b1\u03c0\u03cc \u03bc\u03cc\u03bb\u03b9\u03c2 \u03bf\u03ba\u03c4\u03ce measured batches, \u03b5\u03bd\u03ce \u03ad\u03bd\u03b1 \u03bc\u03b5\u03c4\u03b1\u03b3\u03b5\u03bd\u03ad\u03c3\u03c4\u03b5\u03c1\u03bf run 32 batches \u03bc\u03b5\u03c4\u03b1\u03ba\u03af\u03bd\u03b7\u03c3\u03b5 \u03c4\u03b7\u03bd capacity \u03ba\u03b1\u03c4\u03ac 24%. \u03a4\u03bf slot-based design \u03b4\u03b5\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03b7\u03c1\u03ce\u03b8\u03b7\u03ba\u03b5 \u03c3\u03b5 \u03c0\u03bb\u03ae\u03c1\u03c9\u03c2 \u03c3\u03c9\u03c3\u03c4\u03ae \u03c5\u03bb\u03bf\u03c0\u03bf\u03af\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03b7 \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1 \u03c3\u03b5 batch sizes 8 \u03ba\u03b1\u03b9 16 \u03b4\u03b5\u03bd \u03b5\u03c0\u03b1\u03bb\u03b7\u03b8\u03b5\u03cd\u03c4\u03b7\u03ba\u03b5 \u03b5\u03bc\u03c0\u03b5\u03b9\u03c1\u03b9\u03ba\u03ac.<\/p>\n<p>\u039f\u03b9 \u03c0\u03b5\u03c1\u03b9\u03bf\u03c1\u03b9\u03c3\u03bc\u03bf\u03af \u03b4\u03b5\u03bd \u03b1\u03ba\u03c5\u03c1\u03ce\u03bd\u03bf\u03c5\u03bd \u03c4\u03b1 \u03b5\u03c5\u03c1\u03ae\u03bc\u03b1\u03c4\u03b1\u00b7 \u03ba\u03b1\u03b8\u03bf\u03c1\u03af\u03b6\u03bf\u03c5\u03bd \u03c0\u03bf\u03cd \u03c4\u03b5\u03bb\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 \u03b7 \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b7 \u03ba\u03b1\u03b9 \u03c0\u03bf\u03cd \u03b1\u03c1\u03c7\u03af\u03b6\u03b5\u03b9 \u03b7 \u03c3\u03c7\u03b5\u03b4\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03ae \u03c5\u03c0\u03cc\u03b8\u03b5\u03c3\u03b7. \u03a4\u03bf \u03b9\u03c3\u03c7\u03c5\u03c1\u03cc\u03c4\u03b5\u03c1\u03bf \u03c3\u03c5\u03bc\u03c0\u03ad\u03c1\u03b1\u03c3\u03bc\u03b1 \u03b5\u03af\u03bd\u03b1\u03b9 \u03bc\u03b5\u03b8\u03bf\u03b4\u03bf\u03bb\u03bf\u03b3\u03b9\u03ba\u03cc: \u03c4\u03bf serving \u03b5\u03bd\u03cc\u03c2 \u03bd\u03ad\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5 \u03c0\u03c1\u03ad\u03c0\u03b5\u03b9 \u03bd\u03b1 \u03c7\u03b1\u03c1\u03b1\u03ba\u03c4\u03b7\u03c1\u03af\u03b6\u03b5\u03c4\u03b1\u03b9 \u03c0\u03ac\u03bd\u03c9 \u03c3\u03b5 \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc hardware, \u03bc\u03b5 budgets \u03c0\u03bf\u03c5 \u03b4\u03b5\u03bd \u03c0\u03b1\u03c1\u03b1\u03bc\u03bf\u03c1\u03c6\u03ce\u03bd\u03bf\u03c5\u03bd \u03c4\u03bf task, \u03bc\u03b5 profiling host \u03ba\u03b1\u03b9 accelerator \u03ba\u03b1\u03b9 \u03bc\u03b5 \u03c3\u03b1\u03c6\u03ae \u03b4\u03b9\u03ac\u03ba\u03c1\u03b9\u03c3\u03b7 \u03b1\u03bd\u03ac\u03bc\u03b5\u03c3\u03b1 \u03c3\u03b5 measured result, analytical rule \u03ba\u03b1\u03b9 future design.<\/p>\n<section class=\"td-service-cta\">\n<div class=\"td-service-cta-content\">\n<p class=\"td-service-cta-kicker\">AI \u03c5\u03c0\u03bf\u03b4\u03bf\u03bc\u03ae \u03bc\u03b5 \u03bc\u03b5\u03c4\u03c1\u03ae\u03c3\u03b9\u03bc\u03bf capacity<\/p>\n<p class=\"td-service-cta-title\">\u03a3\u03c7\u03b5\u03b4\u03b9\u03ac\u03c3\u03c4\u03b5 \u03c4\u03bf serving \u03b3\u03cd\u03c1\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc workload \u03c4\u03b7\u03c2 \u03b5\u03c0\u03b9\u03c7\u03b5\u03af\u03c1\u03b7\u03c3\u03b7\u03c2<\/p>\n<p>\u0397 TWO DOTS \u03c7\u03b1\u03c1\u03c4\u03bf\u03b3\u03c1\u03b1\u03c6\u03b5\u03af requests, model calls, latency, batching, monitoring, \u03ba\u03cc\u03c3\u03c4\u03bf\u03c2 \u03ba\u03b1\u03b9 fallback \u03ce\u03c3\u03c4\u03b5 \u03ad\u03bd\u03b1 AI workflow \u03bd\u03b1 \u03b1\u03be\u03b9\u03bf\u03c0\u03bf\u03b9\u03b5\u03af \u03c3\u03c9\u03c3\u03c4\u03ac CPU, GPU \u03ba\u03b1\u03b9 \u03b4\u03b9\u03b1\u03b8\u03ad\u03c3\u03b9\u03bc\u03bf budget.<\/p>\n<div class=\"td-service-cta-actions\"><a class=\"td-service-cta-button\" href=\"https:\/\/twodots.gr\/aftomatismoi-epicheiriseon-ai\/\">Vede\u021bi Automatiz\u0103ri pentru \u00eentreprinderi \u0219i IA<\/a><\/div>\n<\/div>\n<\/section>\n<section id=\"sychnes-erotiseis\" class=\"td-faq-section\">\n<div class=\"td-faq\">\n<p class=\"td-faq-heading\">\u00centreb\u0103ri frecvente<\/p>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a4\u03b9 \u03b5\u03af\u03bd\u03b1\u03b9 \u03ad\u03bd\u03b1 masked diffusion LLM;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u0395\u03af\u03bd\u03b1\u03b9 \u03b3\u03bb\u03c9\u03c3\u03c3\u03b9\u03ba\u03cc \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf \u03c0\u03bf\u03c5 \u03c0\u03b1\u03c1\u03ac\u03b3\u03b5\u03b9 \u03ba\u03b5\u03af\u03bc\u03b5\u03bd\u03bf \u03b1\u03c0\u03bf\u03ba\u03b1\u03bb\u03cd\u03c0\u03c4\u03bf\u03bd\u03c4\u03b1\u03c2 \u03c3\u03c4\u03b1\u03b4\u03b9\u03b1\u03ba\u03ac masked \u03b8\u03ad\u03c3\u03b5\u03b9\u03c2 \u03bc\u03ad\u03c3\u03c9 denoising steps, \u03b1\u03bd\u03c4\u03af \u03bd\u03b1 \u03c0\u03c1\u03bf\u03c3\u03b8\u03ad\u03c4\u03b5\u03b9 \u03b1\u03c5\u03c3\u03c4\u03b7\u03c1\u03ac \u03ad\u03bd\u03b1 token \u03ba\u03ac\u03b8\u03b5 \u03c6\u03bf\u03c1\u03ac \u03cc\u03c0\u03c9\u03c2 \u03ad\u03bd\u03b1 autoregressive \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0393\u03b9\u03b1\u03c4\u03af \u03b5\u03bc\u03c6\u03b1\u03bd\u03af\u03c3\u03c4\u03b7\u03ba\u03b1\u03bd \u03b1\u03ba\u03c1\u03b9\u03b2\u03ce\u03c2 11 \u03b5\u03c0\u03af\u03c0\u03b5\u03b4\u03b1 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1\u03c2;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a3\u03c4\u03bf \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03bf D2F setup, \u03bf \u03ba\u03b1\u03bd\u03cc\u03bd\u03b1\u03c2 \u03c0\u03c1\u03bf\u03c3\u03b8\u03ae\u03ba\u03b7\u03c2 blocks \u03ba\u03b1\u03b9 \u03c4\u03b1 thresholds \u03bf\u03b4\u03ae\u03b3\u03b7\u03c3\u03b1\u03bd \u03c3\u03b5 step counts \u03b1\u03c0\u03cc 178 \u03ad\u03c9\u03c2 468 \u03b1\u03bd\u03ac 29 steps. \u039f\u03b9 \u03c3\u03c5\u03b3\u03ba\u03b5\u03ba\u03c1\u03b9\u03bc\u03ad\u03bd\u03b5\u03c2 \u03c3\u03c4\u03b1\u03b8\u03b5\u03c1\u03ad\u03c2 \u03c0\u03b9\u03b8\u03b1\u03bd\u03cc\u03c4\u03b1\u03c4\u03b1 \u03b5\u03be\u03b1\u03c1\u03c4\u03ce\u03bd\u03c4\u03b1\u03b9 \u03b1\u03c0\u03cc \u03c4\u03b7 \u03c1\u03cd\u03b8\u03bc\u03b9\u03c3\u03b7.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u039c\u03c0\u03bf\u03c1\u03b5\u03af \u03bf scheduler \u03bd\u03b1 \u03c0\u03c1\u03bf\u03b2\u03bb\u03ad\u03c8\u03b5\u03b9 \u03c4\u03b7 \u03b4\u03c5\u03c3\u03ba\u03bf\u03bb\u03af\u03b1 \u03b1\u03c0\u03cc \u03c4\u03bf prompt;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u038c\u03c7\u03b9 \u03b1\u03be\u03b9\u03cc\u03c0\u03b9\u03c3\u03c4\u03b1 \u03bc\u03b5 \u03c4\u03b1 \u03c4\u03ad\u03c3\u03c3\u03b5\u03c1\u03b1 signals \u03c0\u03bf\u03c5 \u03b4\u03bf\u03ba\u03b9\u03bc\u03ac\u03c3\u03c4\u03b7\u03ba\u03b1\u03bd. \u0397 \u03ba\u03b1\u03bb\u03cd\u03c4\u03b5\u03c1\u03b7 \u03c4\u03b9\u03bc\u03ae \u03ae\u03c4\u03b1\u03bd R\u00b2=0,150 \u03b3\u03b9\u03b1 prompt length \u03c3\u03c4\u03bf GSM8K, \u03ba\u03ac\u03c4\u03c9 \u03b1\u03c0\u03cc \u03c4\u03bf \u03c0\u03c1\u03b1\u03ba\u03c4\u03b9\u03ba\u03cc \u03ba\u03b1\u03c4\u03ce\u03c6\u03bb\u03b9 0,5 \u03c4\u03b7\u03c2 \u03bc\u03b5\u03bb\u03ad\u03c4\u03b7\u03c2.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0393\u03b9\u03b1\u03c4\u03af \u03c4\u03b1 \u03bc\u03b9\u03ba\u03c1\u03ac generation budgets \u03ae\u03c4\u03b1\u03bd \u03c0\u03b1\u03c1\u03b1\u03c0\u03bb\u03b1\u03bd\u03b7\u03c4\u03b9\u03ba\u03ac;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a3\u03c4\u03b1 128 \u03ba\u03b1\u03b9 256 tokens \u03ba\u03cc\u03c0\u03b7\u03ba\u03b5 \u03b1\u03bd\u03c4\u03af\u03c3\u03c4\u03bf\u03b9\u03c7\u03b1 \u03c4\u03bf 100% \u03ba\u03b1\u03b9 \u03c4\u03bf 83% \u03c4\u03c9\u03bd requests. \u03a4\u03bf benchmark \u03bc\u03ad\u03c4\u03c1\u03b7\u03c3\u03b5 \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 truncation \u03ba\u03b1\u03b9 \u03ad\u03ba\u03c1\u03c5\u03c8\u03b5 \u03c4\u03bf \u03c6\u03c5\u03c3\u03b9\u03ba\u03cc latency variance \u03c4\u03bf\u03c5 workload.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0391\u03c0\u03cc \u03c0\u03bf\u03cd \u03c0\u03c1\u03bf\u03ad\u03ba\u03c5\u03c8\u03b5 \u03c4\u03bf 16\u00d7 throughput;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u03a3\u03c4\u03bf batch size 16, \u03c4\u03bf synchronized batching \u03bc\u03bf\u03b9\u03c1\u03ac\u03c3\u03c4\u03b7\u03ba\u03b5 \u03ad\u03bd\u03b1 forward pass \u03b1\u03bd\u03ac denoising step \u03b3\u03b9\u03b1 \u03cc\u03bb\u03bf \u03c4\u03bf batch \u03ba\u03b1\u03b9 \u03ad\u03c6\u03c4\u03b1\u03c3\u03b5 1,939 requests\/s \u03ad\u03bd\u03b1\u03bd\u03c4\u03b9 0,121 requests\/s \u03c3\u03c4\u03bf single-request baseline.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03b7 GPU \u03b4\u03b5\u03bd \u03b5\u03af\u03bd\u03b1\u03b9 \u03c3\u03b7\u03bc\u03b1\u03bd\u03c4\u03b9\u03ba\u03ae;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u038c\u03c7\u03b9. \u03a3\u03b7\u03bc\u03b1\u03af\u03bd\u03b5\u03b9 \u03cc\u03c4\u03b9 \u03c3\u03c4\u03bf \u03bc\u03b5\u03c4\u03c1\u03b7\u03bc\u03ad\u03bd\u03bf setup \u03c4\u03bf \u03ac\u03bc\u03b5\u03c3\u03bf bottleneck \u03ae\u03c4\u03b1\u03bd \u03ba\u03c5\u03c1\u03af\u03c9\u03c2 \u03bf CPU-side control loop. \u0386\u03bb\u03bb\u03b1 models, engines \u03ba\u03b1\u03b9 hardware \u03bc\u03c0\u03bf\u03c1\u03bf\u03cd\u03bd \u03bd\u03b1 \u03ad\u03c7\u03bf\u03c5\u03bd \u03b4\u03b9\u03b1\u03c6\u03bf\u03c1\u03b5\u03c4\u03b9\u03ba\u03cc breakdown.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u0391\u03c0\u03bf\u03b4\u03b5\u03af\u03c7\u03b8\u03b7\u03ba\u03b5 \u03cc\u03c4\u03b9 \u03c4\u03bf batching \u03b4\u03b5\u03bd \u03bc\u03b5\u03b9\u03ce\u03bd\u03b5\u03b9 \u03c4\u03b7\u03bd \u03c0\u03bf\u03b9\u03cc\u03c4\u03b7\u03c4\u03b1;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u038c\u03c7\u03b9 \u03c0\u03bb\u03ae\u03c1\u03c9\u03c2. \u03a4\u03bf paper \u03c0\u03b1\u03c1\u03ad\u03c7\u03b5\u03b9 \u03b4\u03bf\u03bc\u03b9\u03ba\u03cc \u03b5\u03c0\u03b9\u03c7\u03b5\u03af\u03c1\u03b7\u03bc\u03b1 \u03bc\u03b5 \u03c4\u03c1\u03b5\u03b9\u03c2 \u03c0\u03c1\u03bf\u03cb\u03c0\u03bf\u03b8\u03ad\u03c3\u03b5\u03b9\u03c2, \u03b1\u03bb\u03bb\u03ac \u03b4\u03b5\u03bd \u03bf\u03bb\u03bf\u03ba\u03bb\u03ae\u03c1\u03c9\u03c3\u03b5 \u03ac\u03bc\u03b5\u03c3\u03b7 quality evaluation \u03c3\u03c4\u03b1 batch sizes 8 \u03ba\u03b1\u03b9 16.<\/p>\n<\/div>\n<\/details>\n<details class=\"td-faq-item\">\n<summary class=\"td-faq-title\">\u03a0\u03bf\u03b9\u03bf \u03b5\u03af\u03bd\u03b1\u03b9 \u03c4\u03bf \u03b2\u03b1\u03c3\u03b9\u03ba\u03cc production takeaway;<\/summary>\n<div class=\"td-faq-content\">\n<p>\u039d\u03b1 \u03bc\u03b5\u03c4\u03c1\u03ac\u03c4\u03b5 \u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc workload \u03c3\u03c4\u03bf \u03c0\u03c1\u03b1\u03b3\u03bc\u03b1\u03c4\u03b9\u03ba\u03cc hardware, \u03bd\u03b1 \u03b1\u03c0\u03bf\u03c6\u03b5\u03cd\u03b3\u03b5\u03c4\u03b5 truncation-biased benchmarks \u03ba\u03b1\u03b9 \u03bd\u03b1 \u03c3\u03c7\u03b5\u03b4\u03b9\u03ac\u03b6\u03b5\u03c4\u03b5 batching \u03ba\u03b1\u03b9 scheduling \u03c3\u03cd\u03bc\u03c6\u03c9\u03bd\u03b1 \u03bc\u03b5 \u03c4\u03bf computational pattern \u03c4\u03bf\u03c5 \u03bc\u03bf\u03bd\u03c4\u03ad\u03bb\u03bf\u03c5.<\/p>\n<\/div>\n<\/details>\n<\/div>\n<\/section>\n<div class=\"td-source-list\">\n<p id=\"piges\" class=\"td-source-list-title\">Surse<\/p>\n<ul>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2608.23807\" target=\"_blank\" rel=\"noopener\">Amin, Afroz, Moghadampanah \u03ba\u03b1\u03b9 Nikolopoulos \u2014 Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware<\/a><\/li>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2508.09192\" target=\"_blank\" rel=\"noopener\">Wang et al. \u2014 Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing<\/a><\/li>\n<li><a href=\"https:\/\/docs.nvidia.com\/cuda\/cuda-programming-guide\/04-special-topics\/cuda-graphs.html\" target=\"_blank\" rel=\"noopener\">NVIDIA CUDA Programming Guide \u2014 CUDA Graphs \u03ba\u03b1\u03b9 \u03bc\u03b5\u03af\u03c9\u03c3\u03b7 \u03c4\u03bf\u03c5 CPU launch overhead<\/a><\/li>\n<\/ul>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Modelele LLM cu difuzie mascat\u0103 dezv\u0103luie mai multe tokenuri simultan, dar punerea lor \u00een func\u021biune necesit\u0103 profilarea resurselor CPU, GPU, precum \u0219i stabilirea bugetelor pentru procesarea \u00een loturi \u0219i generarea de con\u021binut.<\/p>","protected":false},"author":1,"featured_media":106447,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_gspb_post_css":"","footnotes":""},"categories":[199],"tags":[8988,7565,21047,20982,21046],"class_list":["post-98378","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-techniti-noimosyni","tag-ai-infrastructure","tag-artificial-intelligence-2","tag-gpu-computing","tag-llm-serving","tag-masked-diffusion-llm"],"_links":{"self":[{"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/posts\/98378","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/comments?post=98378"}],"version-history":[{"count":0,"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/posts\/98378\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/media\/106447"}],"wp:attachment":[{"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/media?parent=98378"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/categories?post=98378"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/twodots.gr\/ro\/wp-json\/wp\/v2\/tags?post=98378"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}