With over 20 years of experience, we transform your digital presence. We specialize in website and E-Shop development, SEO and Digital Marketing, ERP software and smart automation that take your business to the next level.
Address:
154 Egnatia Street, Thessaloniki 546 36
Phone:
+30 231 231 4095
E-mail:
hello@twodots.gr
Ready for the next step?;
OmegaUse‑OfficeVal measures quality, time, and cost across 100 Office tasks, demonstrating why AI agents require usability checks and human oversight.
An AI agent can generate a document in just a few minutes and at minimal cost. However, this does not mean that the document is accurate, usable, or ready to be delivered to a client. OmegaUse‑OfficeVal, a new open benchmark for time-consuming office tasks, aims to measure precisely this difference: not whether the model made a few correct clicks, but whether the final Word, Excel, PowerPoint, or PDF file is truly worthwhile for the person who requested it.
For professionals, e-commerce teams, and marketers, the message is practical. Speed and low inference cost are important, but they do not, on their own, prove productivity. If a deliverable requires extensive testing, formatting corrections, or rework, part of the theoretical savings is offset by human labor costs. That is why adopting AI as a ’first-line solution« requires clear role boundaries and acceptance criteria, not just low execution costs.
Short answer: OmegaUse‑OfficeVal shows that the evaluated AI agents are cheaper and generally faster than the specific human baseline, but they lag behind in terms of the quality of the final Office artifact. For a business, the correct unit of measurement is not clicks or tokens; it is the file that opens, remains editable, meets requirements, and does not impose hidden repair time on the employee.
The benchmark includes 100 long-running tasks, based on requests from professionals and adapted to ensure privacy. Each task provides the agent with a high-level instruction and one or more input files. The goal is to produce a final deliverable, such as a processed document, spreadsheet, presentation, or PDF.
The choice of the final artifact as the unit of evaluation is critical. The agent can work with a GUI, scripts, APIs, or a hybrid approach. It is not graded for following a specific path, but for delivering something that is correct, accessible, editable, and useful. This is a much closer simulation of a real-world task assignment.
Four Criteria for Evaluating an Office Agent
It tracks whether the agent followed steps, made clicks, or completed a specified path. It is useful for debugging, but not sufficient for accepting a deliverable.
It checks whether the DOCX, XLSX, PPTX, or PDF file is valid, editable, complete, and free of unwanted changes that require manual correction.
It rejects the entire operation if a critical check fails, such as opening a file, the ability to edit it, or preserving its structure.
It balances completed requirements against unwanted changes that increase the time or cost of manual repairs.
On average, tasks require 2.32 hours of human labor, with a median of 2.03 hours and a maximum of 8.35 hours. So we’re not talking about quick tasks like «change the color of a cell,» but rather workflows where planning, consistency, and error checking must be maintained over a long period of time.
The team started with 1,715 tasks proposed by professionals. After an initial review to ensure they met the criteria of a genuine operational need, a clear description, and a specific deliverable, 595 remained. Three senior experts independently assessed whether each task was sufficiently complex and long-term, yet still feasible for a human to complete. When at least two agreed, the task moved on to the next stage. This resulted in a pool of 282 candidates, from which 100 were ultimately selected.
The instructions were depersonalized without losing their original intent, while subjective requirements that could not be reliably verified were removed. The input files were reconstructed using LLM and manually corrected for realism, privacy, copyright, and consistency. Final acceptance required unanimous approval from three senior experts.
This process is valuable for any team building its own AI evaluation set: good tests are not just a random collection of prompts. The same principle explains why the AI benchmarks need transparency about the dataset, the scoring, and the failures hidden behind an average. They require clear results, controlled inputs, the removal of sensitive data, and criteria that can be applied repeatedly.
The 100 tasks are accompanied by 220 input files and require 115 output files. The inputs include 77 images, 63 DOCX files, 31 PPTX files, 25 XLSX files, 14 PDF files, and 10 video or audio files. The outputs consist of 48 DOCX, 40 PPTX, 24 XLSX, and 3 PDF files. The distribution was not artificially balanced; it was maintained to reflect the needs identified in the dataset.
This explains why a demonstration using plain text alone is not enough to demonstrate office automation. A real-world request may be brief, while the critical details are contained in images, videos, or embedded elements within a document. The agent must understand the content while simultaneously preserving the structure, formatting, layout, and editability.
The taxonomy covers intents such as reformat, restructure, annotate, extract, compute, and beautify, across domains such as academic documents, education, financial data, technology, administrative tasks, and business operations. For an e-commerce or marketing department, these correspond to familiar needs: reorganizing reports, performing calculations in spreadsheets, creating presentation decks, and cleaning up material for delivery.
Each task is associated with two metrics: human work time and a task price proxy. The time was measured by 20 annotators with basic office suite skills. At least two people completed each task without LLM assistance. When the times differed significantly, a third attempt was assigned. The final time was the average of the two fastest valid attempts.
The price proxy was derived directly from practitioners for approximately 20% of the projects, when there was reliable prior outsourcing experience. For the remaining projects, three experts provided independent estimates based on project category, complexity, and typical compensation. The aggregation method limited the impact of any outlier estimate.
The mean proxy value was $6.86 and the median was $5.11. The authors present the values in USD, converting CNY at an exchange rate of 1 CNY to 0.14609 USD, calculated as a 180-day average from January 19 to July 16, 2026. These amounts do not constitute a global labor price list; they are annotations for this specific benchmark and should be interpreted within that context.
OmegaUse‑OfficeVal avoids both exclusively human grading and the LLM‑as‑judge approach. It uses code-based verifiers derived from detailed rubrics, reviewed by experts, and calibrated by comparing human judgments with the code’s results.
There are two dimensions. The first checks whether the artifact is valid, opens, remains editable, and can be used in practice. The second evaluates the requirements of the specific task, rewarding what has been completed and penalizing unwanted changes that increase the need for repairs.
In total, there are 219 usability checks—an average of 2.19 per assignment—and 2,009 task-completion checks—an average of 20.09. If even one usability check fails, the task receives a score of zero. Only a usable deliverable passes the weighted completion score. For businesses, this makes perfect sense: an «almost correct» spreadsheet that won’t open is virtually worthless.
The GLM‑5.2, Qwen3.7‑Plus, Kimi K2.6, DeepSeek‑V4‑Pro, and MiniMax M3 models were evaluated, using the best submission by a junior worker for each task as the human baseline. The human achieved a total score of 27.79. The best model, GLM-5.2, scored 17.91, while Qwen3.7-Plus, Kimi K2.6, DeepSeek-V4-Pro, and MiniMax M3 scored 17.51, 17.00, 14.48, and 13.82, respectively.
On average, a human took 2.324 hours per task. The models ranged from 0.184 hours for DeepSeek-V4-Pro to 2.275 hours for MiniMax M3. The average cost per task was $6.8560 for the human proxy baseline, compared to $1.4823 for GLM‑5.2, $0.2152 for Qwen3.7-Plus, $0.7719 for Kimi K2.6, $0.6111 for DeepSeek-V4-Pro, and $1.7572 for MiniMax M3.
The conclusion drawn from the data is not that agents «aren’t worth it.» It is that their effectiveness has two distinct dimensions. They are already cheaper and generally faster, but this advantage has not translated into the quality of human-level output. Choosing based solely on token cost or latency can therefore lead to the wrong business decision.
Quality, time, and cost do not result in the same ranking
The four values are taken from the main results table of the study and pertain exclusively to this specific benchmark and scaffold.
The best submission by a junior worker for each task served as the human quality baseline.
GLM-5.2 had the highest average quality among the five agents evaluated.
Qwen3.7-Plus had the lowest average inference cost per task.
DeepSeek-V4-Pro had the lowest average time per task.
The researchers report a score, a time-weighted score, and a price-weighted score. This shows not only how well a model performed on average, but also how much human time or task value its successes account for. The model with the best simple average is not necessarily the one that performs best on the most economically significant tasks.
The GLM-5.2 had the highest simple score among the models, while the Qwen3.7-Plus had the highest time-weighted score, 34.73 versus 30.13, and a price-weighted score of 106.50 versus 93.51. These numbers do not directly predict a company’s ROI, but they demonstrate that the ranking changes when the weighting of the evaluation changes.
For a marketing team, this means that tests must reflect the actual portfolio of work. A model that makes many small corrections may have a good average, but could fail on a critical quarterly report. Conversely, a more expensive run may be worth it when it protects a high-stakes deliverable.
The distribution of scores shows the difference in reliability. Humans had the highest percentage of tasks scoring above 50, at 21%, and the lowest percentage of zero-score results, at 29%. GLM‑5.2 had 14% tasks scoring above 50. Qwen3.7-Plus had the lowest zero-score rate among the LLM agents, 38%, which the authors interpret as greater resilience in producing partial progress.
For DeepSeek-V4-Pro and MiniMax M3, the zero-score rates were 50% and 51%. Because the usability gateway filters out non-usable artifacts, these rates highlight a problem lurking behind impressive demos: the file may exist but may not constitute a reliable delivery.
At the same time, scores declined as human work time increased, for both humans and LLMs, with a more pronounced drop observed in the models. The authors attribute this pattern to prolonged planning, detailed file manipulation, and more opportunities for structural errors to accumulate.
The first change is to define the deliverable before the prompt. The team must describe what «opens,» «remains editable,» «does not break formulas or layout,» and «meets all requirements.» These can be converted into acceptance checks, even if there is no fully automated verifier.
The second is the distinction between usability and completeness. First, we check whether the artifact can be used without requiring a risky repair. Then, we rate its completeness. The third is the recording of repair time. If an employee needs forty minutes to correct a low-quality AI output, that time is part of the actual cost. At this point, a clear process for Who takes over when an automation fails?.
The fourth is weighting by business value. Critical tasks should not get lost in the average. An evaluation set for e-commerce can assign greater weight to financial reports, catalog operations, or presentations sent to customers, provided that the weights are based on actual priorities rather than arbitrary assumptions.
The benchmark is not a ready-made procurement policy. However, it provides a practical framework for acceptance testing for teams that automate documents, spreadsheets, presentations, or PDFs.
Acceptance Checklist for Office AI Workflows
Build the evaluation set using representative DOCX, XLSX, PPTX, and PDF files from the group, after removing any personal and confidential data.
Automatically reject files that won't open, lose their formatting or structure, can no longer be edited, or corrupt critical layouts.
Rate separately the items that have been completed and the unwanted changes that increase repair time or operational risk.
Measure the time spent on human review, correction, and re-execution along with the inference cost, so that the cost is not artificially inflated.
Give greater weight to artifacts that affect customers, financial decisions, campaigns, or regulatory obligations.
Require final approval by the owner for critical deliveries, and repeat the same tests whenever the model, prompt, tool, or file template changes.
OmegaUse‑OfficeVal does not prove that a specific model will perform the same way in your environment. It tests specific models, with a specific scaffold and 100 specific tasks. However, it provides a robust measurement framework: real-world requests, final artifacts, usability gates, task-level rubrics, and economic costs.
Today, safe adoption resembles a controlled collaboration between humans and agents more than it does a blind replacement. This complements the broader picture of AI in the Workplace and Greek Businesses, where true value depends on processes, skills, and responsible oversight. Agents can reduce time and inference cost, but the benchmark results show that final quality control remains essential, especially in time-consuming tasks where errors accumulate.
For a business owner, the question isn’t simply «How quickly did he complete the file?» It’s «how much useful work did it deliver, how much rework was required, and what was the value of the work it completed?» That’s where the conversation shifts from the AI demo to measurable productivity. For small teams, the same logic applies when the AI is treated as a first-time hire: Its role requires goals, boundaries, and quality control.
The decision to produce
Don't approve an Office agent just because they're fast or inexpensive.
Require a usable artifact, task-specific verification, recording of repair time, and a clearly designated human owner. If a file is not ready for delivery or requires extensive reconstruction, a low inference cost does not equate to productivity.
Business automation and AI from TWO DOTS
Turn Office AI demos into controlled workflows.
TWO DOTS designs automation solutions with acceptance criteria, human approval points, deliverable checks, and measurable repair costs for documents, reports, and recurring operations.
Frequently Asked Questions (FAQs)
It is an open benchmark featuring 100 time-consuming Office tasks, final deliverables, human time, a task price proxy, detailed rubrics, and code-based verifiers.
Evaluates the final DOCX, XLSX, PPTX, or PDF file as an artifact: whether it opens, remains editable, meets the requirements, and does not contain errors that increase the time required for manual correction.
The mean is 2.32 hours, the median is 2.03 hours, and the maximum is 8.35 hours.
Because the benchmark treats usability as a gateway. If the artifact does not open, cannot be edited, or fails another critical check, it is not considered a usable deliverable.
Yes, in this particular setup, all the evaluated agents were cheaper and generally faster, but they fell short in terms of the quality of the output.
The GLM‑5.2 had the highest simple score among the models, 17.91, compared to 27.79 for the human baseline.
Not directly. The scores apply to specific tasks, models, and scaffolds. A company must adapt the methodology to its own data, costs, risks, and repair times.
First, the file is checked to ensure it is usable and safe to process. Then, its completeness, unwanted changes, repair time, and the business value of the work are assessed.
