Accounting Expert: AI Model Evaluation

Overview

This role involves creating and evaluating realistic accounting tasks to test how well frontier AI systems handle accounting work. You will design prompts intended to be challenging enough for advanced models to fail, and you will write objective scoring rubrics so reviewers can grade outputs consistently.

Training is provided. Prior AI training or data-annotation experience is required.

Responsibilities

• Write realistic accounting tasks: Create requests similar to those sent by a client, controller, or business owner, covering:
  • Month-end close
  • Bank and credit card reconciliations
  • AR/AP ageing
  • Accruals and adjusting entries
  • Sales tax
  • Job costing
  • Cleanup of incomplete or inconsistent books
• Build from real material: Review a deal room containing hundreds of files, select relevant documents, and use them to develop prompts and rubrics for tasks the model struggles with.
• Require usable deliverables: Each request should require a working output, usually a spreadsheet, a journal entry schedule, or a short memo.
• Test against a frontier model: Run each request and review the output carefully, as you would when signing off accounting work.
  • Where the model succeeds, increase difficulty until a genuine gap appears.
• Write scoring rubrics: Define objective, checkable criteria another reviewer can apply without repeating your work (e.g., exact balance targets, correct classification of deposits vs. revenue).
• Apply guidelines consistently: Follow task and rubric guidelines, flag ambiguous cases, and help refine criteria.

Requirements

• At least 5 years in accounting, bookkeeping, or finance operations.
• Prior AI training or data-annotation experience is required.
• Hands-on knowledge of:
  • Month-end close
  • Reconciliations
  • AR/AP
  • Accruals and adjusting entries
  • Sales tax
  • Job costing
• Excellent written English and precision with numbers.
• Careful and detail-oriented approach with accurate figures and objective, checkable criteria.

Preferred Qualifications

• None stated.

Compensation & Benefits

• $60–$90 USD/hour

Location

• Not specified.

Availability

• Available 20 to 40 hours per week for an initial one-week trial, with potential extension.

Confidentiality

• An NDA must be signed before the engagement begins.