Accounting Expert: AI Model Evaluation

Overview

The organization is seeking experienced accountants to create realistic accounting tasks that test how well frontier AI systems handle accounting work. This is an expert judgement role focused on task design, careful review of model outputs, and precise, consistent documentation.

Training is provided. Prior AI training or data-annotation experience is required.

Responsibilities

• Write realistic accounting tasks that reflect the kinds of requests a client, controller, or business owner would send, covering:
  • month-end close
  • bank and credit card reconciliations
  • AR/AP ageing
  • accruals and adjusting entries
  • sales tax
  • job costing
  • clean-up of incomplete or inconsistent books
• Build tasks from real material by reviewing a deal room containing hundreds of files, selecting relevant documents, and using them to develop prompts and rubrics for tasks a frontier model struggles to complete correctly.
• Require usable deliverables such as working outputs (e.g., a spreadsheet, a journal entry schedule, or a short memo).
• Test against a frontier model by running each request and reviewing the output with the same level of care used when signing off accounting work. Increase difficulty until a genuine gap appears.
• Write scoring rubrics with objective, checkable criteria so another reviewer can grade outputs consistently.
• Apply guidelines consistently by following task and rubric guidelines, flagging ambiguous cases, and helping refine criteria.

Requirements

• At least 5 years in accounting, bookkeeping, or finance operations.
• Prior AI training or data-annotation experience is required.
• Hands-on knowledge of:
  • month-end close
  • reconciliations
  • AR/AP
  • accruals and adjusting entries
  • sales tax
  • job costing
• Excellent written English and precision with numbers.
• Careful, detail-oriented approach with accurate figures and objective, checkable criteria.

Preferred Qualifications

• Experience reviewing and grading accounting outputs with objective criteria.

Compensation & Benefits

• $60 – $90 USD / hour
• Availability: 20 to 40 hours per week for an initial one-week trial, with potential extension.

Location

• Not specified.

Confidentiality

• An NDA must be signed before the engagement begins.