Accounting Expert: AI Model Evaluation

Overview

The organization is seeking experienced accountants to create realistic accounting tasks that evaluate how well frontier AI systems handle accounting work. This is an expert judgement role focused on realistic task design, careful review of model output, and precise, consistent documentation.

Training is provided. Prior AI training or data-annotation experience is required.

Responsibilities

• Write realistic accounting tasks: Create prompts of the kind a client, controller, or business owner would send, covering:
  • month-end close
  • bank and credit card reconciliations
  • AR/AP ageing
  • accruals and adjusting entries
  • sales tax
  • job costing
  • clean-up of incomplete or inconsistent books
• Build from real material: Review a deal room containing hundreds of files, select relevant documents, and use them to develop prompts and rubrics for tasks the frontier model struggles to complete correctly.
• Require usable deliverables: Each prompt should result in a working output, usually a spreadsheet, a journal entry schedule, or a short memo.
• Test against a frontier model: Run each prompt and review the output with the same care used when signing off accounting work. Increase difficulty where the model succeeds until a genuine gap appears.
• Write scoring rubrics: Define objective, checkable pass/fail criteria that another reviewer can apply without repeating your work.
• Apply guidelines consistently: Follow established task/rubric guidelines, flag ambiguous cases, and help refine the criteria.

Requirements

• At least 5 years in accounting, bookkeeping, or finance operations.
• Prior AI training or data-annotation experience is required.
• Hands-on knowledge of:
  • month-end close
  • reconciliations
  • AR/AP
  • accruals and adjusting entries
  • sales tax
  • job costing
• Excellent written English and precision with numbers.
• Careful and detail-oriented work with accurate figures and objective, checkable criteria.

Preferred Qualifications

• Prior experience reviewing and grading accounting-style deliverables for consistency and correctness.

Compensation & Benefits

• $60 - $90 USD / hour

Location

• Remote

Availability

• Available 20 to 40 hours per week for an initial one-week trial, with potential extension.

Confidentiality

• An NDA must be signed before the engagement begins.