Accounting Expert: AI Model Evaluation

Overview

The organization is seeking experienced accountants to write realistic accounting tasks that test how well frontier AI systems handle accounting work. This expert judgment role focuses on designing realistic task prompts, carefully reviewing AI outputs, and documenting scoring standards consistently.

Training is provided. Prior AI training or data-annotation experience is required.

Responsibilities

• Write realistic accounting tasks: Create requests similar to those a client, controller, or business owner would send, covering:
  • month-end close
  • bank and credit card reconciliations
  • AR/AP ageing
  • accruals and adjusting entries
  • sales tax
  • job costing
  • cleanup of incomplete or inconsistent books
• Build from real material: Review a deal room containing hundreds of files, select relevant documents, and use them to develop prompts and rubrics for tasks a frontier model struggles to complete correctly.
• Require usable deliverables: Ensure each request calls for a working output—typically a spreadsheet, a journal entry schedule, or a short memo.
• Test against a frontier model: Run each request and review the output with the care required when signing off accounting work. Increase task difficulty when the model succeeds until a genuine gap appears.
• Write scoring rubrics: Define objective, checkable criteria that another reviewer can apply consistently (e.g., verifying reconciled ending balances and correct classification of items).
• Apply guidelines consistently: Follow task and rubric guidelines, flag ambiguous cases, and help refine the criteria.
• Confidentiality compliance: Sign an NDA before the engagement begins.

Requirements

• At least 5 years in accounting, bookkeeping, or finance operations.
• Prior AI training or data-annotation experience is required.
• Hands-on knowledge of:
  • month-end close
  • reconciliations
  • AR/AP
  • accruals and adjusting entries
  • sales tax
  • job costing
• Excellent written English and precision with numbers.
• Careful, detail-oriented work with accurate figures and objective, checkable criteria.

Preferred Qualifications

• None specified.

Compensation & Benefits

• $60 – $90 USD / hour

Location

• Not specified.

Availability

• Available 20 to 40 hours per week for an initial one-week trial, with potential extension.

Example Assignment (Illustrative)

• Review a client deal room with incomplete and inconsistent books, selecting relevant documents (e.g., bank statements, ledgers, invoices, ageing reports).
• Write a prompt from the perspective of a business owner or controller requesting reconciliation and the adjusting entries needed to close the month.
• Run the prompt against a frontier AI model and review the spreadsheet or journal entry schedule line by line.
• If correct, increase difficulty (e.g., add a customer deposit, timing difference, or sales tax complication) until the model fails.
• Write a rubric with objective grading criteria (e.g., verifying reconciled ending balances and correct accounting treatment of deposits).