Skip to content

aisingapore/math-psle

Repository files navigation

PSLE Math Analysis

How to use:

  • Eval-results-and-error-analysis:
    • raw_mmd.md is OCR MD output of PSLE paper from MathPix API
      • uses LaTeX for representing math formulas
    • To get raw_mmd.md -> questions.jsonl, use parse-md.ipynb
    • use run-p1a-ans.ipynb to generate ai-answers.md, then manually check with answer key and manually compose error-analysis.md
      • ai-answers.md has some minor manual touch ups but shouldn't be too far from raw output
      • i.e. I manually add the Answer Key Answer currently
  • parse-md.ipynb
    • question format optimised for use of Structured Input into model, sometimes multiple images can be sent, no images ok.
  • run-p1a/p1b-eval.ipnb
    • have different code to handle different question and answer formats for MCQ (p1-bookA) and SAQ (p1-bookB)

To Improve:

  • MathPix API is convenient and powerful for OCR but expensive for scale
    • i.e. 2B llama tokens requires ~ 3.3M pages of OCR, which will cost ~$80K with current pricing
    • looking into self-hosting Marker
  • Use "official" eval prompts:
    • for visual math QA involving diagrams, see MathVista or whatever is SOTA
    • for word problems only, see MATH or whatever is SOTA

About

evaluation of AI on PSLE math

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published