<aside> 💡 MathGPT, a math-specific large language model (LLM) developed by Mathpresso, the creator of QANDA—Asia's premier AI-driven learning platform—has established a new global benchmark in math problem solving. It outperformed Microsoft’s Tora, the previous record holder in math benchmarks such as MATH (high school math) and GSM8K (grade school math), achieving first place among models under 13B. It also surpassed GPT-4 in the MATH Benchmark. We aim to enhance its accuracy and extend its language offerings, with plans to integrate the model into our products and offer it to businesses in need of specialized educational LLM solutions.

</aside>

Technical Details

Approaches

We adopted a data-centric methodology in developing MathGPT, securing state-of-the-art (SOTA) performance through this approach.

• Mathematics Data Curation
  • We collected a vast amount of data in various formats within the field of mathematics.
  • A comprehensive collection of diverse data types within the mathematics domain was amassed.
  • Our primary data source was Mathpresso’s proprietary database, obtained through its solution search app, QANDA.
• Programming of Thoughts (PoT) Approach
  • The PoT method was utilized, wherein a Large Language Model (LLM) crafts and executes a program to navigate the reasoning process flawlessly.
• Synthetic data generation
  • We leveraged existing LLMs to generate synthetic datasets, reflecting step-by-step solutions by humans and including datasets for the PoT approach
• Data selection for quality Improvement
  • We improved the performance of our model by filtering out low-quality or irrelevant data from training data sets

Performance evaluation

We evaluated the performance of MathGPT against the datasets commonly used for LLM evaluation. Several datasets were curated to evaluate the math problem-solving capabilities of machine learning models. They differ in the types and the difficulty levels of math problems. MATH and GSM8K are most commonly used as an indicator of math problem solving capabilities among those datasets. MATH dataset includes competition-level high school math problems in various areas such as algebra, number theory, probability, and geometry. GSM8K dataset contains grade-level word problems.

Additional Note

The paper detailing the technical aspects of this project is under review for publication.

Mathpresso’s next step and offerings

Mathpresso plans to continuously enhance the accuracy and performance of the 'MathGPT' model and ultimately integrate it into its learning interface as an AI tutor. Its vision is to weave this enhanced AI Tutor into its existing services, which include QANDA App, 1:N online school (QANDA Study), 1:1 online tutoring (QANDA Tutor), ensuring a deeply personalized educational experience for students.

In addition, Mathpresso is planning to develop MathGPT for other languages. The current version of MathGPT is trained with only English datasets. The following versions of MathGPT will support other languages, leveraging its vast math data.