Malaysia · Bahasa Malaysia

Benchmark profile

MyCulture makes the evaluation instrument visible.

Each underlying Malaysian cultural knowledge item is instantiated in parallel MCQ, CSRQ, and CRQ formats. This holds the knowledge target constant while changing answer exposure and scoring.

Dataset Read paper ↗
1,821knowledge targets
MCQVisible options
CSRQCanonical structure
CRQOpen response
Questions5,463
Parallel formats3
Cultural dimensions6
Human-audited sample5%
Qualified auditors5

Coverage

Six windows into Malaysian cultural knowledge.

MyCulture is a controlled testbed, not a claim to cover every Malaysian identity, community, or tradition.

01

Seni

Arts

Creative traditions, visual practices, and cultural expression.
02

Pakaian

Attire

Traditional garments, materials, meanings, and community use.
03

Adat

Customary practices

Social practices, ceremonies, and community traditions.
04

Hiburan

Entertainment

Games, performances, and shared recreational practices.
05

Makanan

Food

Dishes, preparation, ingredients, and food traditions.
06

Agama

Religions

Religious traditions and culturally situated practices.

Parallel format example

Same knowledge. Three elicitation and scoring procedures.

The example below assesses knowledge about the traditional Lansaran game of the Murut community.

01MCQ
Visible final options
Question

Apakah kenyataan yang betul mengenai permainan Lansaran dan budaya suku kaum Murut?

Which statements about the Lansaran game and the culture of the Murut community are correct?

Statements
  1. Lansaran dibina secara individu tanpa melibatkan masyarakat.Lansaran is built individually, without involving the community.
  2. Permainan Lansaran memerlukan ukuran kawasan antara 8 hingga 30 kaki persegi.The Lansaran game requires an area of between 8 and 30 square feet.
  3. Kayu “Selangan Batu” digunakan dalam pembinaan lantai Lansaran kerana ketahanannya.“Selangan Batu” wood is used to build the Lansaran floor because of its durability.
  4. Lansaran hanya untuk kanak-kanak dan tidak melibatkan orang dewasa.Lansaran is only for children and does not involve adults.
Options
  1. II, III
  2. I, IV
  3. I, II, IV
  4. II, IV
Selected answer A

Candidate final-answer combinations are visible.

02CSRQ
Option-free + deterministic
Question

Apakah kenyataan yang betul mengenai permainan Lansaran dan budaya suku kaum Murut?

Which statements about the Lansaran game and the culture of the Murut community are correct?

Response primitives
  1. Lansaran dibina secara individu tanpa melibatkan masyarakat.Lansaran is built individually, without involving the community.
  2. Permainan Lansaran memerlukan ukuran kawasan antara 8 hingga 30 kaki persegi.The Lansaran game requires an area of between 8 and 30 square feet.
  3. Kayu “Selangan Batu” digunakan dalam pembinaan lantai Lansaran kerana ketahanannya.“Selangan Batu” wood is used to build the Lansaran floor because of its durability.
  4. Lansaran hanya untuk kanak-kanak dan tidak melibatkan orang dewasa.Lansaran is only for children and does not involve adults.
Constructed answer B, C

No candidate subsets; exact canonical verification.

03CRQ
Open response + judge
Question

Huraikan ciri-ciri fizikal, cara pembinaan dan peranan komuniti dalam permainan tradisional Lansaran suku kaum Murut berdasarkan amalan tradisi yang dinyatakan.

Describe the physical features, construction method, and community role in the traditional Lansaran game of the Murut community, based on the traditional practices described.

Rubric · six required points
  1. Tempat permainan Lansaran berukuran antara lapan (8) hingga 30 kaki persegi.The Lansaran play area measures between eight (8) and 30 square feet.
  2. Lansaran dapat menampung lebih kurang 30 orang pemain.Lansaran can hold about 30 players.
  3. Lansaran dibina secara bergotong-royong di ruang khas rumah panjang suku kaum Murut.Lansaran is built through gotong-royong (communal work) in a dedicated space of the Murut longhouse.
  4. Beberapa batang kayu bulat yang keras dan liat sebesar lengan orang dewasa dikenali sebagai kayu “Selangan Batu”.Several hard, tough round logs as thick as an adult’s arm are known as “Selangan Batu” wood.
  5. Kayu “Selangan Batu” diikat dengan rotan saga bagi menampung lantai Lansaran.The “Selangan Batu” wood is tied with saga rattan to support the Lansaran floor.
  6. Lantai Lansaran dibina sekaki ke bawah daripada paras lantai rumah secara tergantung.The Lansaran floor is suspended one foot below the level of the house floor.
Generated answer Text

Open response evaluated against all six rubric points by a judge.

Full model performance

Explore every score. Compare every instrument.

The complete MyCulture result table becomes an analytical surface: filter model families, rank any metric, and draw direct score profiles across MCQ, CSRQ, and CRQ.

Rows in view23of 23 published rows
Leader · CSRQ strict56.78GPT-5
Mean MCQ → CSRQ strict gap29.74points in view · excludes format failures

Instrument score profile

Follow each model as the evaluation format changes.

Scores are percentages. Lines connect discrete instruments for comparison; they do not represent time.
Hover a point for its exact score5/6 models
0255075100MCQCSRQ LCSRQ SCRQ LCRQ S

Full published results

Every MyCulture model configuration.

Click a metric header to sort. Select up to six rows to draw.
FamilyModelReasoning
OpenAIGPT-5Yes88.1478.1056.7868.4122.90
OpenAIGPT-4o—90.2878.0056.0168.6121.14
GoogleGemini-2.5-ProYes85.5677.7855.6866.8720.26
OpenAIGPT-5-miniYes84.7377.5055.4667.2525.65
Z.aiGLM-4.6—87.3777.4554.9156.7217.68
AI SGQwen-SEA-LION-v4-32B-IT—83.2576.1653.5462.5518.23
Z.aiGLM-4.6Yes84.4676.3753.3863.2017.74
AlibabaQwen3-30B-Instruct—82.4875.2152.5059.0317.13
AlibabaQwen3-235B-Instruct—80.1273.5551.2962.9318.34
OpenAIGPT-4o-mini—78.3674.3049.2065.7820.37
OpenAIGPT-5-nanoYes80.1273.6148.3863.1020.76
GoogleGemini-2.5-Flash-LiteYes79.5772.9048.0062.5618.95
Z.aiGLM-4-32B-128K—75.1871.8847.8359.2616.09
GoogleGemini-2.5-FlashYes82.8771.0547.3965.2321.47
AlibabaQwen3-30B-ThinkingYes79.5770.4943.8262.4817.35
AlibabaQwen3-235B-ThinkingYes61.0162.6242.1250.1318.51
AlibabaQwen3-4B-ThinkingYes71.6166.4237.6748.4710.98
AlibabaQwen3-4B-Instruct—76.0665.6435.4855.9715.27
SeaLLMSeaLLMs-v3-7B—29.9852.9626.3047.627.86
BaselineRandom baseline—25.5340.334.28——
SeaLLMSeaLLMs-v3-1.5B*—0.000.160.0039.546.45
MesoliticaMaLLaM-1.1B*—0.000.000.0039.565.44
MesoliticaMaLLaM-5B*—0.000.000.0046.106.64
* Output could not be extracted in a parseable format for at least one structured setting.Reasoning models use medium effort where available. CRQ values use the paper's reported judge configuration.

Construction and validation

Grounded in sources. Filtered by evidence. Audited by people.

Reliable Malaysian cultural materials are transformed into matched evaluation items through automated checks and a qualified-resident audit loop.

  1. 01
    Curate sources

    JKKN materials and relevant academic publications.

  2. 02
    Generate matched items

    Draft MCQ, CSRQ, and CRQ versions from shared evidence.

  3. 03
    Filter automatically

    Check format compliance, answerability, and evidence consistency.

  4. 04
    Surface ambiguity

    Strong models flag potentially ill-posed items for review.

  5. 05
    Audit and iterate

    Five qualified Malaysian residents audit a 5% subsample.

Intended use

Use MyCulture to study evaluation sensitivity, not to reduce culture to a single score.

Recommended
  • Matched MCQ–CSRQ audit
  • Judge-sensitivity analysis
  • Format-following diagnostics
  • Malaysia-centered cultural evaluation
Use with care
  • Claims of complete cultural coverage
  • Claims of genuine understanding
  • Universal ranking conclusions
  • Unreviewed high-stakes deployment

Ready to examine the full methodology?

Read the paper and explore the dataset and evaluation toolkit on GitHub ↗.

Open the paper ↗