About

arXiv Code Data Trained Models

🤔   Problem: Dialect differences cause performance issues for many types of users of language technologies. If we want fair, inclusive, and equitable NLP, our systems need to be dialect invariant: performance should be constant over dialect shifts.

💡   Solution: Multi-VALUE is a suite of resources for evaluating and achieving English dialect invariance. It contains tools for systematically modifying written text in accordance with 189 attested linguistic patterns from 50 varieties of English. Researchers can use this to (1) build dialect stress tests and (2) train more robust models using Multi-VALUE as data augmentation.

🧪   Experiments: You can reproduce experiments showing significant performance disparities in dialect QA, MT, and Semantic Parsing tasks. To fill these gaps, you can start by training on synthetic dialect data.

Attribution: Multi-VALUE was designed and built at the SALT Lab 🧂 at Stanford University 🌲 and the Georgia Institute of Technology 🐝 and was supported by Amazon Fairness in AI. This resource draws heavily from the Electronic World Atlas of Varieties of English (eWAVE), which aggregates the work of over 80 field linguists.

What Multi-VALUE can do for you

In short, Multi-VALUE can produce synthetic forms of dialectal text with modular control over which linguistic features are expressed. This will allow you to systematically isolate and the effects of certain syntactic and morphological structures on the performance of English NLP systems. Research applications include:

Getting Started

Use the following demo to start using Multi-VALUE.

Demo

[VECTOR DIALECT] Hong Kong English (abbr: HKE)
      Region: South and Southeast Asia
      Latitude: 22.26
      Longitude: 114.25

'I talk with them yesterday.'

{(2,7): {'value': 'talk', 'type': 'bare_past_tense'}}

Important Considerations

Limitations

Additional Considerations

Please keep the following points in mind when using Multi-VALUE

Data Agreement

I will not use Multi-VALUE for malicious purposes including not limited to: deception, impersonation, mockery, discrimination, hate speech, targeted harassment and cultural appropriation. In my use of this resource, I will respect the dignity and privacy of all people.

BibTeX

✂ ️Copy to Clipboard
copied

								  @inproceedings{ziems-etal-2022-multi-value,
      title = "Multi-VALUE: Evaluating Cross-dialectal NLP",
      author = "Ziems, Caleb  and
      Held, Will  and
      Yang, Jingfeng  and
      Yang, Diyi"
}