AI Positioning
Speaknow English Assessment – Using AI to Score Communicative Proficiencyfree
Aug 2023
Authentic
Introduction
In the age of digital communication, authenticity and accuracy in language proficiency assessments are more critical than ever. In both academic and workforce contexts, the expectation of effective communication skills is paramount.
Those skills are not merely a measure of one’s ability to use language “correctly” but to use it effectively to communicate ideas. The domain of communicative competence is inseparable from aspects of linguistic proficiency, and thus, an effective language assessment must take a multifaceted approach to measuring language proficiency. Speaknow offers a revolutionary solution leveraging artificial intelligence (AI) to measure communicative language proficiency resulting in personalized, high-quality, fair language assessment.
The Speaknow AI Rater is a proprietary AI-powered exam rating engine, which serves as the basis for Speaknow’s cutting edge assessment products. Our products generate accurate, reliable automated assessment results, revolutionizing the space by providing customers with an affordable solution that meets and often exceeds the quality standards of traditional assessment frameworks. The Speaknow AI Rater not only ensures an authentic, reliable, precise, and unbiased assessment but also incorporates measures to mitigate deceptive tactics and uphold the integrity of the exam. This white paper explores the various facets of the Speaknow AI Rater.
In a Nutshell: The Principles of the Speaknow AI Rater
The Speaknow AI Rater represents a breakthrough in evaluating open-ended responses in both spoken and written form. Trained on large amounts of genuine speech and writing samples which have been analyzed and tagged by multiple raters, it rates the following parameters:
Speech: fluency, pronunciation, and cohesion Writing: spelling, punctuation, organization, and topic development Speech and writing: grammar, vocabulary
These ratings are provided separately and combined into an overall score that is both comprehensive and meaningful. The overall score and the constituent ratings are provided as part of the exam’s result.
The Speaknow Assessment has been aligned with the CEFR (Council of Europe Framework of Reference for languages) from its conception. The CEFR is based on an understanding of evaluation of language proficiency as “promoting the teaching and learning of languages as a means of communication” (Council of Europe, 2020). In aligning itself to the CEFR, the Speaknow Assessment adopts this communicative emphasis on language assessment. The structure of the questions is intended to elicit what the exam taker “can do” with language. Similarly, the rating rubrics are based on those of the CEFR and the Speaknow AI Rater measures those proficiencies.
The Speaknow AI Rater leverages proprietary algorithms to assess each parameter independently. Trained on data tagged by expert raters with extensive experience in CEFR rating protocols, these algorithms excel at preserving the integrity of the evaluation, ensuring results that are accurate and unbiased.
In the research and development of the Speaknow AI Rater, the scoring models for each parameter have been meticulously engineered to be content-neutral. By grounding the algorithms in authentic speech and written data spanning a variety of topics, we enable the adaptability of test questions across diverse use cases. Importantly, this flexibility is achieved without any detriment to the robustness and accuracy of the resultant ratings.
Measuring Communicative Proficiency Via Authentic Language Production: The Speaknow Difference
What sets the Speaknow AI Rater apart from other AI solutions is its ability to score extended, open-ended responses to questions.
Rather than using different types of data to obtain scores for different aspects of language production, the Speaknow Assessment has been developed on the principle that language proficiency is multifaceted. While each of the aspects of proficiency may be rated separately, they are produced in combination. Thus, rather than using constructed response questions, such as read-alouds, to measure pronunciation and fluency, or retells to measure grammar, the Speaknow AI Rater is able to take extended responses to prompts and to rate the different aspects of language production based on the same responses. Each of the responses to speaking prompts is rated for vocabulary, grammar, pronunciation, fluency, and cohesion.
Similarly, writing proficiency is scored on the basis of extended written responses, either emails or essays. The written texts are analyzed for both rhetorical and linguistic features.
Using extended responses allows the Speaknow AI Rater to provide a detailed profile of an exam taker’s authentic English proficiency – the same type of proficiency that is required in authentic, communicative settings.
High-Quality Data - The Foundation of Speaknow's Algorithms
High-Quality data is the foundation of the Speaknow AI Rater. The quantity of data used for training, the quality of the ratings, and the multiple layers of data evaluation ensure accurate, precise algorithms. The following measures enhance the training process:
Independent Constituent Scoring Approach
The Speaknow AI Rater employs multiple AI models to grade multiple features of each language parameter.
While results of different parameters are generally somewhat correlated, to reach unbiased results the rating in each parameter should not affect the ratings of separate parameters. Thus, separate, independent models are used for each aspect of the grading to minimize the influence of one aspect of language production (such as pronunciation) on an unrelated one (such as grammar). This enables the final report to present a complete, comprehensive picture of the exam taker’s language proficiency.
Staying Updated: Continual Retraining of Models
Speaknow's algorithms are not static; they evolve.
As large amounts of new data are generated daily, and as user demographics expand and shift, the models undergo retraining. Continual quality assurance by human raters ensures algorithmic consistency. Additionally, expert linguists analyze discrepancies, guiding feature adjustments and refining the algorithms.
Fairness and Unbiased Results: Equality in Assessment
Speaknow takes pride in providing an equal opportunity platform for language proficiency assessment. With AI-driven scoring, we ensure that every exam taker, regardless of their background, stands on an equal footing. Our algorithms are blind to demographic details during the initial scoring, ensuring unbiased outcomes.
Moreover, the diverse backgrounds of our data tagging teams fortify our commitment to fairness. With data tagged by a multitude of raters from diverse language backgrounds, native and non-native, the resulting trained models ensure unbiased grading.
The capabilities of the Speaknow AI Rater offer a distinct advantage over traditional human raters. This approach effectively eliminates potential biases associated with human judgment, as well as simple human error. Furthermore, the capability to independently evaluate multiple elements of language proficiency provides an accuracy level that is challenging for human graders to consistently match.
To conclude, the Speaknow AI Rater ensures a precision and depth of language assessment that sets a new benchmark in the field.
Gauging True Abilities
To ascertain that exams genuinely gauge a user's proficiency, AI plays a pivotal role:
Anti-Deception: Open-ended questions and natural speech training negate any deceptive tactics aimed at tricking the system to provide false results. Tactics such as multiple repetitions, excessively loud or fast speech, or reading prepared answers are ineffective. Additionally, relevancy checks ensure that answers are appropriate to the questions being asked, and not prepared in advance or taken from an outside source. We continually ensure that genuine proficiency improvement remains the only pathway to better scores.
Proctoring: To further validate the authenticity of results, advanced proctoring solutions, including audio and video checks, ensure that exams reflect the actual abilities of exam takers, preventing cheating attempts like plagiarism or use of external help.
Conclusion
The Speaknow AI Rater offers a robust, accurate, and trustworthy tool for evaluating language proficiency.
Harnessing the power of artificial intelligence, it provides a comprehensive, fair, and authentic assessment experience and produces accurate, reliable results rapidly. Whether you are an individual seeking to gauge your language skills or an organization aiming to adopt a high quality, affordable solution fitted to your specific use case, Speaknow is your reliable partner in delivering authenticity, precision, and excellence in language assessment.
