The JLPT, and its growing list of problems

I have known of the JLPT - the Japanese Language Proficiency Test - since I started studying Japanese years ago.  I've followed its various changes, the way that its popularity has grown considerably over the years, and I have also challenged myself to take the test to check my level more than once.

I was going to give a history of the test for people who aren't familiar with it, but blogging isn't the same as writing an essay.  For more information, go to the source.

I took the test in December 2010 with the simple goal of testing my Japanese level, and with no stakes riding on the result.  This article is about my experiences with the new test, and not at all about the contents of the test.

I should first mention that I attended the test in Toronto (Canada), and felt that it was very professionally handled in this test center.  Although the price had risen from $60CDN to $70CDN in 2010, and they decided not to serve refreshments (previously coffee + a muffin) at the break, I felt only minor disappointment about these things at the time.

I openly welcomed two of the proposed changes to the test, and originally felt that other changes wouldn't affect me much.  I was wrong.  My experience with the test revealed that many of these changes did nothing to improve the overall test experience, some did a grave injustice to the test respondents, and many areas which should have been overhauled were left with all their flaws intact.

The first change I took to be positive was the new biannual test schedule (instead of the old annual test schedule).  Unfortunately, the summer test is only available in a few countries in Asia, so this doesn't benefit me as I thought it would.

The other positive change was to address the gulf between the former 3級 and 2級, which was always problematic; think "gap between 2nd grade and the start of high school".  So they renumbered the old 3級 and 4級 to "N4" and "N5" respectively, and inserted a "N3" which is supposed to be about midway between.  I still think this is positive, but not as clearly as I had thought, due to other reasons I'll state below.

Four years ago, I easily passed the old 3級 level, and I attended the new N3 level this year.  My Japanese teacher encouraged us to take N2, but it still seemed slightly out of reach.  My feeling during the exam was that N3 was basically an appropriate level and still a bit challenging.  I felt that I was likely to achieve roughly a 70% overall, with a chance of being a bit lower on any given section.

My results, however, did not resemble this whatsoever.  My teacher and other classmates were as stunned as I was at my marks; there is no way in the world a 70% turns into a 12% in vocabulary, or a 26% in listening comprehension.  These could not possibly be my marks; they appeared to be the marks of an entirely different person.  In fact, that could well be what happened - but I'll never know.

This is really where my problems began, as there is simply no justification and no recourse with the JLPT.  All sales final.  Do not pass GO, do not collect $200.  Inscrutable, and they like it that way...

First, any professional certification or other reasonably official test will return results in a reasonable time; this should be basically within a month, and organizations around the world are constantly working hard to get these results returned more quickly.  Organizations other than the JLPT, that is:  I waited over 3 months to receive the results - from December 5 until after the earthquake, on March 17.

Three and a half months is completely unacceptable, and has not improved in all the years since I first knew about the test; in fact, I believe it has degraded by a couple of weeks.  As this is a multiple-choice test which can be computer graded by card reading/scanning, there is absolutely no reason for it to take this long.  Even though all the grading is done in Japan, the $70 fee should be sufficient to cover courier transit of the test materials both ways across the Pacific, accounting for only 2-5 days of total time.

Next, a certification test should be able to justify any set of marks returned.  For example, a key indicating for each question, which answer was chosen by the test respondent, versus which was the correct answer, together with the weight (marks) assigned to that question.  No such information is provided by JLPT.

I'm not asking them to return a full test... just the respondent's mark justification.  Many certification tests have a secondary revenue stream selling previous years' tests and answer keys - indeed, JLPT also had this policy in the past.  However, a recent change has changed this, ensuring that a test respondent has no ability to find out where he or she went wrong.  No record of the questions, the answers, the marks assigned per question, or of respondent's choices.  Sad, really.

Finally, the most confusing part - one which completes the circle of obfuscation, and provides absolute inscrutability to the testing organization:  SCALED SCORES.  The theory behind this is that actual test difficulty may vary from year to year, whereas the planned difficulty was not intended to change; the grade awarded to a respondent matches the "pattern" of answers the respondent provided, rather than having a predefined number of marks assigned to each question.

The testing organization has made references to several research papers on this type of grading, but the actual trouble with using this approach is that there is no transparency.  There is no audit, no identification of the basis upon which any given set of responses garners a different grade (which can't help but be subjective), nor any guarantee that they have actually implemented it properly as per the original theorists... or that this grading system even works convincingly, as each of the research papers referenced are written in Japanese, above the level of any test respondent who might take the JLPT.  For all we know, the JLPT has chosen to assign a random mark to a specific pattern of answers...

In summary, the JLPT is supposed to be a certification of Japanese ability, but it fails miserably at all the proper measures of a test:
(a) being a measure of overall ability, rather than just comprehension,
(b) returning answers to the respondent in a reasonable timeframe,
(c) being able to justify a respondent's specific grade and how to improve in future, and
(d) being able to explain properly how any grade is awarded.

I really wanted to put my trust in the JLPT, as it's basically the only widely-available measure of Japanese ability, but I simply can't accept it as a proper measure or even a useful tool in its current state.  Sadly, virtually every change to the test has been engineered to increase the authority and enhance the impunity of the testing organization, while simultaneously reducing its obligation to explain itself.  Such hubris is unconscionable, and will likely come back to haunt the organization in future, if students no longer find the test to be helpful.

There needs to be an alternative test for Japanese ability, as this one should be retired.  I would be honored to take any test which provides a proper framework addressing the points I have listed above, even a "new, improved" JLPT, if they should dare to take this criticism to heart and undertake constructive change.