Literacy

At Quanta, John Pavlus writes about a new "artificial intelligence" program called Bidirectional Encoder Representations from Transformers (BERT) that regularly scores better than human beings on a particular kind of reading comprehension test.

The test measures how well a reader -- whether he, she or it -- comprehends material they have just taken onboard. It sounds a lot like some of the old Standard Reading Assessment tests I remember from school, only without the crutch of the multiple choice sorter. You could usually bet that if there were four possible answers that two of them were so obviously wrong it was funny, a third might sound OK but clearly showed some fatal flaw and the fourth was not just a right answer but the only possible right answer when the test-taker fired up more than two or three neurons.

Anyway, some old AI programs did really badly on the test, but it turns out that BERT regularly chalks up an 80 or 80 plus (kudos to Quanta for making the scientists in the illustration all resembled Sesame Street's Bert, by the way).

Some linguists and other scientists, though, think that BERT and similar problems "understand" what they read a lot less than they have just figured out some shortcut tricks that get to the right answer without knowing how or why it is right. It could be, for example, that my method of solving a multiplication problem featuring two three-digit numbers without a calculator or paper -- what's commonly called a wild-ass guess, for the curious -- will give me the right answer in that particular time. But it doesn't mean that I solved the problem. It just means that in this particular case my shortcut worked.

Although I found the article and the discussion very interesting, I think the headline hints at the reason it's ultimately moot and probably will be for the foreseeable future. It reads: "Machines Beat Humans on a Reading Test. But Do They Understand?"

Because if the AI was going to read like a person does, then the question would be something more like, "But Are They Inspired?" Because that's one of the things that reading can do. Leaving all the testing, algorithms, programs and shortcuts aside -- along with the nagging little reminder that the programs were created by people, not by themselves -- and that seems like a bar that it'll take BERT awhile to clear.