PhD project: Behaviour-based pattern recognition

Name: Jean-Paul Oosten

Supervisor:
prof. dr. L.R.B. (Lambert) Schomaker

Summary of PhD project:

Teaching the computer to read

Ever thought it would be useful to be able to search through a book, just pressing “Ctrl-F” to find a keyword? New books are often available both in tangible and digital form. But a lot of historical archives have books that are not available digitally. Historians that use these books for their research have to do a lot of work just to find a page that contains what they are looking for. Although typed text is reasonably easy to convert to a digital form, handwritten text is very difficult. Typed text can be processed character by character. However, this is often not possible with handwritten words. As an example, the image to the right shows an example of a word that is hard to read, even for humans, without context. The word is “hartelijke”, Dutch for “cordial”. Without the entire word, certain letters would be very hard to recognize. In isolation, the second letter (“a”) can be mistaken for a very nice “u” as well.

Monk is a system that provides a search engine for handwritten text. Search for “hartelijke” with Monk, and you will find the pages that this word was written on, and it will even highlight the word in the image of the page. An important element in teaching the computer how to read is having a human teacher. Without someone telling the computer what word an image represents, the computer will not know whether his guess is correct. The Monk system already has easy tools to give the computer enough input to learn this task.

New research looks closer at human reading behavior. Humans read a text by making small eye movements. The eye usually moves from left to right, but sometimes it jumps back a little. Apparently, this behavior allows for effective reading, whereas the traditional way of reading for the computer consists of processing the entire image, pixel by pixel. We are now looking at methods of teaching the computer how read using similar eye movements. In the field of handwriting recognition, researches often use certain techniques for modeling a process that develops over time (such as eye movements). We have found that some of these techniques have caveats that are not described well in the literature. In a recent publication, we have discussed these caveats in detail, and we will now focus on improving these methods.

Reading handwritten text is not the final destination, however. Our goal is to learn how to recognize faces and objects in a complex scene as well. Possible applications for this will be robots in care for elderly or in hospitals.