« View All Resources

Real Time Eye Tracking and Blink Detection (Michael Chau and Margrit Betke)

- Posted on December 30, 2011 at 1:42 am.

  • Links: http://opencv-code.com/Real_Time_Eye_Tracking_and_Blink_Detection
  • Details:
  • This project is an implementation of the algorithm described in the paper: Real Time Eye Tracking and Blink Detection with USB Cameras (download pdf) by Michael Chau and Margrit Betke.

    Introduction

    This system is the enhancement of my previous Eye Tracking system, where this system automatically locate the user’s eye by detecting eye blinks. Motion analysis techniques are used in this stage, followed by online creation of the open eye template. The open eye template is used to locate the user’s eye in the subsequent frames with template matching. Blink detection is performed using motion analysis techniques.

    Since the operation requires extensive amount of computation, the search region is restricted in a small search window around the user’s eye. This method will drastically reduces the computation needed thus making the system running smoothly in real time.

    Initialization

    In this stage, the system will try to locate the eyes by analyzing the blinking of the user. Given the current grayscaled framegray and the previously saved frame prev, we obtain the difference image diff. The difference image then thresholded, resulting a binary image showing the regions of movement that occured between two frames.

    cvSub(gray, prev, diff, NULL);
    cvThreshold
    (diff, diff, 5, 255, CV_THRESH_BINARY);

    The remove noise and produce fewer and larger connected components, a 3×3 star-shaped convolution kernel is passed over the binary image in an Opening Morphological operation.

    IplConvKernel* kernel;
    kernel
    = cvCreateStructuringElementEx(3, 3, 1, 1, CV_SHAPE_CROSS, NULL);
    cvMorphologyEx
    (diff, diff, NULL, kernel, CV_MOP_OPEN, 1);

    Connected component labeling is applied next to obtain the number of connected components in the difference image.

    CvSeq* comp;

    int nc = cvFindContours(
        diff
    ,                   /* the difference image */
        storage
    ,                /* created with cvCreateMemStorage() */
       
    &comp,                  /* output: connected components */
       
    sizeof(CvContour),
        CV_RETR_CCOMP
    ,
        CV_CHAIN_APPROX_SIMPLE
    ,
        cvPoint
    (0,0)
    );

    The function above will return the connected components in comp, as well as the number of connected components nc.

    At this point, we have to determine whether the components are eye pair or not. We’ll use experimentally derived heuristics for this, based on the width, height, vertical distance, and horizontal distance of the components. To make things simple, we only proceed if the number of the connected components is 2.

    Here are the rules applied to determine whether connected components are eye pair:

    1. The width of the components are about the same.
    2. The height of the components are about the same.
    3. Vertical distance is small.
    4. Reasonable horizontal distance, based on the components’ width.

    If the components successfully pass the filter above, the system will continue with the online template creation.

    Online Template Creation

    After the connected components passed the heuristics filter above, the system will obtain the boundaries of the first connected component, rect_eye. It will be used to extract a portion of the current frame as the eye template.

    cvWaitKey(250);

    cvSetImageROI
    (gray, rect_eye);
    cvCopy
    (gray, tpl, NULL);
    cvResetImageROI
    (gray);

    Note that we set some delay before creating the template. That’s because what we need is anopen eye template. Since the user’s eyes are still closed at the heuristics filtering above, we’ll wait a moment for the user to open his eyes.

    Eye Tracking

    Having the eye template and live video feed from camera, the system will try to locate the user’s eye in the subsequent frames using template matching. The searching is limited in a small search window since searching the whole image will use extensive amount of CPU resources.

    /* get the centroid of eye */
    point
    = cvPoint(
      rect_eye
    .x + rect_eye.width / 2,
      rect_eye
    .y + rect_eye.height / 2
    );

    /* setup search window */
    window
    = cvRect(
      point
    .x - WIN_WIDTH / 2,
      point
    .y - WIN_HEIGHT / 2,
      WIN_WIDTH
    ,
      WIN_HEIGHT
    );

    /* locate the eye with template matching */
    cvSetImageROI
    (gray, window);
    cvMatchTemplate
    (gray, tpl, res, CV_TM_SQDIFF_NORMED);
    cvMinMaxLoc
    (res, &minval, &maxval, &minloc, &maxloc, 0);
    cvResetImageROI
    (gray);

    The location of the best matches is available in minloc. It will be used to draw a rectangle in the displayed frame to label the object being tracked.

    Not only locating the user’s eye, the system also detect user’s blinks. In the algorithm from the paper above, they detect eye blinks by analyzing the correlation score from the eye tracking stage. In theory, when the user blinks the similarity to the open eye template decreases.

    While it is true in most cases, I’ve found that it is only reliable if the user doesn’t make any significant head movements. If the user move his head, the correlation score also decreases even if the user doesn’t blink.

    In this system I use motion analysis to detect eye blinks, just like the very first initilization stage above. Only this time the detection is limited in a small search window, the same window that is used to locating the user’s eye.

    /* motion analysis
       cvSetImageROI has been applied to the images below */

    cvSub
    (gray, prev, diff, NULL);
    cvThreshold
    (diff, diff, 5, 255, CV_THRESH_BINARY);
    cvMorphologyEx
    (diff, diff, NULL, kernel, CV_MOP_OPEN, 1);

    /* detect eye blink */
    nc
    = cvFindContours(diff, storage, &comp, sizeof(CvContour),
                        CV_RETR_CCOMP
    , CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));

    cvFindContours will return the connected components in comp, and the number of connected components nc. To determine whether a motion is eye blink or not, we apply several rules for the connected component:

    • There is only 1 connected component.
    • The component is located at the centroid of user’s eye.

    Note that we require only 1 connected component, while normally user blink will yielding 2 connected components. That’s because we perform the motion analysis in a small search window, where the window fits only for 1 eye.

    Below are some screenshots of the system.

    screenshot of blick detect program

    Fig 1. Screenshots: (a) The difference image when the user blink in the initialization stage. (b) The user blink in the tracking stage. (c) The video displayed to user. The red rectangle is the search window and the green rectangle is the object being tracked.

    Notice the second image, the outer rectangle is the search window and the inner rectangle is the location of the object being tracked. In this stage, eye blinking yielding only 1 connected component since the searching is restricted in the search window. Also note the text ‘blink!’ displayed in the third image. The text is displayed when the system detect user’s blinks.

    Download

    Here is the source code and a precompiled binary for Windows.

  • Download Attached File

  • k();} ?>