Pixy Pet tracks best if the object is a bright saturated color. It also helps if there are not a lot of similarly colored things in the environment to distract your Pixy.
Sometimes, moving to an area with different lighting (e.g. daylight vs. flourescent) can change the color appearance and confuse Pixy.