Color Machine Vision in Assembly InspectionBased on paper presented at:
1997 International Robots and Vision Conference
May 13-15, 1997
Cobo Center, Detroit, Michigan
AbstractColor vision can be an important tool in verifying the presence of the correct components in assemblies. Components in these assemblies are more often than not multicolored. Examples range from automotive fuses and interior fabrics to electronic components and compact disk packaging.
Complex coloration is important in human perception but it has presented serious obstacles to traditional machine vision approaches. Our proprietary statistical approach, minimum description, overcomes these problems. It also allows true training by example without the necessity of color space transformations, threshold setting, or other traditional operations requiring special technical knowledge or judgement.
IntroductionThere is a large class of assembly and packaging inspection applications which involves verifying that the right components are present in the right locations. In many of these applications the relative proportion of different colors, in one or more specified regions, can be used as the basis of inspection.
This image shows a classic example of such an application: an automobile junction box containing color coded fuses. The requirement is to verify that all fuses have been inserted in the correct locations. Note the non-uniformity of all colors and the very close similarity of the yellowish 20 and 25 amp fuse colors.
This image shows part of a printed circuit board containing a variety of components including diodes, resistors, integrated circuits, and capacitors. Here most components have been inserted using automatic equipment, some may have been inserted by hand. The requirement is to verify that all required components are actually in place, and that those which are polarity sensitive have been inserted in the correct orientation.
Modern automotive interiors often use multicolored fabrics, such as those shown here, for seating and interior panels. During production it may be necessary to determine which of several fabrics is being used in an assembly, and possibly even the orientation of this fabric. While easily handled by WAY-2C, fabrics like this may be extremely difficult to differentiate using traditional color machine vision methods .
One of the more interesting applications
of color based recognition we have encountered is illustrated in this image. In this case a variety of slightly different sun visors are manufactured in the same work cell.
The visors may be one of several different colors and may be for the left or right hand side of the vehicle. Some visors have vanity mirrors, some do not; some of the mirrors may be lighted.
Finally (not shown) a visor may have one of several different labels depending on the country of
destination. The inspection task is to determine all the above visor properties, and to print a bar code
and label which correctly indicate these properties.
Another interesting application of color based recognition is illustrated here. In this case a company packages compact disks in transparent plastic "jewel cases". Each package must contain a variety of components including: a paper insert in the bottom of the case, a tray to hold the disk, the disk itself, a booklet and a spine label each of these can be in a variety of colors and designs. All of these are surrounded by a transparent wrapper. Finally one or more stickers may be applied to the outside of the wrapper. The inspection task is to determine that each of the listed components is present and correct.
Each of these inspection tasks is characterized by objects of complex shape and color distribution. Yet after a brief training session, few humans would have any difficulty in inspecting the assemblies based on viewing the colors at the relevant locations. No formal segmentation of the region to be inspected, or setting of thresholds would be required.
Human inspectors, of course, have reliability problems, not because the job is hard, more likely because it is so easy that it becomes boring. Attention wanders, and defects are missed. Other human disadvantages are the comparatively slow speed and high unit cost per inspection.
Reliability and speed are hallmarks of computers. What one needs to perform the above inspections is a reliable, fast, cheap color distribution sensing computer which one can train simply by example but which can still handle even the most complex color distributions.
The requirements, theoretical basis for, and implementation of such a system are discussed below.
Color Vision System RequirementsThe first requirement of a general purpose color vision system is that it be able to handle any color distribution. This immediately puts some severe constraints on the matching method since the more one tries to reduce the number of independent variables used in the comparison function the more the ability to distinguish distributions will be reduced.
This requirement suggests in advance that methods which compute the mean, median, standard deviation, variance etc. of the test and reference data and then compare these statistics rather than the data themselves are likely to have limited success in comparing distributions. And indeed history has shown this to be the case.
A second requirement is the ability to train by showing. This makes it suitable for use by unskilled operators and removes the necessity of such sources of error and annoyance as threshold setting.
Other requirements are reliability, speed and low cost.
Theoretical BackgroundOne obvious way to maintain a large number of parameters to match, as implied by the requirement of generality, is to base classification on the entire color distribution, as represented by the 3-dimensional histogram. Functions can then be used which compare test histograms on a cell by cell basis.
Consider a pair of histograms, one representing the reference distribution R and the other a test distribution T. There are obviously many functions which might be used to compare these histograms to decide which reference histogram a test histogram most resembles. Some of these functions are referred to as distance functions since they decrease in size as the similarity between distributions increases. Typical distance functions include:
Over the years the above and similar functions have been tried with only limited success for color based inspection.
In addition to the problem of comparison functions there is problem of selection of the data on which to base the comparison. Most existing color machine vision systems which offer statistical matching must analyze the three color components separately. However both common sense and a rigorous analysis show that such separate analysis must inevitably introduce additional uncertainty. The deficiency may sometime be decreased somewhat when working with saturated colors and variable intensities by converting to HSI (hue, saturation, intensity) space while RGB space may be the better choice with unsaturated colors such as subtle tints. Most systems which must rely on separate component statistics offer the option of working in either space.
By replacing individual component analysis by a simultaneous analysis of all three color components, the advantages of one color space over another should become negligible. However, other problems have inevitably arisen when traditional statistical matching approaches have been applied to the complex color distributions associated with most scenes of interest. Too often they seem to work reasonably well when test and reference color distributions are almost identical, but require awkward, ad hoc, corrections or operator intervention to avoid bad decisions as the test and references begin to differ significantly. One way to interpret this is that the problems are inherently difficult, and will provide employment for "algorithm specialists" for the indefinate future. Another interpretation is that there is a better way to measure the similarity of the distributions. After all, birds, bees and other fauna with relatively small brains and little formal mathematical training seem to have little difficulty in performing this type of color recognition. Also, the most commonly used statistical matching methods are based on the assumption that the most likely color is very close to the mean color, and each color distribution has a single normal bell-shaped, Gaussian shape. The fact that the assumptions are valid in only a tiny fraction of the potential applications of the images of interest suggest a possible explanation of the difficulties.
Minimum Description ClassificationMinimum description is an information theoretic method of supervised pattern recognition which handles non-Gaussian, multi-modal and multi-dimensional data well. Its unifying theme is data analysis based on minimizing the amount of information necessary to describe a set of observations (1,2,3). Based on comparison of probability distributions, minimum description is consistent with both maximum likelihood estimation and the time honored philosophy of the simplest explanation being the best. It also lends itself to simple geometrical interpretations.
The method has now proven itself in a wide variety of industrial, government, and military inspection and process control applications.
Some Challenging ExamplesTwo examples of the system's power are shown here.
The first is an image pair illustrating the ability to interpret resistor color codes in the presence of glints, shadows and less than perfect markings and positioning. Each interpretation region is one pixel high and extends over the entire width of the region in which the component is expected. Note how classifications are not significantly affected by varying proportions of green and white in the background.
This shows three steps in separating a randomly multicolored capacitor from a multicolored background and then determining the polarity based on locating the dimple at one end. In the final image the end with the most bright green regions indicating background-like color is assumed to be the one with the dimple.
Practical Color Inspection SystemsWhile the theoretical foundation of minimum description may be of academic interest, the real test of the validity of the method is when applied to actual industrial inspection problems. A practicing industrial engineer can't get involved in a development problem every time he or she needs a sensor to determine whether widgets are correctly assembled. Ideally one should be able to implement simpler inspection systems from scratch in a couple of hours.
This is particularly true in this modern era of flexible manufacturing, short production runs and high quality, flexible easily trained inspection systems are essential. When a product may change several times in a shift, inspection system setup and retraining times of the order of a couple of minutes or less may be required.
For more complicated inspections, such as junction box inspection, more complex inspection systems, such as that shown here are necessary. In these cases the inspection scripts may have to handle a variety of different components and arrangements of these components using operators with only minimal computing skills. Training and inspection scripts are generated automatically from a database containing information on the location of the various fuses. Minor adjustments in position can be made simply by moving windows around with a mouse, and selecting actions from menus.
Our Assembly Color Inspection System and Canister Thermal Image Monitoring Systems are examples of the latter type of system. These start with some basic inspection scripts, but add a set of interface routines which can take the output of one operation to generate new scripts to be used in training, testing and on-line inspection. Such specialized applications make it possible to setup for a completely new assembly in a matter of several hours, and to switch between previously setup assemblies in minutes.
References1. Shannon, C.E., The mathematical theory of communication, Bell System Technical Journal, 27, 379 and 623, 1948.
2. McConnell, R.K. Jr. and Blau, H.H. Jr., A powerful, inexpensive approach to real-time color classification, Proceedings Soc. Mfg. Engs. Applied Machine Vision Conference '92, June 1-4, 1992, Atlanta, SME Technical Paper MS92-164, Society of Manufacturing Engineers, Dearborn, Michigan, 1992.
3. McConnell, R.K., Train-by-show in color-based assembly and packaging inspection, Machine Vision Applications, Architectures, and Systems Integration VI, Susan Snell Solomon, Bruce G. Batchelor, John W. V. Miller, editors, Proc. SPIE, Vol, 3205, 210-217, 1997.