A Robot Sees Clearly

With help from Multipix Imaging and MVTec Software GmbH, The Irish Manufacturing Research (IMR) have developed a robotic pick & place application for the medical sector. Machine vision software now enables robots to reliably recognize and process complex shaped, highly reflective components.

The interaction between robots and image processing software has made a giant leap forward in recent years. The main reasons are much more precise robotics hardware being available and improved performance of industrial image processing software. As a result, this combination is finding it’s way into more and more new industries and applications. This article is an example of the growing number of applications using robots with machine vision software, provided by The Irish Manufacturing Research (IMR).

The IMR is a leading research and technology institution with a broad portfolio of research, training, and consulting services within Industry 4.0. With the help of Multipix Imaging and MVTec, The IMR developed a fully automated robotic research application. MVTec HALCON machine vision software was used to identify and localize components for knee implants and subsequently grip and place them safely. The solution was developed for the healthcare industry.


Switch from manual to automated removal

The requirement was to replace existing manual picking with more efficient and less expensive automatic robot-based systems. The task required 3D image processing to robustly identifying and locate the randomly positioned parts and communicate with the robot system to repeatedly pick and place them safely.  For some parts this was very successful, however, some highly polished mirror-like finish and some very intricate shapes were much more difficult to implement. “For humans, this task is much easier to do initially,” explains Kevin Dooley, Project Manager at the IMR.

The IMR chose Multipix Imaging and MVTec as its project partners.

Multipix Imaging is a vision specialist, delivering a complete range of vision technology for today’s demanding applications.  Multipix has a highly experienced support team to ensure customers achieve exceptional results. Multipix tries to understand as much as possible about customer’s requirements before advising on the best technology to provide a successful solution. Multipix commitment then continues in supporting it’s customers during development and beyond. MVTec is a leading international software company for machine vision and are based in Munich, Germany. MVTec’s HALCON, MERLIC, and Deep Learning software products are used in a wide variety of industries.

Anum Rehman, Senior Researcher at the IMR explained “High-quality machine vision software was a critical component in the implementation because the nature of the surfaces of the implants is extremely challenging,”. The difficulty lies in the wide variance of the surfaces, from matt to highly reflective, as well as the complex shape forms.  There were additional challenges coping with the process environment: Parts are obscured by container walls, randomly arranged and must be gripped and deposited from containers of different sizes.

It was a highly demanding application. The complex-shaped, highly reflective parts needed to be localized in 6 degrees of freedom (DoF), in any orientation and the maximum cycle time needed to be less than 15 seconds. The solution needed to be a single system, capable of being used across all polishing stages, seamlessly operating and communicating with the robotic system. Lastly and importantly, for the application to be commercially viable, its price had to be below 30,000 Euros.


Interaction between hardware and software accurately to the millimeter

The implementation used a 6-axis UR3 robot, equipped with an end-of-arm tool (EOAT) located in the cell. To optimize the performance and to provide uniform ambient lighting, a large diffuse ring light was used with a standard area scan 2D industrial camera, with HALCON software running on a laptop PC host. The location of the 3D objects was calculated using HALCON 3D tools, and the coordinates of the localized parts were sent via TCP/IP to the robot controller.

The workflow was as follows: The container with the components is inserted into the robot’s work area.  All implants are then independently picked, without touching other implants, and sorted according to where they belong. To do this, the robot must know which component it is currently gripping and where to place it. Identical components are placed in the same container. Another challenge here was the uneven bottom surface of the bins, so it required a 6D positioning (x, y, z, A, B, C).


Shape Matching comes into play

For the robot to “see” the parts, the use of powerful machine vision software was necessary. This was found with MVTec HALCON, the comprehensive standard software for industrial image processing. Among other features, the software includes an integrated development environment (HDevelop) as well as state-of-the-art machine vision technologies, such as extensive 3D vision and deep learning. “We have been a distributor of MVTec HALCON for many years, as well as a partner of the IMR. Therefore, we knew both HALCON and IMR’s requirements. Thus, it was possible to test the functionalities of HALCON quickly and in a targeted manner. We were able to quickly ensure that all requirements for the imaging software were met,” says Simon Hickman, Managing Director of Multipix Imaging.

The relevant HALCON 3D technique for this application was Shape-Based 3D Matching. Shape-Based Matching locates the position of objects accurately and robustly. These HALCON tools are rotation / scale invariant.  They can accurately deal with perspective distortion, local deformation and partially occluded objects including objects that are on the edge of the image that are partly missing. It also copes very well with non-linear illumination variations. Shape-based matching processes 8- or 16-bit images, as well as colour and multichannel images.

Objects can be trained using CAD data only, without template images. “These are exactly the requirements we had for the machine vision software to be used in the application,” says Anum Rehman. In practice, the image recognition process pre-loads the 3D CAD models of the objects for picking into HALCON, to create 3D object models. HALCON then calculates different views of the parts specified by the user.

The views are automatically generated by placing virtual cameras around the 3D CAD model and projecting the 3D CAD model into the image plane of each virtual camera position. For each view obtained in this way, a 2D object representation is calculated. Thus, no real images of the object are used to generate the model, only the 3D CAD model. The object representations of all views are stored in the 3D shape model, which is returned by the operator and saved in a file for subsequent matching.

When picking the part, a 2D camera image provides the profile of the part and this is compared to the 3D CAD profiles within HALCON. Based on the comparison, a score is generated between 0 and 1 to determine the object pose.  3D coordinates are then generated and sent to the robot.


Benefits for the healthcare companies

The application developed by the IMR was completed in approximately four months and presented to various partner companies. One partner company has already deployed a solution based on the IMR’s research application. “We are proud that we were already able to convince a partner with our research results. The automated, robot-based application has met our expectations and those of our customers in terms of reliability, efficiency, and cost,” says Dooley.

As a result, the IMR is already developing other projects using robots and HALCON machine vision software, such as the detection of liquid reagents in a biomedical application. As for the benefits of automation for other applications and industries, Simon Hickman of Multipix Imaging says, “We are confident that more healthcare companies will take advantage of the combination of robots and machine vision software.  We see numerous future applications in quality control and robotic applications for pick-and-place.”

Contact us today to find out more on robotic pick and place

Contact us