Background and Problem Description:
A major automotive manufacturer uses an AGV system to deliver parts to the assembly line in a just-in-time fashion. The system is considered “mission critical” and uses a relatively large number of vehicles (~50) that must operate wirelessly with the system’s central computer to carry out their delivery instructions. If any of the vehicles ever lose communications with the central computer, the vehicle is designed to stop until the problem can be resolved. Depending on the type of communication failure, the AGV could only be restarted in one of two ways: 1) an automatic reboot of its on-board computer which takes approximately 5 minutes, or 2) through manual intervention which could take up to 20 minutes. These “communication failures” were occurring roughly 10-15 times each shift. This downtime is unacceptable as it severely impacts the auto manufacturer’s ability to build product.
The auto customer and AGV manufacturer had performed extensive troubleshooting in an effort to determine the cause of the wireless issues. Their analysis led them to believe that there were multiple causes including: wireless coverage inadequacies, unidentified and intermittent RF interference, and excessive shock/vibration to the vehicles due to “potholes” along the cart path. They were unable to determine how to fix these issues on their own and they contacted Versona Systems because of our extensive experience with wireless networks in mission critical environments coupled with our in-depth knowledge of real-time application design, development, and deployment.
Once the Versona Systems experts arrived we took an entire systems perspective instead of only focusing on the wireless pieces of this puzzle. Our pursuit of root cause certainly involved employing standard wireless troubleshooting techniques to check for potential wireless issues (i.e. – Spectrum Analysis, Wireless Coverage Checking, Wire Shark Ethernet packet captures, and visual inspections). However, we also requested:
- O/S log files from the on-board AGV computers experiencing communication failures
- Communication log files from the HP network bridge from each AGV experiencing communication failures
- Communications log files from the AGV System central computer
- An inventory of model numbers and firmware levels for each on-board AGV computer and network bridge
- Access to the engineering teams from HP, Cisco, and the AGV System manufacturer (for assistance in interpreting the various log files)
We wanted to be sure that we could follow AGV system communications literally from end-to-end as this would be the only way to determine (for certain) where the communication breakdown was occurring.
After days of constant, real-time monitoring of the system, pouring through log files and Ethernet packet captures, and discussions with various engineering teams we began to get a clearer picture of the issues that confronted us. Our analysis led us to the identification of 2 problems that were causing the majority of the communication failures:
1. The network bridges on all AGV’s were not using the same firmware levels
2. There were low-level code incompatibilities between the HP network bridges and the Cisco access points that required firmware updates to the HP network bridges.
By taking a total systems approach (instead of merely looking at the traditional wireless issues of coverage, power levels, and channel assignments), we were able to leverage our considerable experience with Project Management, Advanced Wireless, and Systems Integration to supply the HP engineering team with enough data to enable them to create a firmware update for their network bridge that is truly compatible with a Cisco wireless infrastructure. We also worked with the AGV manufacturer to ensure that the network bridges for all AGV’s received this new firmware update.
Benefits of Using An Expert:
When you go to the doctor for a second opinion, you expect them to review every detail in your casefile from start to finish. As a patient, we do our best to “self-diagnose” our condition but a new set of eyes, from an expert in the field, may reveal something that was missed. As in this case, the customer believed they were plagued by a “traditional” wireless issue (they just couldn’t quite identify it). They came to this logical conclusion based on incomplete data they received from non-wireless professionals. In contrast, Versona Systems uses a proven methodology along with the industry’s most talented wireless engineers to troubleshoot and resolve mission critical wireless/communications issues. By taking an entire systems perspective and methodically troubleshooting the environment we were able to isolate and apply a fix for the true factors affecting this wireless infrastructure. For this customer, eliminating the AGV communication failures dramatically increased the assembly line’s uptime, which in an automotive plant translates to serious dollars.