The adoption of robotics and automation systems for inspection, maintenance and repair of facilities is becoming more prevalent. The application is especially valuable when used in inspection of net zero energy generation infrastructure. As the world faces a climate change crisis, it is imperative that industries move towards achieving net zero emissions.
There are many challenges that come with managing complex infrastructure offshore; mainly related to workforce resilience and cost reduction. To address these challenges, the Smart Systems Group department at the University of Glasgow is working towards creating a Symbiotic Multi-Robot Fleet (SMuRFs) approach. This would include a diverse group of robots working together to achieve a shared goal. By creating a cyber physical system, the team aims to combat challenges related to reliability, safety and productivity that inhibit the deployment of robotics in hazardous environments.
A cyber physical system could potentially streamline processes through optimization, increased interaction and centralized command to allow for query-based learning for a human-in-the-loop. This approach has high potential in helping reach net-zero as robots can increase the productivity of energy assets through autonomous inspection.
Multi-Robot Fleet Autonomy for Inspection Missions
The project’s importance lies in its potential to help facility managers and operators in the future when they will be required to scale up the coordination of their robots via the utilization of a wide range of robots for different types of inspection and maintenance missions.
Generally, when a robot faces a challenge that it is unable to overcome, a human must intervene to rectify the mission status for a robot. This keeps humans from completing other important tasks. Additionally, it is unsafe for humans to intervene in missions that are conducted in nuclear and offshore facilities. In such cases, instead of human intervention, the team aims to implement symbiotic interactions where a group of robots can help each other to overcome challenges within safety, resilience and reliability issues.
Figure1: A composite image depicting a Symbiotic Multi-Robot Fleet (SMuRF) in the future in an offshore renewable energy context.
Husky UGV for SMuRF Inspection Missions
To conduct an autonomous waypoint mission, a Husky UGV with dual UR5 arms was used as part of a SMuRF. Husky was chosen because of its ability to survive challenging terrain and carry heavy payloads. Furthermore, documentation and ROS packages for the robot are readily available for programming, making it an ideal choice to keep pace with the changing priorities of autonomous inspection missions.
“Husky provides easy integration of the required sensors that were needed in the development of our symbiotic system approach. It provides a scalable and modular architecture to readily customize the robotic platform to a range of mission requirements.” – Daniel Mitchell – PhD student, University of Glasgow.
Integrated Sensors
Figure 2: An image of the dual UR5 Husky A200 utilized during its first autonomous mission
In addition to the arms, Husky had several sensors mounted on it which were used during the autonomous mission. A 3D LiDAR was integrated and positioned to detect the wider environment and to give a 3D picture of the surroundings along with any potential threats above the robot. A 2D LiDAR was positioned to detect any potential threats low on the ground. The robot also had a sensor deployed on board one of the grippers that included a FMCW radar to perform structural health monitoring during the mission.
Implementation of Symbiotic Interactions
Husky was grouped with a DJI Tello drone, a Boston Dynamics Spot robot and an environmental monitoring system (LIMPET) as part of the SMuRF mission. Husky conducted a segment of a multi-robot autonomous inspection mission where it went from position A to position B completing its segment of the mission.
A user interface was created to allow for the human operator to coordinate and communicate with the robots. This included initiating different missions across the robotic platforms and overseeing the mission via several webcams acting as security cameras around the facility. The robot had a sensor onboard to perform structural health monitoring during the mission. This included a FMCW radar, which was used for inspection of corrosion, and surface and subsurface detection of defects within a decommissioned wind turbine blade. In a traditional scenario, a human would have to be deployed within a confined space to conduct a detailed visual inspection.
Operational Decision Support Interface
Employing humans for teleoperation of the robots would defeat the purpose of an autonomous system. Instead, the team created an operational decision support interface that allows for a single human operator to designate tasks to a multi-robot fleet. This allowed for teleoperation of some of the robotic fleet as it can be beneficial for a human-in-the-loop to teleoperate for specific complex tasks.
Figure 3: (Left) Operational Decision Support Interface where a human operator can command and control the SMuRF.
(Right) The Symbiotic Digital Architecture of the system of systems approach included in the design.
For the second part of the mission and to demonstrate symbiotic interactions, a fault was simulated on the Husky robot. As a result, when the second part of the mission was initiated, the robot did not operate as intended. Husky was programmed to use the arms to complete a second inspection. The self-certification software onboard the robot detected the fault and shared this information with the human operator in the loop via a descriptive message. A symbiotic interaction took place between Husky and Spot via the operational interface to rectify the problem where Spot was able to collect a battery pack from a stored position autonomously during the mission. The symbiotic interaction can be seen in this video where TensorFlow was used to train the Spot robot to detect the battery pack and Husky robot:
The team opted to use the tried and tested Husky robot as part of their Symbiotic Multi-Robot Fleet (SMuRF). This helped them save time and the cost of developing a robotic platform from scratch. It also ensured that their research efforts were strategically applied to key areas within their project.
The research presents the world’s first method to coordinate a wide range of robotic platforms implemented onboard real-world robotic platforms. It shows promising results in how the team’s symbiotic approach can advance multi-robot fleet mission performance within a constrained, beyond visual line of sight for inspection, maintenance and repair missions. The symbiotic interactions between diverse robotic platforms to respond to changing mission priorities demonstrates how this approach can greatly improve autonomous missions.
Future of Symbiotic Multi-Robot Fleet Approach
The team’s future plans are already in place. They recently collaborated with the University of Manchester, University of the West of England and Royal Holloway of London to apply their Symbiotic Multi-Robot Fleet approach in the nuclear sector. A Jackal UGV with radiation sensors was used in a robot fleet at an analog nuclear facility to replicate how robotics can be used for post-operational clean-out activities.
Additional plans include further development of the symbiotic approach through probabilistic modeling to simulate more scenarios in advance of the mission to reduce risks and improve efficiency in monitoring the success of a mission. These would include simulations for evaluating the probability of success of a mission and identifying approaches that may be more effective.
The team members involved in this project consist of Daniel Mitchell (PhD Student), Samuel Harper (Research Associate), Dr. Jamie Blanche (Research Associate), Wenshuo Tang (Research Associate), Shivoh Chirayil Nadakumar (PhD Student Heriot-Watt University), Dr. Theodore Lim (Lecturer at Heriot-Watt University), Dr. Ahmad Taha (Lecturer in Autonomous Systems and Connectivity), Prof. Muhammad Imran (Professor of Communication, Systems/Dean Transnational Engineering Education), Prof. David Flynn (Professor of Cyber Physical Systems)
Publications
D. Mitchell et al., “Symbiotic System of Systems Design for Safe and Resilient Autonomous Robotics in Offshore Wind Farms,” Oct. 2021, doi: 10.1109/ACCESS.2021.3117727.
D. Mitchell et al., “A review: Challenges and opportunities for artificial intelligence and robotics in the offshore wind sector,” Energy and AI, vol. 8, p. 100146, May 2022, doi: 10.1016/J.EGYAI.2022.100146.
O. F. Zaki et al., “Self-Certification and Safety Compliance for Robotics Platforms,” 2020, p. OTC-30840-MS, doi: 10.4043/30840-ms.
T. Semwal and F. Iqbal, CYBER-PHYSICAL SYSTEMS : solutions to pandemic challenges., 1st ed. ROUTLEDGE- Taylor & Francis Group, 2022.
Awards
1 Minute Visualize your Thesis Award- University of Glasgow – Symbiotic Multi-Robot Fleets – Mr. Daniel Mitchell
ISSS Postgraduate Research Prize for 2021 at Heriot-Watt University – Symbiotic System of Systems Approach
Images of Impact Competition – Future Impact 2023 – Runner Up – Mr. Daniel Mitchelll
If you would like to learn more about the Symbiotic Multi-Robot Fleet Autonomy approach, you can visit their new website and YouTube channel.
If you would like to learn more about Husky, you can visit our website here.
If you would like to learn more about Husky Observer, our all-terrain rugged robot with a payload for inspection, you can visit our website here.