The growing need for critical infrastructure and technology as a service has fuelled a global demand for smart systems across all sectors of industry. Organizations are looking to augment their workforce with tools to increase productivity, efficiency, and create safer workplaces through analysis and removing people from hazardous situations.
The Smart Systems Group from the Heriot-Watt University in Edinburgh is working to promote sustainability and to create resilience in systems, organizations, networks, and societies. Their projects include collaboration with a global network of academic and industrial partners to deliver the flexibility, resilience, and sustainability that our global infrastructure requires. The work was completed in collaboration with the Offshore Robotics for the Certification of Assets (ORCA) Hub, MicroSense Technologies Ltd, and Offshore Renewable Energy Catapult. The cluster of this work was driven by the commercial needs of the offshore energy sector.
FMCW Radar inspection paired with two UR5 manipulators completing raster scan for detection of corrosion (left), FMCW deployed on autonomous confined space inspection mission on the pan-tilt unit at the front at the OREC facility (right).
Developing a Connected, Collaborative Environment
The team’s project, titled “Symbiotic System Design for Safe and Resilient Autonomous Robotics” is utilizing Husky UGV to create resilient and safe autonomous services for the offshore energy sector. To do this, the Smart Systems Group designed a “Symbiotic Digital Architecture” (SDA) that connected people, infrastructure, the environment, and robotics into a hyper-enabled collaborative environment. Their project addressed three key functions:
- Using robotic assistants to remove people from hazardous environments
- Unlocking the value of autonomous services for more effective asset integrity management operations through advances in mission resilience using improved situational awareness and run-time reliability analysis
- Creating an SDA that could be transferred to other missions and robotic platforms to accelerate the adoption of trustworthy autonomous service
But as is natural in the world of research and development, such an ambitious project had several considerations regarding past innovation and the goals that the team was looking to meet. To begin, they had to establish how to couple human knowledge and experience into a collaborative partnership with autonomous systems, while also monitoring a dynamic range of variables across factors such as environmental conditions, human interaction, robotic platform reliability, and dynamic safety cases.
Next, to improve “foresight” to changes in the mission environment, the team created a patented radar technology for the detection of obstacles outside of the mission space (e.g. people behind walls approaching a point of room entry during run-time autonomous inspections). To support such functions, they also created a sensor that could detect people and objects in opaque environments. Finally, for asset integrity inspection within a confined space, the team used radar technology and AI analysis.
Husky UGV Enables SDA Design
The use of an out-of-the-box robotic platform like Husky UGV allowed the team to test and develop many core features of their project. The Symbiotic Digital Architecture (SDA) supports the integration of distributed systems and human-in-the-loop observers, to demonstrate augmented run-time operational reliability and safety compliance.
The design of their SDA incorporates the complex mission variables across environmental conditions, safety cases, mission objectives, robotic platform state of health, infrastructure requirements, and remote human observer queries. It enables near to real-time and bidirectional communication and information exchange across all mission assets. Through these design features, the system has the potential to incorporate 1000 different assets e.g. sensors, actuators, robots, etc!
Husky UGV also enabled the SDA and its applications:
- A digital twin of the robotic asset converts diagnostic information from the run-time reliability ontology into human-readable actionable information about the health and fault status of the robotic platform. A 3D model of Husky UGV with a pair of manipulators was available to use which was highly adaptable for their system. The team could colour code the model to ensure that faults could be easily viewed by the operator.
- A ghosting function for the manipulators to simulate the trajectories of the arms before committing the action to the real-world asset to minimize risk and increase trust for a remote operator. Due to the accuracy of the 3D model created, the team could implement their ghosting function on the model in the synthetic environment and for the robot in the real-world. This allowed for the arms to be controlled from the twin and mirror the real-world asset.
- The description and integration of a novel inspection solution – Frequency Modulated Continuous Wave (FMCW) that supports multipurpose requirements, namely, surface and subsurface asset integrity inspection, safety compliance, and collision avoidance in poor visibility. The highly customizable architecture of Husky UGV enabled the team to easily integrate their novel sensing unit with the robot. They could position the sensor on the pan-tilt unit or on the manipulators. Husky UGV’s architecture also meant that the researchers could store their data onboard Husky UGV and/or send the data collected straight to the digital twin.
- Validation of the resilient autonomous mission using an integration of technologies within the SDA in a confined space asset integrity mission. The ontology could be integrated and linked with the team’s digital twin where the datasheets for Husky UGV were widely available and had plenty of information for the team to set up the fault and warning thresholds. The AI-assisted ontology then had more information to utilize when autonomously detecting faults and relaying the warnings back to the human-in-the-loop in the digital twin.
Demonstration of the robot’s “ghost twin”, showing the planned position of the manipulator as a faded manipulator and current position of manipulator (left) and augmented reality interface (right)
Husky UGV provided a compatible interoperable platform that supported the team’s technology-agnostic approach to the design, implementation, and evaluation of our SDA. Its highly customizable architecture supported the integration of their novel safety and asset integrity sensing technology – FMCW radar. Ultimately then, Husky UGV allowed the team to create a case study to show that their symbiotic system and self-certification framework are platform agnostic.
Creating a Dynamic Simulation Environment
The main functions of this team’s project included the collective goal of asset integrity inspection in the challenging offshore environment analogue utilized for their demonstration. In addition to core mission waypoints and objectives, the team introduced three major system issues to simulate changes in the operating environment, resulting in the need for the robotic platform to continually reassess the mission and system status; to evaluate the core objective of returning safely to the base station.
In doing so, the team demonstrated an ‘adapt and survive’ scenario, where the dynamic conditions imposed on the mobile robotic platform created the need for the onboard AI-assisted decision-making core to generate a continuously updating system. This reliability control system would be capable of ensuring that the robotic asset can accomplish processes such as identifying barriers to the success of the mission using integrated sensing or offering solutions to the computational decision-making elements, represented by both the AI and/or the human-in-the-loop operator.
The successful integration of the symbiotic system elements ensures that the robotic platform is always capable of completing its mission under optimal conditions or remains capable of assessing when the platform should replan and return to base when conditions become less favorable. At the core of this work is the need for the robotic platform to demonstrate resilience while operating autonomously and entirely within the envelope of safety compliance. Husky UGV can operate in a wide range of environmental conditions, increasing the uptime of the system.
“The platform provides a scalable and modular architecture to readily customize the robotic platform to a range of mission requirements. Using such a robust off-the-shelf system allowed us to demonstrate to decision-makers the significant advancements our research had made on the challenges of run-time safety compliance and reliability of robotic platforms.”
– Professor David Flynn
The SDA Loadout
To get this project up and running, the Husky UGV was equipped with a pair of UR-5 manipulators, a 16 beam Velodyne 3D LIDAR and a low-mounted SICK 2D LIDAR sensor. For asset integrity inspection and to support navigation in opaque conditions, they equipped a FMCW radar. Furthermore, they installed run-time reliability ontology onboard the platform and utilized a Microsoft HoloLens for augmented reality diagnosis of faults with access to the digital twin.
Design of the resilient symbiotic systems framework
Without the Clearpath robotic development platform, however, the Smart Systems Group would have been forced to create a robotic platform from scratch or purchase another type of robotic platform which may not have the mechanical and software components they required. The former approach, though, would be costly and time-consuming.
Instead, by using a tried and tested platform, the team could get straight into creating and developing further solutions, ultimately ensuring that their research efforts are strategically applied to key areas for advancement. The latter approach may have created potential difficulty in mounting their sensors and deploying their system in a wide range of scenarios.
The team chose Husky UGV as it simplified the integration of sensors, allowing the team to focus on developing their symbiotic system software. As well, Professor David Flynn spoke about its easy-to-use utility: “The platform provides a scalable and modular architecture to readily customize the robotic platform to a range of mission requirements. Using such a robust off-the-shelf system allowed us to demonstrate to decision-makers the significant advancements our research had made on the challenges of run-time safety compliance and reliability of robotic platforms.”
As well, the team found Husky UGV to be particularly suited for teaching and for advanced students such as mechanical engineering or Masters students and as a proof of concepts platform since the platform is ROS based and can traverse a wide range of terrains.
Asset Integrity Dashboard highlighting a tiered level of increasing description of the delamination fault on the decommissioned wind turbine blade
Creating Designs for the Future
A symbiotic system reflects the lifecycle learning and co-evolution with knowledge sharing for the mutual gain of robotic platforms and remote human operators. This enables run-time verification of safety, reliability, and resilience during autonomous missions.
The Smart Systems Group at Heriot-Watt University demonstrated their system at tier 1- ‘Adapt and Survive’; where an autonomous mission or service has predefined mission objectives. The “adapt and survive” control system can forecast and automatically mitigate against known and unknown risks in the mission environment. This capability is transformative in terms of run-time safety compliance and mission resilience. For example, if there were an emergent fault within Husky UGV, the system would automatically re-optimize the mission to ensure optimal outcomes that never violated safety rules. Alternatively, if someone entered the mission space or was actually located in the mission space, the SDA would reprioritize the mission to ensure the safety of or recovery of the individual.
To summarize then, the system can achieve the following:
- Evaluate the implications of a scenario of variables from the environment, infrastructure, human interaction, and robot reliability
- Share knowledge with and collaborate with a remote human observer
- Mitigate known and unknown threats to the resilience and safety case of the autonomous mission
In a future demonstration showcase the team’s plans to utilize a second robotic platform for a collaborative autonomous mission where the robotic platforms autonomously support each other. The value created from the symbiotic system framework will apply to host bidirectional data exchange and advance the goal of servitization, further advancing the roadmap to trusted autonomy and safety compliance in the offshore environment.
Ultimately, the team is quite happy with the success of their project. They have proven the seamless integration of an improved dashboard for beyond visual line of sight operation of autonomous systems due to the links between the human-in-the-loop, digital twin, run-time reliability ontology, and robotic platform.
Digital twin featuring a battery fault
The team is also hoping to garner some recognition for the project as they’ve submitted their symbiotic system demonstration summary video for the IEEE Access Best Video Award. As well, they had hoped to submit their Self-Repairing Cities- Robots for Resilient Infrastructure, ORCA Hub news article, and accompanying video footage to the Leeds Robotic Competition, but unfortunately it was canceled due to COVID-19.
The Smart Systems Group at Heriot-Watt University is composed of Daniel Mitchell, Dr. Osama Zaki, Dr. Jamie Blanche, Joshua Roe, Leo Kong, Samuel Harper, Dri. Valentin Robu, Dr. Theodore Lim, and Professor David Flynn.
If you are interested in reading more about the project, you can check-out the Smart Systems Group’s following publications:
- Symbiotic System Design for Safe and Resilient Autonomous Robotics in Offshore Wind Farms
- Interactive Digital Twins Framework for Asset Management Through Internet
- Self-Certification and Safety Compliance for Robotics Platforms
- Reliability and Safety of Autonomous Systems Based on Semantic Modelling for Self-Certification
- An Evaluation of Millimeter-wave Radar Sensing for Civil Infrastructure
- Run-Time Analysis of Road Surface Conditions Using Non-Contact Microwave Sensing
To learn more about Husky UGV, visit our website.