Virtual maintenance and training of fish farms through photogrammetry and virtual reality

BACKGROUND

One of the ongoing challenges facing the Scottish Salmon sector is open water dive planning. Fish net pens require regular maintenance or upgrade, with little or no pre-dive information available for planning purposes. Open water diving in Scottish waters is considered one of the most complex and dangerous roles in the sector and is still undertaken manually. This requires reconnaissance dives for development of a workplan detailing the skills, tools, and materials  required. Currently, a paper-based planning activity, this process requires multiple dives in challenging conditions, not only increasing costs and complexity for farm and contractor teams, but also the associated health and safety risks.

This project set out to determine the feasibility of using virtual reality (VR) and underwater 3D scanning technology to assess fish net pens for routine farming operations and to provide a platform for VR simulated training. This innovative technology proposes development of a ‘digital twin’ of fish farms by combining underwater 3D scanning and photogrammetry with design and production datasets allowing for better access to information ahead of routine maintenance dive operations.

By providing a platform for VR simulated training and therefore reducing overall dive time requirements, this technology has the potential to generate costs savings as well as helping to mitigate the health and safety risks of current processes. However, for certain remote locations across the sector, connectivity remains a limiting factor for uptake of such technology-based benefits. Therefore, the team at University of Strathclyde highlighted the need to investigate the ability of private 5G networks to provide digital access to photogrammetry data and moreover, how this connectivity can deliver further benefits to the Scottish Fish Farming sector.

AIMS

The overall aim of this project was to effectively demonstrate the use of VR technology and underwater photogrammetry along with private 5G networks for development of a training platform for the salmon farming sector.

This project was laid out with three main objectives:

1. To evaluate feedback from farmer and contractors on the use of VR technology for maintenance training & planning at net pen sites.
2. To identify private 5G networks for farms and areas of deployment.
3. To explore concepts of digital twins for farms using underwater ROV based photogrammetry and long-term benefits to farmer.

3D model of fish farm

PROJECT OUTPUTS

Proof-of concept: VR training scenarios for maintaining fish farms

The first phase of this work was to develop a proof-of-concept VR dive training scenario for maintaining net pen sites from initial concept to testing a prototype with the end-user. Through collaborations with Scottish Sea Farm (SSF) and dive contractors at Ocean Kinetics, project partners studied historic audio/video dive datasets to develop an understanding of fish farm operations, dive planning processes and data requirements for visualisation.

Following this, the project team worked with Viewport3 & SSF contracted dive teams to develop a 3D scanning process and carried out scanning of a selected fish pen. The 3D scanned model was then optimised by ViewPort3 and visualised on web platform for communication.

Both historical and 3D scanning data was used to build a VR environment, using game engine technology, which allowed for evaluation of future training scenarios with interactive immersive technology (VR), followed by user testing and validation with SSF.

Project partners successfully developed a technology concept allowing SSF and Ocean Kinetics teams to get hands-on and evaluate VR technology for their needs. This work sparked interest from SSF, who would like to explore this technology to train processing plant staff and explore public relations for whole salmon farming sector to showcase aquaculture to wider sector.

The NMIS team is looking to explore this technology further at next stage of R&D stage to build VR scenario for salmon processing plants and underwater dive activities.

Showcasing underwater photogrammetry for remote inspection and maintenance using Viewport3 technology

Photogrammetry delivers a low-cost capture solution for point cloud data, the underpinning data behind a virtual environment. Point cloud data, is a simple X, Y & Z reference coordinate, which when referenced can be used to build virtual models of real-world environments.

Underwater photogrammetry systems such as Viewport3, provide capture and processing of sub-sea point cloud data. Typically used in the Oil & Gas sector, the surface capture of underwater components can be used to visualise the build-up of potentially corrosive organic growth for development of preventative maintenance schedules and location specific dive planning.

The nature of point cloud data allows for the capture of wide reference visualisation of environments, which can be continually updated by local scans taken during maintenance dives.

This work showcased value of building digital twins of farms using underwater scanning through photogrammetry.

Overall, the feedback for this output was positive, but due to high cost of 3D scanning it was decided that this technology requires further development to reduce the financial implications. The cost of creating a photogrammetry model is approximately £15,000 versus the current ROV dives costing in the range of £3,000, making it non-practical for commercial uptake at present.

The development of an open digital twin platform of island environments and surrounding infrastructure could provide a positive boost for the aquaculture sector by allowing for simulation of new areas of fish farms, predict impact of weather, optimise logistics and improve planning for operations.

Project partners

Proof-of-concept: digital twin of fish net pens

To demonstrate this technology, the project team used Nvidia Omniverse technology to build a 3D model of Shetland using satellite imagery and then developed a generic overview of SSF fish pens within the model scene. This work was then showcased to SSF teams with mixed reviews from stakeholder.

The concept of 3D models of the whole environment was seen to be beneficial for the farm with opportunity to bring additional datasets such as sonar measurements of the seabed. This would allow the farm to plan for future sites, plan maintenance operations and use the model for training in future.

However, the concept of connecting digital twins to external stakeholders was not widely accepted with challenges around IP and cost of system.

Development of wireframed farm maintenance web platform

Throughout the initial research process, further bottlenecks within maintenance operations between the farm and dive contractors were highlighted. These challenges included a lack of visibility between operational team and dive contractors, difficulties in forward planning and the time-consuming paper-based planning processes.

This led the team to develop wireframes for a new web platform supporting maintenance operations, allowing farmers to manage their assets by geolocating historic datasets, prioritising dives, and visualising real-time data from divers.

The feedback for this platform from farms & contractors was mixed due to costly integration requirements with their current IT systems. However, the overall approach was favourable and could be considered for development.

Development of private 5G requirements for a location within SSF Shetland

Communication technologies are key enablers for innovation and collaboration, working with Celona (provider of private 5G equipment) the NMIS team trialled the feasibility of private 5G connectivity facilities to test network requirements for delivering immersive data over large distances for salmon farm environments.

As part of this, project partners were able to simulate the impact of a private 5G network for a farm location in Shetland and proposed benefits of at least 100mpbs downlink and 10mbs uplink with full control of spectrum.

The SSF team were not previously aware of private 5G networks and their capability to manage their own uplink & downlink with their software-defined radio, and as such, found this approach innovative, allowing them to connect either fibre or low-earth satellite communication to private 5G antenna and beam it across the water to enable their workers to access high bandwidth communication.

This output received positive feedback from all the stakeholders, and it is one of the technologies planned to be taken forward for the full R&D programme and to build a physical demonstrator in Shetland.

IMPACT

This feasibility project successfully demonstrated multiple innovations for the aquaculture sector, including virtual reality technology, private 5G networks and underwater photogrammetry to meet training needs within salmon farming.

The challenges that face the Scottish Salmon sector are not unique, however the application of technology to the sector could offer significant improvements to health & safety and overall productivity. Open-water diving is a hazardous environment to work in and thus provision of a VR training platform provides benefits in terms of safety and risk mitigation. Moreover, it demonstrates the advantage of using digital technologies in developing skills of current and new divers allowing for practice of repair jobs multiple times prior to the actual dive.

As a feasibility assessment, this project has shown the potential to grow the trend for digital networks in the sector and the associated benefits. Furthermore, the introduction of digital technologies has the potential to expand job opportunities within the sector, recruiting expertise from a skills base which would not normally be linked to Salmon farming such as Virtual & Augmented reality developers, app developers, data management and cyber security.

The unique collaborations brought on by this work have developed long-term opportunities for several cross-sector companies. For example, Aberdeen-based offshore energy sector company Viewport3, introduced to the aquaculture sector through this project, has developed a relationship with Ocean Kinetics discussing applications for scanning technology. In addition, the digital technology team at NMIS has established connections with both the Sustainable Aquaculture Innovation Centre and Scottish Sea Farms, gaining knowledge in the range of challenges currently facing the sector and, moreover, evaluating potential technology tools to improve processes to further support the aquaculture sector through future collaborations.

ADDITIONAL INFORMATION

• Shetland fish farm chosen as test bed for virtual reality (Press & Journal)
• UK Seafood Innovation Fund promotes new diving VR tool in aquaculture (We Are Aquaculture)
• Subsea sector provides platform for new diving VR tool (Fish Farmer Magazine)
• Virtual maintenance and training of fish farms through photogrammetry (Seafood Innovation Fund)
• Could a virtual reality tool boost the health and safety for divers on fish farms? (Responsible Seafood Advocate)