SBIR-STTR Award

Deft Control Software (DCS) for Remote Robotic Operations with Underlying Structure
Award last edited on: 3/20/2023

Sponsored Program
SBIR
Awarding Agency
NASA : JSC
Total Award Amount
$874,657
Award Phase
2
Solicitation Topic Code
Z5.02
Principal Investigator
Fredrik Ryden

Company Information

Olis Robotics (AKA: BluHaptics Inc)

3601 Fremont Avenue North
Seattle, WA 98103
   (206) 747-8277
   info@bluhaptics.com
   www.bluhaptics.com
Location: Single
Congr. District: 07
County: King

Phase I

Contract Number: NNX17CJ39P
Start Date: 6/9/2017    Completed: 12/8/2017
Phase I year
2017
Phase I Amount
$124,754
BluHaptics proposes Deft Control Software (DCS), which utilizes machine learning to enable intuitive and efficient control of robotic arms in remote operations with underlying structure. The human-centered control methodology utilizes 3d sensor fusion for workspace visualization, machine learning with on-the-fly training, and pilot assist features to garner operator trust, improve safety, mitigate training latency, and support rapid task switching. The integrated algorithms identify and track underlying structure to enable pilot assistance and other safety features such as collision avoidance. DCS utilizes a common interface across robotic platforms and supports variable levels of autonomy specific to each task and/or operator. DCS permits robotic execution of exceedingly complex tasks that require high-levels of cognition and precise motor control which, to date, have been intractable for purely manual or automated control schemes to accomplish.The overall Phase I and II objectives are to: (1) demonstrate the value of a DCS interface to support intuitive manual control for remote operations, (2) demonstrate 3d visual-feedback and operator assistance supported by machine learning for tasks with underlying structure and varying levels of complexity, and (3) demonstrate the DCS platform can be extended to support different classes of robots with varying levels of autonomy. The objectives specific to Phase I are to: (1) Demonstrate intuitive manual control of a simulated NASA robot, (2) mitigate program risk by demonstrating basic assistive features, and (3) refine Phase II technical objectives based on collected user feedback and specific scenario requirements.

Potential NASA Commercial Applications:
(Limit 1500 characters, approximately 150 words) DCS algorithms developed as part of this programs can serve as a foundation with the potential to increases the range of beneficial tasks robots can perform and allows for improved safety and efficiency of operations across NASA mission scenarios. Future DCS algorithms could enable advanced capabilities for grappling of large payloads in space (Canadarm2 scope), for establishing and caretaking precursor structures and equipment on extraterrestrial surfaces (Valkyrie scope), and for supporting both robotic (rover scope) and astronaut-based (Robonaut 2 and MANTIS) exploration and science. Indeed, DCS could impact operational scenarios for many NASA funded projects, especially those relating to in-space manufacturing.

Potential NON-NASA Commercial Applications:
(Limit 1500 characters, approximately 150 words) DCS algorithms developed for NASA have numerous applications for other U.S. Government organizations. The algorithms can be used by the Combatting Terrorism Technical Support Office (CTTSO) to safely disarm Improvised Explosive Devices (IEDs) and support the Navy with its mandate from the EPA to safely mitigate subsea unexploded ordinances (UXOs). The algorithms could also enable the Navy Research Laboratory to repair, reposition and update space satellites. The largest single commercial market for DCS algorithms developed for NASA is arguably with subsea oil and gas field service companies who routinely utilize ROVs from OEMs such as Oceaneering and Saab to solve complex intervention challenges relating to the maintenance of subsea assets. Additionally, prospective commercial opportunities may be present in nuclear decommissioning or catastrophe response applications on demolition equipment like that offered by Brokk Robotics (Sweden). It should also be noted that DCS is expected to enable increased utility for a range of robotic systems and therefore has implications for the commercial viability of other hardware products (both sensor systems and robots) which would encourage further development and commercialization in the sector.

Technology Taxonomy Mapping:
(NASA's technology taxonomy has been developed by the SBIR-STTR program to disseminate awareness of proposed and awarded R/R&D in the agency. It is a listing of over 100 technologies, sorted into broad categories, of interest to NASA.) Algorithms/Control Software & Systems (see also Autonomous Systems) Autonomous Control (see also Control & Monitoring) Command & Control Image Processing Intelligence Man-Machine Interaction Operating Systems Perception/Vision Robotics (see also Control & Monitoring; Sensors) Teleoperation

Phase II

Contract Number: 80NSSC18C0044
Start Date: 4/23/2018    Completed: 4/22/2020
Phase II year
2018
Phase II Amount
$749,903
Future space missions will increasingly rely upon tele-operated robots to perform work remotely. This will require substantial improvements in robotic function, cognition, and human/machine interaction. The goal of this work is to realize this future by addressing key challenges in robotic space operations.BluHaptics has successfully demonstrated both the feasibility and potential impact of using our Deft Control Software (DCS) to provide pilot assistance to enable safe, intuitive and efficient remote teleoperation of NASA’s robotic systems. The primary goal of our Phase II effort is to develop and deliver a solution that enables intuitive tele-robotic control in dynamic scenarios, such as when targets and possibly interfering objects are moving in the workspace. This proposed approach combines 3D sensing and novel machine learning algorithms with simultaneous localization and mapping (SLAM) for workspace visualization SLAM, to obtain real-time tracking that provides pilot guidance. By increasing situational awareness and implementing safety features such as collision-avoidance, DCS can garner pilot trust, improvesafety, mitigate training time and support rapid task switching.We will perform physical experiments and task demonstrations using two separate robotic platforms that are readily available to our team: (1) MANTIS -- a compact and highly dexterous manipulator designed to fit inside an ExpressRack locker; and (2) Schilling Titan4 -- a work-class hydraulic manipulator commonly used in offshore oil and gas operations. Phase II work will demonstrate how these operator assistance capabilities impact the feasibility and effectiveness of complex robotic operations in two task scenarios that are key to the success of future NASA missions.

Potential NASA Commercial Applications:
(Limit 1500 characters, approximately 150 words) Even before commencement of this Phase II program, BluHaptics will collaborate with TUI first in their MANTIS Phase II program, and then within a soon-to-be proposed MANTIS Phase III program, where BluHaptics Dex-OS will provide basic control of the KRAKEN arm to enable swapping of microplate sample trays. This initial integration effort will also provide a foundation for the additional MANTIS related work as proposed here. Dex-OS can then be further developed for other programs such as TUI's orbital FabLab, which uses the KRAKEN arm to handle 3D printed parts and perform self maintenance. This type of joint development and collaboration provides a perfect example of how BluHaptics general commercialization strategy is applied in practice. DCS algorithms will enable operations support on earth to utilize simulation-based unsupervised machine learning and one- touch minimally supervised machine learning to quickly train the control system to enable assistive control to perform tasks with underlying structure on the ISS. BluHaptics will pursue Phase III programs with NASA to amplify the capabilities of platforms such as Robonaut2 to enable (for example) telerobotic work on the ISS such as swapping of express lockers, connecting/disconnecting cables, flipping switches and other repetitive tasks that could free up astronauts to perform higher value tasks.



Potential NON-NASA Commercial Applications:
:

(Limit 1500 characters, approximately 150 words) The DCS algorithms developed throughout this program will position BluHaptics to fill a major capability gap for the US Navy, which is the ability to autonomously (minimally supervised) connect electronic and communications subsea assets and to perform periodic maintenance. At a high level, the Navy would like to covertly install/maintain subsea monitoring systems, which is logistically difficult now as a surface vessel is required for the task. The machine learning algorithms developed as part of the program will allow naval operations to use reconnaissance data to train robots for intervention tasks to close the capability gap.Outside the USG sector, DCS algorithms can be adapted and integrated into Dex-OS to support advanced capabilities for the offshore energy industry, who seek to reduce operations expenses by moving control stations from the ship onto shore and also reduce dependencies on large support ships by making the ROVs resident near major subsea asset clusters. DCS algorithms can evolve as intervention tasks become more challenging either due to complexity or due to bandwidth and latency constraints.

Technology Taxonomy Mapping:
(NASA's technology taxonomy has been developed by the SBIR-STTR program to disseminate awareness of proposed and awarded R/R&D in the agency. It is a listing of over 100 technologies, sorted into broad categories, of interest to NASA.) Algorithms/Control Software & Systems (see also Autonomous Systems) Autonomous Control (see also Control & Monitoring) Command & Control Image Processing Intelligence Man-Machine Interaction Operating Systems Perception/Vision Robotics (see also Control & Monitoring; Sensors) Teleoperation