Category: In the News

The Legacy of 9/11 for Disaster Robotics

[youtube]https://youtu.be/1DYi7_lE6cA[/youtube]

I’ll be speaking at the Smithsonian Museum of American History today as part of the 15th Anniversary of 9/11 event. 9/11 was the first reported use of robots for search and rescue and created a legacy that continues to grow for both disaster response and for science and technology. The robots were successful by any standard for rating search and rescue tools- improved performance over existing tools, frequency of use, and acceptance by professionals. They didn’t find any survivors, but neither did anyone else as there were sadly no survivors to find.
I will never forget my time at the World Trade Center as a responder, a scientist, or as a person. The infinite sadness of such an event still haunts me. The Seamus Heaney poem I read in the NYT and quoted in my IAAI talk:
And we all knew one thing by being there.
The space we stood around had been emptied
Into us to keep, it penetrated
Clearances that suddenly stood open.
High cries were felled and a pure change happened.
I believe the many members of the CRASAR team at the World Trade Center and since have kept the memories and have enabled a pure change- as witnessed by the use of robots at at least 50 disasters worldwide. While the robots are a very small story among the amazing stories of loss and triumph, I am proud to tell the story and add to the history of how 9/11  has made waves and ripples in history.

Legacy for Disaster Robotics

9/11 was an existence proof that small robots could be of significant use searching in rubble, reaching places that people and dogs could not, and penetrating two to three times farther than cameras on poles, which were the nearest similar tool. Large heavy robots had been developed for bomb squads but they were too big and heavy to be used in rubble, as seen at the Oklahoma City bombing. Red Whitaker at CMU had built even larger and heavier robots for Chernobyl and Three Mile Island nuclear accidents for the recovery operations, not for immediate search and rescue.

 

Small robots, ranging in size from a shoe box to a carry-on suitcase, that could be carried in one or two backpacks, had been under development by the pipeline and sewer inspection industry and the DARPA Tactical Mobile Robots program directed by John Blitch, the founding director of CRASAR.  If you look at the DARPA TMR logo you see that there is the “urban terrain” of cities but also a rubble pile, because John was thinking of dual use. He had been at the Oklahoma City bombing and had changed his MS thesis topics to robots for disasters (I was his co-advisor).

 

The robots were used starting shortly after midnight on 9/12 through 9/21 for search and rescue by FDNY, INTF1, OHTF1, PATF1,  and VATF1 and then again from 9/23 through 10/2 for recovery operations (structural inspection of the slurry wall) by the NY Department of Design and Construction when the last robot on site wore out.

 

By my count, robots have been used in 49 disasters since then in 17 countries. 24 of those disasters used UGVs- with the majority using the robot models from 9/11: Inuktuns (ex. mine disasters, building collapses), Packbots (ex. Christchurch for searching the cathedral, Fukushima Daiichi), and Talons (ex. Fukushima Daiichi). See Disaster Robotics for more details.

Legacy for Robotics

9/11 created a legacy for robotics in two ways. Search and rescue is often cited as a motivation for new advances in robotics; if you’re a doctor, you often say you want to solve cancer, if you’re a roboticist you often say you want to help with search and rescue.
One is that it created a new subfield of robotics. The IEEE Robotics and Automation Society, the largest and most prestigious professional organization devoted to robotics, has a technical committee on Safety Security and Rescue Robotics with an annual international symposium that started in 2002 (I was a co-founder of the TC and symposium). Both the European Union and Japan are investing heavily in small disaster robots- including sensors and user interfaces- with multiple projects being funded at the $20M to $35M range (the US doesn’t have dedicated programs for funding robotics projects at that level). The SSRR field now includes UAVs, UMVs, and many innovations in ground robots that can crawl and burrow into rubble.

 

9/11 also uncovered technical challenges that the R&D community is still struggling with. Probably the most significant discovery was that remote presence, or teleoperation, is actually the preferred mode of control for almost every response task- even with UAVs and UMVs.  Because the time pressure is so great and because disasters always have a surprise, the responders want to see in real-time what the robot is seeing and being able to opportunistically change up the plan (“wait— what’s that? Let’s look over there..”). Up until 9/11, researchers and developers had assumed that all robots should be taskable agents- you would tell it what to do, it would go off and do it, and then come back- and remote presence was just because we hadn’t created autonomous programs. Now there is the realization that many applications, not just search and rescue, require the human and robot to work together in a joint cognitive system to get the job done.

 

The second most significant discovery is what Jenny Burke would later describe in her PhD thesis as that 2 heads are 9 times better than 1. Up until 9/11, researchers and developers had assumed that 1 person could operate a robot successfully and thus the real challenge was for 1 person to drive 2 or more robots. We had had signs prior to 9/11 that 1 person couldn’t drive a robot in rubble and look at the same time-as one of my grad students who later went with us to 9/11, Jenn Casper, documented, they could do it but they could literally roll past a victim in front of the robot (we started seeing this in exercises with FLTF3 in buildings that were being demolished). The cognitive challenges of thinking like a ferret or meerkat (the size of the robot) were bigger than anyone had expected and then rubble is deconstructed and hard to mentally sort out. 2 heads makes sense in a way- if you are in a new town driving around in traffic and looking for a particular address you’ve never been to, it helps to have a passenger in the car who is looking too.

 

Italian Earthquake: Recommendations for using ground and aerial robots for immediate lifesaving

Our thoughts and prayers go out to the Italian people impacted by the earthquake. We’ve reached out to colleagues in Italy in case any of us here can be of assistance. Below is a general overview of what might be useful and why.

From the scanty news reports in the US, my guess is that this event will favor the use of small tethered ground robots for locating survivors in rubble based on the case studies from 9/11 World Trade Center, Cologne Archives collapse, Berkman Plaza collapse Prospect Towers collapse, L’aquila earthquake, Mirandola earthquake, and multiple mine disasters worldwide (see Disaster Robotics, MIT Press, 2014 for those case studies). UAVs may be of value in estimating extent, ascertaining whether roads are open or can be easily cleared to allow responders rapid access, and general damage assessment and recovery operations (as per Nepal and Chile), but probably not for direct life saving- though I could be wrong.

“Small”  as in pipe inspection robots- not a bomb squad robot like the Packbots used at Fukushima- because if a person or dog could get into a void to reach a trapped person, they probably would despite the personal risk. A tether is useful because it solves wireless and power problems- but more importantly any entry would likely be from the top of the structure or the upper parts, so the robot has to rappel down.

A video camera, color, is essential. Thermal cameras may be of use initially but are very hard to use for navigation in confined spaces. So I wouldn’t recommend thermal by itself, rather as a second camera. The value of a thermal camera goes away after a few days because decomposing bodies present a heat signature. Navigation gets harder as small protrusion become the same temperature as the surroundings.

A robot with 2-way audio will be valuable because the operator can call out and listen for sounds of survivors, then medical experts can talk with the victims. But even just a speaker or a microphone by itself can be useful.

Should someone find a survivor, a small tube can be attached to the robot to provide water to a trapped victim- hook up the end to an aquarium or koi pond pump. (This is a great solution worked out by Eric Rasmussen and we tested with the USMC CBIRF unite.) The robot can probably maneuver and bring a small payloads- a radio, a space blanket, power bars (assuming they aren’t severely injured). LACoFD does so many confined space rescues, they can have a kit the size of a Pringles can for trying to give to people trapped in caves and culverts.

More flooding– recommendations for small UAVs

Flooding continues through the southeast and we are getting some preliminary requests– here’s a quick rundown of  previous blogs:

suggestions from our work at the Texas floods where we flew with Lone Star UASC

a history of use of robots at floods

why the flood of data may be the biggest problem in floods

and some suggestions on flying for floods

plus best practices:

Let’s hope the flooding is not too bad- a bit of the luck of the Irish in time for St. Patrick’s Day.

The Taiwan Earthquake: robots

We are just getting word of several building collapses in the Taiwan earthquake, here are some thoughts and data on how robots have been used in previous collapses…

Ground robots may be of the most value. In a situation like this where the building has collapsed,  small robots will likely to get into voids and go deeper than the 18-20 feet that a camera on a probe or a boroscope can go into. Note that canines would normally be used first to indicate that people are alive (if there was any doubt about occupancy). The ground robots would be used to try to localize the survivors AND allow the rescue team to at the same time understand the internal layout of the structure. If they can better understand the internal layout of the “pixie sticks” of the rubble, they can extricate the victims faster and with less chance of triggering a secondary collapse. Most of the ground robots used, such as the Inuktun series which have been used the most, have 2 way audio so the responders can talk to the victims.

With our colleague Eric Rasmussen, MD FACP, we’ve experimented with how a small robot can carry tubing allowing a survivor to have water. With members of Texas Task Force 1 medical team, we’ve experimented with how doctors can use the robot to communicate with the survivor, assess their injuries, and engage the survivors- as it may take 4-10 hours to be extracted.

Similar situations where ground robots have been used for multi-story commercial building collapses are:

  • 2001 9/11 World Trade Center collapse
  • 2010 Prospect Towers collapse
  • 2011 Christchurch earthquake

Ground robots are often not used in earthquakes, such as the Japanese earthquake, because of building resilience and codes. Residential homes are small, often wood, and fairly easy to locate victims with canine teams and then extracting. Adding a robot doesn’t really speed up anything.

UAVs can give an overview of a collapse, but generally it has been the “inside” view that responders need the most and can’t get any other way.

 

 

CRASAR donates 2 E.M.I.L.Y.s to Hellenic Coast Guard and Red Cross

On January 14, 2016, the Roboticists Without Borders program hosted by the Texas A&M Engineering Experiment Station Center for Robot-Assisted Search and Rescue (CRASAR) donated an E.M.I.L.Y. lifeguard robot to the Hellenic Coast Guard working on Lesvos Island, Greece. The Coast Guard completed training with E.M.I.L.Y. on Jan 17, 2016. Cutter 618 under Captain Nicholas took her on her first mission that afternoon for the 4:00 pm – 12 pm patrol of the straights.  On January 18, 2016, the program donated the second E.M.I.L.Y. to the Hellenic Red Cross. CRASAR has been assisting the Hellenic Red Cross and ProActiva lifeguarding teams (see http://crasar.org/?p=1980 for details and video).

 

This reflects an evolution in robotics, both in cost and usability, and a new era for CRASAR of donating robots. When CRASAR was first established, small land and marine robots were prohibitively expensive, costing between $35K and $400K and often unreliable with less than 20 hours between failures. Small aerial systems had not been invented. Independently of cost, there was a second barrier: robots required experienced operators with many days, if not weeks, of training, especially on maintenance. Therefore it was important for CRASAR to not only provide robots but provide experts who could effectively work side-by-side with the responders in disaster conditions. It wasn’t feasible to just hand the technology to responders in the Cold Zone and wave bye-bye.  Now robots are coming down in price plus are much easier and intuitive to operate and maintain and are more reliable. Thus with robots such as E.M.I.L.Y., it is possible to train the responders in the field and let them use it directly without having to incur the  distraction, logistics, and liability of embedding an outsider.

 

Our mission is to speed adoption and assist responders, not be responders, and it is gratifying to see the technology reach a point where this is becoming possible. One thing hasn’t changed since 2001 and that is the generosity of our Roboticists Without Borders members. Hydronalix CEO Anthony Mulligan donated 2 E.M.I.L.Y.s, plus travel and expenses for himself and the two operators/trainers. He deserves many kudos for his big heart and commitment!

 

Here are pictures of the EMILY with the crew of Cutter 618:

 

IMG_3128 2IMG_3124

 

Here are pictures with the Hellenic Red Cross team:

 

IMG_0237

EMILY fills “The Gap” between lifeguards in boats and on shore to help 4 boats with 200 refugees arrive safely

On Friday morning Jan 15, 2016,  team member Chief Fernando Boiteux (on vacation from his position as head of lifeguards for LA County Fire Department) deployed EMILY  along the northern shore of Lesvos finding a unique ecological niche for her: in the 100 meter “gap” between the beach and where it is deep enough for lifeguard boats to go.

“The Gap” represents a type of no-mans land for lifeguards. It’s the area that the deeper water patrol boats (such as the Hellenic Coast Guard cutters use in the channel between Turkey and Greece and the smaller rigid hull inflatable boats used by NGOs)  cannot enter due to draft restrictions but is too far out for lifeguards on shore to wade and has to be approached by a swimming lifeguard.  If the boat capsizes, people fall or misjudge the depth and jump off, or the boat runs aground, the lifeguards in patrol boats are not in position to help.  The lifeguards on land have to swim floatation devices out, taking valuable time and risking panicking people trying to climb on their heads.

Another challenge posed by “The Gap” is what happens when multiple boats arrive. Lifeguards on shore have to split their attention and may lose situation awareness of what is going on, especially in behind boats or sides that are blocked from view.

 

EMILY was able to fill the gap on Thursday by being able to work in the shallow water gap and to provide situation awareness with her cameras for the Hellenic Red Cross and PROACTIVA lifeguard teams on land who worked tirelessly as nearly a dozen boats arrived at first light along the rocky shore. Once on shore, other NGOs get the refugees to shelter.

 

This video show EMILY in The Gap and how she gives the lifeguards the ability to keep an eye on multiple boats. Note that 1 EMILY enabled 1 lifeguard to watch multiple boats and maintain general situation awareness.

 

We are still short of actual travel costs, so please consider donating at https://www.gofundme.com/Friends-of-CRASAR

Summary of 47 Known Disaster Robot Deployments 2001-2015

 

This is a 15 minute talk I gave (virtually) at the World Engineering Conference and United Nations World Conference on Disaster Risk and Reduction in Japan. It was a talk for a general audience and while there is nothing new, it does provides

  • an introduction to disaster robotics,
  • how many times and where robots have been used,
  • what they have been used for (and what they could be used for), why they aren’t used more frequently, and
  • my recommendations.

Much of the material is captured in detail in Disaster Robotics, MIT Press.

Why It is Unethical NOT to Use Robots for Disasters

robot revolutionI was a panelist on The Robot Revolution panel held by the Chicago Council on Global Affairs in conjunction with the Chicago Museum of Science and Industry (and the fantastic exhibit Robot Revolution!) You can see the panel at the YouTube clip here

What struck me was the focus on the ethics of robotics: were they safe? what about weaponization? they are going to take away jobs? Elon Musk and Stephen Hawking say they are dangerous, and so on.

I believe that it is unethical not to use existing (and rapidly improving) unmanned systems for emergency preparedness, response, and recovery. The fear of some futuristic version of artificial intelligence is like having discovered penicillin but saying “wait, what if it mutates and eventually leads to penicillin resistant bacteria a 100 years from now” and then starting an unending set of committees and standards processes. Unmanned systems are tools like fire trucks, ambulances, and safety gear.

Here are just three of the reasons why I believe it is unethical not to use unmanned systems for all aspects of emergency management. These came up when I was being interviewed by TIME Magazine as an Agent of Change for Hyundai (but don’t blame them)!

1. Robots speed up disaster response. The logarithmic heuristic developed by Haas, Kates, and Bowden in 1977 posits that reducing the duration of each phase of disaster response reduces the duration of the next phase by a factor of 10. Thus, reducing the initial response phase by just 1 day reduces the overall time through the three reconstruction phases to complete recovery by up to 1,000 days (or 3 years). Think of what that means in terms of lives saved, people not incurring additional health problems, and the resilience of the economy. While social scientists argue with the exact numbers and phases in the Haas, Kates, and Bowden work, it still generally works. If the emergency responders finish up life-saving, rescues, and mitigation, faster  then recovery groups can enter the affected region and restore utilities, repairs roads, etc.

At the 2011 Tokoku tsunami, we used unmanned marine vehicles to reopen a fishing port that was central to the economy of the prefecture in 4 hours. It was going to take manual divers 2 weeks to manually cover the same area- but more than 6 months before divers could be schedule to do the inspection. That would have caused the fishing cooperatives to miss the salmon season- which was the center of their economy.

At the 2014 Oso Mudslides, we used UAVs to provide ESF3 Public Works with geological and hydrological data in 7 hours that would normally take 2-4 days to get from satellites and was higher resolution than satellites and better angles than what could have been gotten from manned helicopters– if the helicopters could have safely flown those missions. The data was critical in determining how to protect the responders from additional slide and flooding, protecting residents and property who had not (yet) been affected,  and mitigating damage to salmon fishing.

2. Land, sea, and aerial robots have already been used since 2001. By my count, unmanned systems have been used in at least 47 disasters in 15 countries. It’s not like these are new technologies. Sure, they are imperfect (I’ve never met a robot that I couldn’t think of something that would make it better), but they are not going to get any better for emergency response until professionals have them and can use them- there has to be a market and then market pressure/competition can drive useful advances!

3. Robots can prevent disasters. Consider this: The US has an aging infrastructure. According to the American Society of Civil Engineers, 1 in 9 bridges are structurally deficient. And transportation departments don’t have cost-effective ways to inspect bridges, especially the underwater portions. Remember the I-35W bridge collapse in Minnesota? Minnesota apparently is one of the top states in keeping up with bridge inspections but only 85% of their bridges are compliant with Federal standards on inspection. That doesn’t bode well for them- or the rest of us. We need unmanned systems that can work cost-effectively, 24/7 and in places inspectors can’t go, or can’t go quickly or safely enough.

 

 

 

 

 

Common UAV Software May Not (Yet) Be Reliable for Building Safety or Damage Assessment

Reconstruction exhibiting all four types of anomalies
Sample reconstruction exhibiting all four types of anomalies

(Note: there is lots of great work going on worldwide and we look forward to working with all companies and researchers to help improve this vital technology)

Researchers at Texas A&M and University of Nebraska-Lincoln have found that popular software packages for creating photo mosaics of disasters from imagery taken by small unmanned aerial systems (UAS) may contain anomalies that prevent its use for reliably determining if a building is safe to enter or estimating the cost of damage. Small unmanned aerial systems can enable responders to collect imagery faster, cheaper, and at higher resolutions than from satellites or manned aircraft. While software packages are continuously improving, users need to be aware that current versions may not produce reliable results for all situations. The report is a step towards understanding the value of small unmanned aerial systems during the time- and resource-critical initial response phase.

“In general, responders and agencies are using a wide variety of general purpose small UAS such as fixed-wings or

quadrotors and then running the images through the software to get high resolution mosaics of the area. But the current state of the software suggests that they may not get always the reliability that they expect or need,” said Dr. Robin Murphy, director of the director of the Texas A&M Engineering Experiment Station Center for Robot-Assisted Search and Rescue and the research supervisor of the study.  “The alternative is to purchase small UAVs explicitly designed for photogrammetric data collection, which means agencies might have to buy a second general purpose UAV to handle the other missions. We’d like to encourage photogrammetric software development to continue to make advances in working with any set of geo-tagged images and being easier to tune and configure.”

In a late breaking report (see SSRR2015 LBR sarmiento duncan murphy ) released at the 13th annual IEEE International Symposium on Safety Security and Rescue Robotics held at Purdue University, researchers presented results showing that two different photogrammetric packages produced an average of 36 anomalies, or errors, per flight.  The researchers identified four types of anomalies impacting damage assessment and structural inspection in general.  Until this study, it does not appear that glitches or anomalies had been systematically characterized or discussed in terms of the impact on decision-making for disasters.  The team of researchers consisted of Traci Sarmiento, a PhD student at Texas A&M, Dr. Brittany Duncan an assistant professor at University of Nebraska- Lincoln who participated while a PhD student at Texas A&M, and Dr. Murphy.

The team applied two packages, Agisoft Photoscan, a standard industrial system, and Microsoft ICE, a popular free software package to the same set of imagery. Both software packages combine hundreds of images into a single high resolution image. They have been used for precision agriculture, pipeline inspection, and amateur photography and are now beginning to be used for structural inspection and disaster damage assessment. The dimensions and distances between objects in the image can be accurately measured within 4cm.  However, the objects themselves may have glitches or anomalies. created through the reconstruction process, making it difficult to tell if the object is seriously damaged.

The researchers collected images using an AirRobot 180, a quadrotor used by the US and Germany military, flying over seven disaster props representing different types of building collapses, a train derailment, and rubble at the Texas A&M Engineering Extension Service’s Disaster City®. The team flew five flights over 6 months. The resulting images for each of the five flights were processed with both packages, then inspected for anomalies using the four categories.