Category: News

The Legacy of 9/11 for Disaster Robotics

[youtube]https://youtu.be/1DYi7_lE6cA[/youtube]

I’ll be speaking at the Smithsonian Museum of American History today as part of the 15th Anniversary of 9/11 event. 9/11 was the first reported use of robots for search and rescue and created a legacy that continues to grow for both disaster response and for science and technology. The robots were successful by any standard for rating search and rescue tools- improved performance over existing tools, frequency of use, and acceptance by professionals. They didn’t find any survivors, but neither did anyone else as there were sadly no survivors to find.
I will never forget my time at the World Trade Center as a responder, a scientist, or as a person. The infinite sadness of such an event still haunts me. The Seamus Heaney poem I read in the NYT and quoted in my IAAI talk:
And we all knew one thing by being there.
The space we stood around had been emptied
Into us to keep, it penetrated
Clearances that suddenly stood open.
High cries were felled and a pure change happened.
I believe the many members of the CRASAR team at the World Trade Center and since have kept the memories and have enabled a pure change- as witnessed by the use of robots at at least 50 disasters worldwide. While the robots are a very small story among the amazing stories of loss and triumph, I am proud to tell the story and add to the history of how 9/11  has made waves and ripples in history.

Legacy for Disaster Robotics

9/11 was an existence proof that small robots could be of significant use searching in rubble, reaching places that people and dogs could not, and penetrating two to three times farther than cameras on poles, which were the nearest similar tool. Large heavy robots had been developed for bomb squads but they were too big and heavy to be used in rubble, as seen at the Oklahoma City bombing. Red Whitaker at CMU had built even larger and heavier robots for Chernobyl and Three Mile Island nuclear accidents for the recovery operations, not for immediate search and rescue.

 

Small robots, ranging in size from a shoe box to a carry-on suitcase, that could be carried in one or two backpacks, had been under development by the pipeline and sewer inspection industry and the DARPA Tactical Mobile Robots program directed by John Blitch, the founding director of CRASAR.  If you look at the DARPA TMR logo you see that there is the “urban terrain” of cities but also a rubble pile, because John was thinking of dual use. He had been at the Oklahoma City bombing and had changed his MS thesis topics to robots for disasters (I was his co-advisor).

 

The robots were used starting shortly after midnight on 9/12 through 9/21 for search and rescue by FDNY, INTF1, OHTF1, PATF1,  and VATF1 and then again from 9/23 through 10/2 for recovery operations (structural inspection of the slurry wall) by the NY Department of Design and Construction when the last robot on site wore out.

 

By my count, robots have been used in 49 disasters since then in 17 countries. 24 of those disasters used UGVs- with the majority using the robot models from 9/11: Inuktuns (ex. mine disasters, building collapses), Packbots (ex. Christchurch for searching the cathedral, Fukushima Daiichi), and Talons (ex. Fukushima Daiichi). See Disaster Robotics for more details.

Legacy for Robotics

9/11 created a legacy for robotics in two ways. Search and rescue is often cited as a motivation for new advances in robotics; if you’re a doctor, you often say you want to solve cancer, if you’re a roboticist you often say you want to help with search and rescue.
One is that it created a new subfield of robotics. The IEEE Robotics and Automation Society, the largest and most prestigious professional organization devoted to robotics, has a technical committee on Safety Security and Rescue Robotics with an annual international symposium that started in 2002 (I was a co-founder of the TC and symposium). Both the European Union and Japan are investing heavily in small disaster robots- including sensors and user interfaces- with multiple projects being funded at the $20M to $35M range (the US doesn’t have dedicated programs for funding robotics projects at that level). The SSRR field now includes UAVs, UMVs, and many innovations in ground robots that can crawl and burrow into rubble.

 

9/11 also uncovered technical challenges that the R&D community is still struggling with. Probably the most significant discovery was that remote presence, or teleoperation, is actually the preferred mode of control for almost every response task- even with UAVs and UMVs.  Because the time pressure is so great and because disasters always have a surprise, the responders want to see in real-time what the robot is seeing and being able to opportunistically change up the plan (“wait— what’s that? Let’s look over there..”). Up until 9/11, researchers and developers had assumed that all robots should be taskable agents- you would tell it what to do, it would go off and do it, and then come back- and remote presence was just because we hadn’t created autonomous programs. Now there is the realization that many applications, not just search and rescue, require the human and robot to work together in a joint cognitive system to get the job done.

 

The second most significant discovery is what Jenny Burke would later describe in her PhD thesis as that 2 heads are 9 times better than 1. Up until 9/11, researchers and developers had assumed that 1 person could operate a robot successfully and thus the real challenge was for 1 person to drive 2 or more robots. We had had signs prior to 9/11 that 1 person couldn’t drive a robot in rubble and look at the same time-as one of my grad students who later went with us to 9/11, Jenn Casper, documented, they could do it but they could literally roll past a victim in front of the robot (we started seeing this in exercises with FLTF3 in buildings that were being demolished). The cognitive challenges of thinking like a ferret or meerkat (the size of the robot) were bigger than anyone had expected and then rubble is deconstructed and hard to mentally sort out. 2 heads makes sense in a way- if you are in a new town driving around in traffic and looking for a particular address you’ve never been to, it helps to have a passenger in the car who is looking too.

 

Italian Earthquake: Recommendations for using ground and aerial robots for immediate lifesaving

Our thoughts and prayers go out to the Italian people impacted by the earthquake. We’ve reached out to colleagues in Italy in case any of us here can be of assistance. Below is a general overview of what might be useful and why.

From the scanty news reports in the US, my guess is that this event will favor the use of small tethered ground robots for locating survivors in rubble based on the case studies from 9/11 World Trade Center, Cologne Archives collapse, Berkman Plaza collapse Prospect Towers collapse, L’aquila earthquake, Mirandola earthquake, and multiple mine disasters worldwide (see Disaster Robotics, MIT Press, 2014 for those case studies). UAVs may be of value in estimating extent, ascertaining whether roads are open or can be easily cleared to allow responders rapid access, and general damage assessment and recovery operations (as per Nepal and Chile), but probably not for direct life saving- though I could be wrong.

“Small”  as in pipe inspection robots- not a bomb squad robot like the Packbots used at Fukushima- because if a person or dog could get into a void to reach a trapped person, they probably would despite the personal risk. A tether is useful because it solves wireless and power problems- but more importantly any entry would likely be from the top of the structure or the upper parts, so the robot has to rappel down.

A video camera, color, is essential. Thermal cameras may be of use initially but are very hard to use for navigation in confined spaces. So I wouldn’t recommend thermal by itself, rather as a second camera. The value of a thermal camera goes away after a few days because decomposing bodies present a heat signature. Navigation gets harder as small protrusion become the same temperature as the surroundings.

A robot with 2-way audio will be valuable because the operator can call out and listen for sounds of survivors, then medical experts can talk with the victims. But even just a speaker or a microphone by itself can be useful.

Should someone find a survivor, a small tube can be attached to the robot to provide water to a trapped victim- hook up the end to an aquarium or koi pond pump. (This is a great solution worked out by Eric Rasmussen and we tested with the USMC CBIRF unite.) The robot can probably maneuver and bring a small payloads- a radio, a space blanket, power bars (assuming they aren’t severely injured). LACoFD does so many confined space rescues, they can have a kit the size of a Pringles can for trying to give to people trapped in caves and culverts.

Japanese earthquake: how ground, aerial, and marine robots could be used for response

We have been watching with distress the earthquake in Japan and offered any assistance we could provide- however, the first 48 hours are critical for life saving. The International Rescue System Institute (Tokoku University) and the Center for Robot-Assisted Search and Rescue (Texas A&M University)  are the only two centers devoted to disaster robotics, and we work together, so there is considerable expertise available in Japan.

See below for how ground, aerial, and marine robots can be used and best practices are on the home page. Disaster Robotics has 34 case studies worldwide of how these robots have been used at previous earthquakes and disasters through 2013.

I’ll be adding photos and video as I get a chance– this weekend is Aggies Invent: First Responders that we are sponsoring and have two exciting projects based on CRASAR identified (there are 12 others submitted by other response agencies).

Ground robots for locating survivors inside the rubble and speeding up extrication.

Canines typically find survivors but can’t precisely locate where the survivors. Plus dogs can’t provide the “inside view” of the pile of pixie sticks that the extrication team has to be careful not to disturb.  People and canines often can’t get into the rubble because there is often not even a person or dog sized hole that goes all the way from the surface to the interior. Existing boroscopes and cameras on wands can reach about 18 feet or 6 meters into the pile, which means standard US&R equipment is sufficient for single family homes but not apartment buildings or multi-story commercial buildings which are bigger and deeper.

In those case small robots, the size of a lunchbox or smaller, have been used since 2001 (CRASAR at the 9/11 World Trade Center) in go further inside the rubble to where survivors might be and providing the “two for one” of letting the structural specialist visualize how to best remove the rubble to extract. Dr. Tadokoro’s group has one of my favorite small robots, the Active Scope Camera,  that we used together at the Jacksonville Berkman Plaza II collapse. It’s a 6 meter long “caterpillar” robot that can fit in 5cm voids.
Big robots like those used at Fukushima are less valuable because the voids are smaller and the robots can’t move rubble without risking triggering a secondary collapse that will kill the survivors.

UAVs for general reconnaissance and structural inspection.

UAVs have been used since 2005 for disaster response (yes, starting with CRASAR at Hurricane Katrina). The most common uses have been small UAVs for general reconnaissance and for structural inspection. With photogrammetrics, small UAVs are providing geospatial data that are of value to the geologists and public works groups trying to prevent floods, slides, and further collapses. In general, small UAVs are used more frequently because formal responders like the police or fire rescue have access to helicopters and planes. In more remote areas there may be less coverage, so local assets are important. See best practices for UAVs.
One important lesson from the 3/11 earthquake was that the number 1 place to check to see if it was ok and functioning is a hospital!

UMVs for critical underwater infrastructure inspection and reopening ports.

Unmanned marine vehicles, especially ROVs and miniature boats, have been used since 2005 to inspect bridges and reopen ports immediately after an earthquake so that responders can gain access to the affected areas AND get supplies to the hard hit areas. The value of UMVs extends well into the recovery period, both for inspection but also help remap fishing and shipping
channels.

More flooding– recommendations for small UAVs

Flooding continues through the southeast and we are getting some preliminary requests– here’s a quick rundown of  previous blogs:

suggestions from our work at the Texas floods where we flew with Lone Star UASC

a history of use of robots at floods

why the flood of data may be the biggest problem in floods

and some suggestions on flying for floods

plus best practices:

Let’s hope the flooding is not too bad- a bit of the luck of the Irish in time for St. Patrick’s Day.

Robot Assistant Lifeguard: Update

Exciting things continue to happen with EMILY- there’s an improved EMILY, a team of computer science, aerospace, and industrial engineering students are working on smartEMILY, and 37 undergraduates in senior capstone design are working on Computing For Disasters topic! Tony Mulligan, CEO of Hydronalix, creator of EMILY, and Roboticists Without Borders member, is heading back to Greece this weekend to check in with the teams and we look forward to his updates.

Everything is going great– except that 410 refugees  have died so far this year and the resort-based tourism economy of Lesvos has been wrecked. Our thoughts and prayers go out to the refugees, the generous and kind citizens of Lesvos, and to the NGOs who continue to do the best they can.

Improved EMILY

Improved EMILY with camera and lights inside the float cover instead of on top
Improved EMILY with camera and lights inside the float cover instead of on top

EMILY has been improved. Notice that her video and thermal cameras are now mounted flush so that if a large number of refugees need to hang on to her, they won’t try to grab and break the camera.

The Hellenic Coast Guard loves their EMILY so much, she’s on their Wikipedia page! Check out https://en.wikipedia.org/wiki/Hellenic_Coast_Guard

smartEMILY

 

Back here in Texas,  we are continuing the theme of participatory research, engaging graduate and undergraduate students in generating new concepts for lifeguard assistant robots:

smartEMILY.  The students in my CSCE 635 Introduction to AI Robotics class are working on making EMILY easier to use. As I wrote in my 1/12/2016 blog “The refugee crossings present a new scenario- how to handle a large number of people in the water. Some may be in different levels of distress, elderly or children, or unconscious. One solution is to use EMILY to go to the people who are still able to grab on, while the lifeguards swim to aid the people who need special professional attention. Chief John Sims from Rural/Metro Fire Department, Pima, (our 4th team member) is anticipating situations where rescuers can concentrate on saving children and unconscious victims while sending EMILY to the conscious and responsive people.” We’re calling this idea “smartEMILY” and the students from computer science, aerospace engineering, and industrial engineering are designing the artificial intelligence needed for robust operation. I can’t wait to test on EMILY in April.

 

Computing for Disasters

Two of the projects in  undergraduate students in our CSCE 482 Senior Capstone design class on “computing for disasters”  are also related to EMILY and two others are on other aspects of humanitarian work.

Dr. Zoi and her colleague trying out EMILY's two way audio during trials with the Hellenic Coast Guard
Dr. Zoi and her colleague trying out EMILY’s two way audio during trials with the Hellenic Coast Guard

One project was inspired by our meeting with Dr. Zoi Livaditou https://m.facebook.com/zoi.livaditou  who is working with the Hellenic Coast Guard. Dr. Livaditou, a medical doctor, has a cassette tape of directions to play over a megaphone to the refugees in their language—yes, a cassette tape. She was so excited at the idea of using EMILY’s two-way radio to play her taped phrases. Three groups of students (EMILYlingo, Fast Phrase, and Team Dragon) and  are working on a smart phone app that she can get different speakers in different languages to record phrases and then easily call them up.  It should be faster to find the right phrase, easier to add phrases, and far more convenient.

A more futuristic variant that would be perfect for a large flexible display mounted on EMILY (the stuff of my dreams!) is to display what you are trying to tell the refugees to do.  For example, how to tie a cleat hitch so their boat can be towed. Even just to reinforce how to steer the boat right or left, so the person hears and sees what the directions are. Two teams, Team Tanks and Team TBD, are working on this.

A very promising non-robotic project is the Refugee Predictor. A student team is writing an inductive machine learning program to predict the of boats, approximate time of arrival, and location for the next day’s data. They are hoping that there is a pattern in the weather, water, time of sunrise/sunset, and any other relevant data for the past year that explains why some days there are 20 boats hitting Skala, and other days 8 boats going to Mytelini. What a great use of machine learning!

The other Computing for Disasters project is there to help with data management by us and other NGOs. In particular, if EMILY is on the water for a morning, the “action” may only be a few minutes.  In order to generate a report, someone has to edit the video clip. The students on Team Snips are working to create a website where any of the NGOs can upload a file plus one or more timestamps, and then it will cut out a snippet of a specified length.

We are seeking funding to buy our own EMILY and Fotokite, then return to Greece to continue to learn and to partner with Prof. Milt Statheropoulos’ group at the National Technical University of Athens.

I am still hoping to raise another $2,504 to cover the unpaid expenses from the January trip so please donate at https://www.gofundme.com/Friends-of-CRASAR

 

 

 

The Taiwan Earthquake: robots

We are just getting word of several building collapses in the Taiwan earthquake, here are some thoughts and data on how robots have been used in previous collapses…

Ground robots may be of the most value. In a situation like this where the building has collapsed,  small robots will likely to get into voids and go deeper than the 18-20 feet that a camera on a probe or a boroscope can go into. Note that canines would normally be used first to indicate that people are alive (if there was any doubt about occupancy). The ground robots would be used to try to localize the survivors AND allow the rescue team to at the same time understand the internal layout of the structure. If they can better understand the internal layout of the “pixie sticks” of the rubble, they can extricate the victims faster and with less chance of triggering a secondary collapse. Most of the ground robots used, such as the Inuktun series which have been used the most, have 2 way audio so the responders can talk to the victims.

With our colleague Eric Rasmussen, MD FACP, we’ve experimented with how a small robot can carry tubing allowing a survivor to have water. With members of Texas Task Force 1 medical team, we’ve experimented with how doctors can use the robot to communicate with the survivor, assess their injuries, and engage the survivors- as it may take 4-10 hours to be extracted.

Similar situations where ground robots have been used for multi-story commercial building collapses are:

  • 2001 9/11 World Trade Center collapse
  • 2010 Prospect Towers collapse
  • 2011 Christchurch earthquake

Ground robots are often not used in earthquakes, such as the Japanese earthquake, because of building resilience and codes. Residential homes are small, often wood, and fairly easy to locate victims with canine teams and then extracting. Adding a robot doesn’t really speed up anything.

UAVs can give an overview of a collapse, but generally it has been the “inside” view that responders need the most and can’t get any other way.

 

 

CRASAR donates 2 E.M.I.L.Y.s to Hellenic Coast Guard and Red Cross

On January 14, 2016, the Roboticists Without Borders program hosted by the Texas A&M Engineering Experiment Station Center for Robot-Assisted Search and Rescue (CRASAR) donated an E.M.I.L.Y. lifeguard robot to the Hellenic Coast Guard working on Lesvos Island, Greece. The Coast Guard completed training with E.M.I.L.Y. on Jan 17, 2016. Cutter 618 under Captain Nicholas took her on her first mission that afternoon for the 4:00 pm – 12 pm patrol of the straights.  On January 18, 2016, the program donated the second E.M.I.L.Y. to the Hellenic Red Cross. CRASAR has been assisting the Hellenic Red Cross and ProActiva lifeguarding teams (see http://crasar.org/?p=1980 for details and video).

 

This reflects an evolution in robotics, both in cost and usability, and a new era for CRASAR of donating robots. When CRASAR was first established, small land and marine robots were prohibitively expensive, costing between $35K and $400K and often unreliable with less than 20 hours between failures. Small aerial systems had not been invented. Independently of cost, there was a second barrier: robots required experienced operators with many days, if not weeks, of training, especially on maintenance. Therefore it was important for CRASAR to not only provide robots but provide experts who could effectively work side-by-side with the responders in disaster conditions. It wasn’t feasible to just hand the technology to responders in the Cold Zone and wave bye-bye.  Now robots are coming down in price plus are much easier and intuitive to operate and maintain and are more reliable. Thus with robots such as E.M.I.L.Y., it is possible to train the responders in the field and let them use it directly without having to incur the  distraction, logistics, and liability of embedding an outsider.

 

Our mission is to speed adoption and assist responders, not be responders, and it is gratifying to see the technology reach a point where this is becoming possible. One thing hasn’t changed since 2001 and that is the generosity of our Roboticists Without Borders members. Hydronalix CEO Anthony Mulligan donated 2 E.M.I.L.Y.s, plus travel and expenses for himself and the two operators/trainers. He deserves many kudos for his big heart and commitment!

 

Here are pictures of the EMILY with the crew of Cutter 618:

 

IMG_3128 2IMG_3124

 

Here are pictures with the Hellenic Red Cross team:

 

IMG_0237

Summary of 47 Known Disaster Robot Deployments 2001-2015

 

This is a 15 minute talk I gave (virtually) at the World Engineering Conference and United Nations World Conference on Disaster Risk and Reduction in Japan. It was a talk for a general audience and while there is nothing new, it does provides

  • an introduction to disaster robotics,
  • how many times and where robots have been used,
  • what they have been used for (and what they could be used for), why they aren’t used more frequently, and
  • my recommendations.

Much of the material is captured in detail in Disaster Robotics, MIT Press.

Robots for earthquakes- history of use of ground, aerial, and marine systems plus best practices

Our hearts go out to the victims, families, and responders in Afghanistan and Pakistan. Here are links to

And from our home page, here are helpful 1 page guides and best practices for small unmanned aerial systems that have been incorporated into United Nations humanitarian standards and are continuing to evolve: