Archive for the ‘News’ Category

Japanese earthquake: how ground, aerial, and marine robots could be used for response

We have been watching with distress the earthquake in Japan and offered any assistance we could provide- however, the first 48 hours are critical for life saving. The International Rescue System Institute (Tokoku University) and the Center for Robot-Assisted Search and Rescue (Texas A&M University)  are the only two centers devoted to disaster robotics, and we work together, so there is considerable expertise available in Japan.

See below for how ground, aerial, and marine robots can be used and best practices are on the home page. Disaster Robotics has 34 case studies worldwide of how these robots have been used at previous earthquakes and disasters through 2013.

I’ll be adding photos and video as I get a chance– this weekend is Aggies Invent: First Responders that we are sponsoring and have two exciting projects based on CRASAR identified (there are 12 others submitted by other response agencies).

Ground robots for locating survivors inside the rubble and speeding up extrication.

Canines typically find survivors but can’t precisely locate where the survivors. Plus dogs can’t provide the “inside view” of the pile of pixie sticks that the extrication team has to be careful not to disturb.  People and canines often can’t get into the rubble because there is often not even a person or dog sized hole that goes all the way from the surface to the interior. Existing boroscopes and cameras on wands can reach about 18 feet or 6 meters into the pile, which means standard US&R equipment is sufficient for single family homes but not apartment buildings or multi-story commercial buildings which are bigger and deeper.

In those case small robots, the size of a lunchbox or smaller, have been used since 2001 (CRASAR at the 9/11 World Trade Center) in go further inside the rubble to where survivors might be and providing the “two for one” of letting the structural specialist visualize how to best remove the rubble to extract. Dr. Tadokoro’s group has one of my favorite small robots, the Active Scope Camera,  that we used together at the Jacksonville Berkman Plaza II collapse. It’s a 6 meter long “caterpillar” robot that can fit in 5cm voids.
Big robots like those used at Fukushima are less valuable because the voids are smaller and the robots can’t move rubble without risking triggering a secondary collapse that will kill the survivors.

UAVs for general reconnaissance and structural inspection.

UAVs have been used since 2005 for disaster response (yes, starting with CRASAR at Hurricane Katrina). The most common uses have been small UAVs for general reconnaissance and for structural inspection. With photogrammetrics, small UAVs are providing geospatial data that are of value to the geologists and public works groups trying to prevent floods, slides, and further collapses. In general, small UAVs are used more frequently because formal responders like the police or fire rescue have access to helicopters and planes. In more remote areas there may be less coverage, so local assets are important. See best practices for UAVs.
One important lesson from the 3/11 earthquake was that the number 1 place to check to see if it was ok and functioning is a hospital!

UMVs for critical underwater infrastructure inspection and reopening ports.

Unmanned marine vehicles, especially ROVs and miniature boats, have been used since 2005 to inspect bridges and reopen ports immediately after an earthquake so that responders can gain access to the affected areas AND get supplies to the hard hit areas. The value of UMVs extends well into the recovery period, both for inspection but also help remap fishing and shipping
channels.

NRP All Things Considered: New Robot System Helps Migrants Cross The Mediterranean Safely

http://tinyurl.com/crasar-NPRgreece

More flooding– recommendations for small UAVs

Flooding continues through the southeast and we are getting some preliminary requests– here’s a quick rundown of  previous blogs:

suggestions from our work at the Texas floods where we flew with Lone Star UASC

a history of use of robots at floods

why the flood of data may be the biggest problem in floods

and some suggestions on flying for floods

plus best practices:

Let’s hope the flooding is not too bad- a bit of the luck of the Irish in time for St. Patrick’s Day.

Robot Assistant Lifeguard: Update

Exciting things continue to happen with EMILY- there’s an improved EMILY, a team of computer science, aerospace, and industrial engineering students are working on smartEMILY, and 37 undergraduates in senior capstone design are working on Computing For Disasters topic! Tony Mulligan, CEO of Hydronalix, creator of EMILY, and Roboticists Without Borders member, is heading back to Greece this weekend to check in with the teams and we look forward to his updates.

Everything is going great– except that 410 refugees  have died so far this year and the resort-based tourism economy of Lesvos has been wrecked. Our thoughts and prayers go out to the refugees, the generous and kind citizens of Lesvos, and to the NGOs who continue to do the best they can.

Improved EMILY

Improved EMILY with camera and lights inside the float cover instead of on top

Improved EMILY with camera and lights inside the float cover instead of on top

EMILY has been improved. Notice that her video and thermal cameras are now mounted flush so that if a large number of refugees need to hang on to her, they won’t try to grab and break the camera.

The Hellenic Coast Guard loves their EMILY so much, she’s on their Wikipedia page! Check out https://en.wikipedia.org/wiki/Hellenic_Coast_Guard

smartEMILY

 

Back here in Texas,  we are continuing the theme of participatory research, engaging graduate and undergraduate students in generating new concepts for lifeguard assistant robots:

smartEMILY.  The students in my CSCE 635 Introduction to AI Robotics class are working on making EMILY easier to use. As I wrote in my 1/12/2016 blog “The refugee crossings present a new scenario- how to handle a large number of people in the water. Some may be in different levels of distress, elderly or children, or unconscious. One solution is to use EMILY to go to the people who are still able to grab on, while the lifeguards swim to aid the people who need special professional attention. Chief John Sims from Rural/Metro Fire Department, Pima, (our 4th team member) is anticipating situations where rescuers can concentrate on saving children and unconscious victims while sending EMILY to the conscious and responsive people.” We’re calling this idea “smartEMILY” and the students from computer science, aerospace engineering, and industrial engineering are designing the artificial intelligence needed for robust operation. I can’t wait to test on EMILY in April.

 

Computing for Disasters

Two of the projects in  undergraduate students in our CSCE 482 Senior Capstone design class on “computing for disasters”  are also related to EMILY and two others are on other aspects of humanitarian work.

Dr. Zoi and her colleague trying out EMILY's two way audio during trials with the Hellenic Coast Guard

Dr. Zoi and her colleague trying out EMILY’s two way audio during trials with the Hellenic Coast Guard

One project was inspired by our meeting with Dr. Zoi Livaditou https://m.facebook.com/zoi.livaditou  who is working with the Hellenic Coast Guard. Dr. Livaditou, a medical doctor, has a cassette tape of directions to play over a megaphone to the refugees in their language—yes, a cassette tape. She was so excited at the idea of using EMILY’s two-way radio to play her taped phrases. Three groups of students (EMILYlingo, Fast Phrase, and Team Dragon) and  are working on a smart phone app that she can get different speakers in different languages to record phrases and then easily call them up.  It should be faster to find the right phrase, easier to add phrases, and far more convenient.

A more futuristic variant that would be perfect for a large flexible display mounted on EMILY (the stuff of my dreams!) is to display what you are trying to tell the refugees to do.  For example, how to tie a cleat hitch so their boat can be towed. Even just to reinforce how to steer the boat right or left, so the person hears and sees what the directions are. Two teams, Team Tanks and Team TBD, are working on this.

A very promising non-robotic project is the Refugee Predictor. A student team is writing an inductive machine learning program to predict the of boats, approximate time of arrival, and location for the next day’s data. They are hoping that there is a pattern in the weather, water, time of sunrise/sunset, and any other relevant data for the past year that explains why some days there are 20 boats hitting Skala, and other days 8 boats going to Mytelini. What a great use of machine learning!

The other Computing for Disasters project is there to help with data management by us and other NGOs. In particular, if EMILY is on the water for a morning, the “action” may only be a few minutes.  In order to generate a report, someone has to edit the video clip. The students on Team Snips are working to create a website where any of the NGOs can upload a file plus one or more timestamps, and then it will cut out a snippet of a specified length.

We are seeking funding to buy our own EMILY and Fotokite, then return to Greece to continue to learn and to partner with Prof. Milt Statheropoulos’ group at the National Technical University of Athens.

I am still hoping to raise another $2,504 to cover the unpaid expenses from the January trip so please donate at https://www.gofundme.com/Friends-of-CRASAR

 

 

 

The Taiwan Earthquake: robots

We are just getting word of several building collapses in the Taiwan earthquake, here are some thoughts and data on how robots have been used in previous collapses…

Ground robots may be of the most value. In a situation like this where the building has collapsed,  small robots will likely to get into voids and go deeper than the 18-20 feet that a camera on a probe or a boroscope can go into. Note that canines would normally be used first to indicate that people are alive (if there was any doubt about occupancy). The ground robots would be used to try to localize the survivors AND allow the rescue team to at the same time understand the internal layout of the structure. If they can better understand the internal layout of the “pixie sticks” of the rubble, they can extricate the victims faster and with less chance of triggering a secondary collapse. Most of the ground robots used, such as the Inuktun series which have been used the most, have 2 way audio so the responders can talk to the victims.

With our colleague Eric Rasmussen, MD FACP, we’ve experimented with how a small robot can carry tubing allowing a survivor to have water. With members of Texas Task Force 1 medical team, we’ve experimented with how doctors can use the robot to communicate with the survivor, assess their injuries, and engage the survivors- as it may take 4-10 hours to be extracted.

Similar situations where ground robots have been used for multi-story commercial building collapses are:

  • 2001 9/11 World Trade Center collapse
  • 2010 Prospect Towers collapse
  • 2011 Christchurch earthquake

Ground robots are often not used in earthquakes, such as the Japanese earthquake, because of building resilience and codes. Residential homes are small, often wood, and fairly easy to locate victims with canine teams and then extracting. Adding a robot doesn’t really speed up anything.

UAVs can give an overview of a collapse, but generally it has been the “inside” view that responders need the most and can’t get any other way.

 

 

CRASAR donates 2 E.M.I.L.Y.s to Hellenic Coast Guard and Red Cross

On January 14, 2016, the Roboticists Without Borders program hosted by the Texas A&M Engineering Experiment Station Center for Robot-Assisted Search and Rescue (CRASAR) donated an E.M.I.L.Y. lifeguard robot to the Hellenic Coast Guard working on Lesvos Island, Greece. The Coast Guard completed training with E.M.I.L.Y. on Jan 17, 2016. Cutter 618 under Captain Nicholas took her on her first mission that afternoon for the 4:00 pm – 12 pm patrol of the straights.  On January 18, 2016, the program donated the second E.M.I.L.Y. to the Hellenic Red Cross. CRASAR has been assisting the Hellenic Red Cross and ProActiva lifeguarding teams (see http://crasar.org/?p=1980 for details and video).

 

This reflects an evolution in robotics, both in cost and usability, and a new era for CRASAR of donating robots. When CRASAR was first established, small land and marine robots were prohibitively expensive, costing between $35K and $400K and often unreliable with less than 20 hours between failures. Small aerial systems had not been invented. Independently of cost, there was a second barrier: robots required experienced operators with many days, if not weeks, of training, especially on maintenance. Therefore it was important for CRASAR to not only provide robots but provide experts who could effectively work side-by-side with the responders in disaster conditions. It wasn’t feasible to just hand the technology to responders in the Cold Zone and wave bye-bye.  Now robots are coming down in price plus are much easier and intuitive to operate and maintain and are more reliable. Thus with robots such as E.M.I.L.Y., it is possible to train the responders in the field and let them use it directly without having to incur the  distraction, logistics, and liability of embedding an outsider.

 

Our mission is to speed adoption and assist responders, not be responders, and it is gratifying to see the technology reach a point where this is becoming possible. One thing hasn’t changed since 2001 and that is the generosity of our Roboticists Without Borders members. Hydronalix CEO Anthony Mulligan donated 2 E.M.I.L.Y.s, plus travel and expenses for himself and the two operators/trainers. He deserves many kudos for his big heart and commitment!

 

Here are pictures of the EMILY with the crew of Cutter 618:

 

IMG_3128 2IMG_3124

 

Here are pictures with the Hellenic Red Cross team:

 

IMG_0237

Summary of 47 Known Disaster Robot Deployments 2001-2015

 

This is a 15 minute talk I gave (virtually) at the World Engineering Conference and United Nations World Conference on Disaster Risk and Reduction in Japan. It was a talk for a general audience and while there is nothing new, it does provides

  • an introduction to disaster robotics,
  • how many times and where robots have been used,
  • what they have been used for (and what they could be used for), why they aren’t used more frequently, and
  • my recommendations.

Much of the material is captured in detail in Disaster Robotics, MIT Press.

Robots for earthquakes- history of use of ground, aerial, and marine systems plus best practices

Our hearts go out to the victims, families, and responders in Afghanistan and Pakistan. Here are links to

And from our home page, here are helpful 1 page guides and best practices for small unmanned aerial systems that have been incorporated into United Nations humanitarian standards and are continuing to evolve:

Texas Floods: How small UAVs have been used at 9 floods in 6 countries. Don’t forget about UMVs!

Note: this is a long blog with sections on best practices, where SUAS have been used (and for what missions), the flood of data that interferes with making the most of UAS data and how computer vision can help, and unmanned marine vehicles.

CRASAR is standing by to assist with the flooding in Texas with small unmanned aerial systems (UAS/UAV) and unmanned marine vehicles. Johnny Cash’s song “How High The Water Momma” comes to mind. We’ve been working with floods since 2005 and in July offered a class on flying for floods.

The rain is still too heavy to fly in most affected parts. Coitt Kessler, Austin Fire Department, is coordinating the use of small UAS with the State Operations Center and has been working tirelessly since Thursday. CRASAR is offering the Texas A&M team and the UAVRG team at no cost through the Roboticists Without Borders program. We also hope to try out an app of coordinating small UAS from the newest member of Roboticists Without Borders, Akum.

Hey- If you want to volunteer to fly, please do not fly with out explicitly coordinating with your local fire department and confirming that they in turn have followed standard procedures and coordinated with the state air operations (this is a standard ICS practice and should only take them a few minutes), otherwise there may be a repeat of the dangerous situation where  a) low flying helicopters and SUAS are working too close to each other and b) the data collected was either the wrong data or never made it to a decision maker.  Dangerous situations happened at the Boulder floods and several times in the Texas Memorial Day floods- it shuts down the helicopter operations.  And remember, it hards to become the fire rescue equivalent of a deputy without have met and worked with the fire rescue department- so it may not be realistic to expect to help with this disaster.

Best Practices

Here are links to our best practices for picking UAVs and payloads for disasters:

Where Small UAVs Have Been Used

Small UAVs or UAS have been used at least 9 disasters from flooding or had flooding associated with it: Hurricane Katrina 2005 (the first ever use of a small UAS for a disaster, which was by CRASAR), Typhoon Morakot, Taiwan 2009, Thailand Floods 2011, Typhoon Haiyan Philippines 2013, Boulder Colorado floods 2013, Oso Washington Mudslides 2014, Balkans flooding Serbia 2014, Cyclone Pamela Vanuatu 2014, and the Texas Memorial Day Floods 2015.  CRASAR participated in 3 of the 9 events.

SUAS missions at these floods have been:

  • situation awareness of the flood, affected transportation, and person in distress
  • hydrological assessment- where’s the flooding, state of levees, etc.? Texas has levees that impact people (think New Orleans and Katrina) but also livestock. Another use of small UAS is to determine why the floods are flooding where they are. In the Balkans flooding, the ICARUS team used their UAS and found a illegal dike that was preventing public works engineers from draining the area.
  • searching for missing persons presumably swept away- that was the major use of small UAS at the Texas Memorial Day floods
  • deliver a small line to persons in distress so that they can pull up a heavier line for help- this was also done at the Texas Memorial Day floods
  • debris estimation in order to speed recovery

SUAS proposed, but never flown to the best of my knowledge at an actual disaster (remember a patch to anyone who can help me keep the list of deployments up to date!), missions have been:

  • home owner and business insurance claims- many insurance carriers are actively exploring this and this was a big topic with at our 2015 Summer Institute on Flooding
  • carry wireless repeaters—this was actually done with manned aircraft from the Civil Air Patrol during the Memorial Day floods. The greater persistence and distance may keep that in the CAP list of responsibilities

The Flood of Data and the Promise of Computer Vision

The biggest challenge in using UAS is not flying (or regulations) but rather the flood of data. As I noted in my blog on our Summer Institute on flooding, one of our 20-minute UAS flights for the Texas Memorial Day floods produced roughly over 800 images totaling 1.7GB. There were over a dozen platforms flying daily for two weeks during the floods as well as Civil Air Patrol and satellite imagery. Most of the imagery was being used to search for missing persons, which means each image has to be inspected manually by at least (preferably more). Signs of missing persons are hard to see, as there may be only a few pixels of clothing (victims may be covered in mud or obscured by vegetation and debris) or urban debris (as in, if you see parts of a house, there may be the occupant of the house somewhere in the image). Given the multiple agencies and tools, it was hard to pinpoint what data has been collected when (i.e., spatial and temporal complexity) and then access the data by area or time. Essentially no one knew what they had.  Agencies and insurance companies had to manually sort through news feeds and public postings, both text and images, to find nuggets of relevant information.

Students from our NSF Research Experience for Undergraduates on Computing for Disasters and our partners at the University of Maryland and Berkeley led by Prof. Larry Davis created computer vision and machine learning apps during the Texas floods. The apps searched the imagery for signs of missing persons, including debris that might have been washed away with them and piles of debris large enough to contain a victim. The students also created visualization packages to show where the UAS and other assets had been and what data they had collected.

Don’t Forget About Unmanned Marine Vehicles

As I described in a previous blog on Hurricane Patricia, unmanned marine vehicles have been used for hurricane storm surges but not for flooding. They would be of great benefit for inspecting underwater portions of critical infrastructure such as bridges and pipelines. There’s even EMILY a robot super floatation device that can zoom out to where people are trapped.

 

 

Four Surprises about the Use of Unmanned Ground, Aerial, and Marine Vehicles for Hurricanes, Typhoons, and Cyclones

Hurricanes, typhoons, and cyclones form a category of meteorological events referred to as cyclonic activity.  They damage large areas and destroy the transportation infrastructure, interfering with the ability of agencies to find and assist people in distress, restore power, water, and communications, and prevent the delivery of supplies. As I describe in my TED talk, it can take years for a community to recover- the rule of thumb developed by disaster experts Haas, Kates, and Bowden in 1982 is that reducing the duration of each phase of disaster response reduces the duration of the next phase by a factor of 10. Thus, reducing the initial response phase by just 1 day reduces the overall time through the three reconstruction phases to complete recovery by up to 1,000 days. The sooner emergency response agencies can use unmanned systems, the faster they can respond and we can recover from a disaster.

There are three modes or types of small unmanned vehicles or robots: ground, aerial, and marine systems. Small vehicles have the advantage that they are easy to carry in an SUV or a backpack and deploy on demand when the field teams need them, which the military would call a tactical assets. Larger unmanned systems such as the National Guard flying a Predator to help get situation awareness of several counties or provinces requires much more coordination and planning (and expense); these are strategic assets.

Here are four surprises about small unmanned vehicles for cyclonic events (I’ll be adding links to videos through out the day):

1. Small unmanned ground, aerial, and marine systems have been reported at 7 hurricanes since the first use at Hurricane Charley in 2004.

These events are Hurricane Charley (USA, 2004), Hurricane Katrina (USA, 2005), Hurricane Wilma (USA, 2005), Hurricane Ike (USA, 2008), Typhoon Morakot (Taiwan, 2009), Typhoon Haiyan (Philippines, 2013), and Cylone Pam (Vanuatu, 2015).

2. Ground robots are generally not useful.

Ground robots have only be used at 2 of the 7 events: Charley and Katrina. Cyclonic activity tends to damage or destroy swaths of single story family dwellings, not multi-story commercial buildings. If houses are flattened, the debris is not more than 20 feet deep, so traditional techniques work. If houses or apartments are damaged but standing and there is a concern that people are hurt inside, canines can determine in seconds if a person is inside. A door or window would have to be breached to insert a robot (or a person), which means the apartment would then be open to robbers. We learned that while helping Florida Task Force 3 search the retirement communities in Florida affected by Hurricane Charley in 2004. Florida Task Force 3 did use a robot to enter two apartment buildings that were too dangerously damage to enter during Hurricane Katrina, but they didn’t have a canine team which is now generally considered the preferred method.

3. Marine vehicles may be the most useful kind of robot for both response and recovery.

AEOS-1 with accoustic imager inspecting underwater portion of bridge

AEOS-1 with accoustic imager inspecting underwater portion of bridge

Marine vehicles have been used for only 2 of the events, Hurricane Wilma and Hurricane Ike, but could have been effective for all 7. Hurricanes and Typhoons are a double whammy for marine infrastructure- the underwater portion of bridges, seawalls, pipelines, power grid, and ports. First the event creates storm surges along the coast, then flooding occurs inland and hits the coast again.  Bridges and ports can appear to be safe but the surge and flooding can have scoured the ground from under the pilings, leaving them resting on nothing. Debris can have broken off a piling underwater, creating a hanging pile. This means that transportation routes can be cut off during the response, hampering the movement of responders but also hampering bringing in enough food and supplies to feed a country, such as at the Haiti earthquake, which is normally done with ships.  The economy can’t recover until the infrastructure is back in place.

Checking for these conditions is typically done with manual divers but the conditions are dangerous- the current is still high, the water is cloudy and debris is floating everywhere, and divers often have to resort to feeling for damage. There are few divers and it can take months to schedule them, as we saw at the Tohoku tsunami. Marine vehicles, both underwater and on the surface, can be outfitted with acoustic imagers that act as a combination of ultrasound and a camera to check for these conditions. In Japan, we re-opened a port in 4 hours versus weeks by a dive team, and dive teams would not be able to start work for six months after the disaster. The six month delay would have caused the city to miss the salmon fishing season, which is the big economic driver for the region.

See UMV and UAV at Hurricane Wilma here.

4. Small unmanned aerial systems have been used the most frequently of the three types of robots.

SUAS have been used in all but two of the 7 events, Hurricane Charley and Hurricane Ike. Small UAS were still experimental in 2004 when Hurricane Charley occurred but the next day after our experiences as part of Florida Task Force 3, I called Mike Tamilow at FEMA and offer to make introductions to facilitate use for the next hurricane. Unfortunately it wasn’t until next year and several hurricanes later that SUAS were used for Katrina by us and other teams from the Department of Defense. Despite the success of these deployments, SUAS didn’t really take off (pun intended) until 2011 when the technology had matured and come down in price.