Category: Director’s Blog

Flood of data may be the biggest problem in dealing with floods

We just concluded a three day Summer Institute on Floods (July 26-28, 2015), which was hosted by our “mother” center, the Center for Emergency Informatics. The Summer Institute focuses on the data-to-decision problems for a particular type of disaster. This year’s Summer Institute was the second in two-part series on floods- even though there was a drought last year, our TEEX partners were very worried about flooding for Texas and flooding is the number one disaster in the world. It was originally scheduled for early June but had to be moved due to the floods and allowed us the opportunity to discuss topics while they were still fresh on everyone’s minds.

 

The Summer Institute brought together representatives from 12 agencies, 15 universities, and 5 companies for two days of “this is what we did during the Texas floods” and one day of “this is what we could do or do better” experimentation with small unmanned aerial vehicles, crowd sourcing, computer vision, map-based visualization packages, and mobile phone apps for common operating pictures and data collection. The field exercise was designed to track the resource allocation of field teams, how they can acquire data from the field from UAVs and smartphones, then how the data can be effectively organized, analyzed, and visualized.

Here are my preliminary findings:

  • Agencies are adopting small UAVs, social media, and smartphones apps to acquire new types of data, faster and over larger geospatial and temporal scales than ever. Small UAVs were used during the Texas floods in all of the affected regions (big shout out to Lone Star UAS Center- we flew under their direction during the floods). Agencies were monitoring social media, in one case to uncover and combat a rumor that a levee had broken. Texas Task Force 1 and other agencies used General Dynamics’ GeoSuite mobile app for giving tactical responders a common operating picture (big shout out- GeoSuite was introduced and refined starting with the 2011 Summer Institute).
  •  Wireless communications is no longer the most visible barrier to acquiring data from the field in order to make better decisions.  While responders in the field may not have as much bandwidth as they did before a disaster, cell towers on wheels are being rapidly deployed and the Civil Air Patrol flew repeater  nodes. That said, wireless communications isn’t solved as hilly geography can prevent teams from connecting to sparse temporary nodes. Plus keep in mind that large parts of rural Texas have limited or no connectivity under normal conditions.
  • The new barrier is what to do with the data coming in from unmanned systems, news feeds, social media feeds, and smart phones. Consider that a single 20-minute UAV flight produced roughly over 800 images totaling 1.7GB. There were over a dozen platforms flying daily for two weeks as well as Civil Air Patrol and satellite imagery. Most of the imagery was being used to search for missing persons, which means each image has to be inspected manually by at least (preferably more). Signs of missing persons are hard to see, as there may be only a few pixels of clothing (victims may be covered in mud or obscured by vegetation and debris) or urban debris (as in, if you see parts of a house, there may be the occupant of the house somewhere in the image). Given the multiple agencies and tools, it was hard to pinpoint what data has been collected when (i.e., spatial and temporal complexity) and then access the data by area or time. Essentially no one knew what they had.  Agencies and insurance companies had to manually sort through news feeds and public postings, both text and images, to find nuggets of relevant information.
  • The future of disasters was clearly in organizing, analyzing, and visualizing the data. Responders flocked to SituMap, an interactive map-based visualization tool that Dr. Rick Smith at TAMU Corpus Christi started developing after the 2010 Summer Institute and is now being purchased by TEEX and other responders. The agencies awarded $900 in prizes to NSF Research Experiences for Undergraduate students for software that classified imagery and displayed where it was taken won 1st, 2nd, and 3rd. Multiple agencies requested those apps be hardened and released as well as the Skywriter sketch and point interface (CRASAR developed that for UAVs and it is being commercialized) and the wide area search planning app developed over the last two summers by other students from the NSF REU program. In previous years, the panels have awarded prizes primarily to hardware- UAVs, battery systems, communications nodes, etc. This year, the attitude was “those technologies are available, now help us use the data they generate?”

Each year we hear the cry from emergency management “we’re drowning in data, help” and this year it was more than a bad pun.

 

How to Fly at Floods: Summer Institute PLUS Roboticists Without Borders UAV training

One word about the floods and about UAVs: informatics. Read on to see what I mean 😉

As I blogged earlier, on July 16-28, the Center for Emergency Informatics’ 2015 Summer Institute is on flooding and will bring together state agencies and municipalities who were part of the Texas floods with researchers and industry for a two-day workshop and 1 day exercise. The exercise will include UAVs flying the missing persons missions and the recovery and restoration missions.

Notice that it’s the Center for Emergency Informatics hosting the event because it’s about the right data getting to the right agencies or stakeholders at the right time and displayed in the right way that will enable them to make the right decisions. UAVs (and marine vehicles such as Emily and the small airboats being developed at Carnegie Mellon, Texas A&M, and University of Illinois) have a big role to play. But UAVs are useful only if the entire data-to-decision process works, aka informatics.

The Summer Institute July 26-28 will also host a training session for Roboticists Without Borders members specifically on UAVs and the best practices of how to fly at floods and upcoming hurricanes and collect the useful data– what do the decision makers need? Again, this is the informatics, the science of data, not the aeronautics. The training is independent of platform- because what the decision makers need is what they need 😉  The current (and evolving) best practices are derived from three sources:

  1. CRASAR RWB deployments going back to 2005 Hurricane Katrina Pearl River cresting and including the Oso Mudslides and our deployment with Lone Star UAS Center to the Texas floods,
  2. the reports and analyses of what has worked at typhoons and other flooding events worldwide, and
  3. what researchers through out the world, especially the IEEE Safety, Security, and Rescue Robotics technical community, are doing.

For example, video is not as useful as high resolution imagery for searching for missing persons. Infrared isn’t helpful except in the early morning. Some missions and terrains require remote presence style of control, other can use preprogrammed flight paths. Complex terrains such as river bluffs may require flight paths that are vertical slices, not horizontal slices. Many more and I’m sure we will learn more from each other.

The training session will consist of evening classes on July 26 and 27, with field work on July 28 at the 1,900 acre Riverside Campus. We will fly fixed-wing and rotorcraft for response missions (reconn, missing persons, flood mitigation) and for recovery/restoration (damage assessment, debris estimation, household debris estimation, power utility assessment, transportation assessment). The scenarios will be designed by experts from Texas Task Force 1 and the representatives from the agencies that would use the information, including the fire rescue, law enforcement,  Texas insurance commission, SETRAC, etc.).

It’s not too late to join Roboticists Without Borders and attend! It’s free.

Hope to see you there!

 

Robot boats as swift water rescuers, not just for critical infrastructure and restoration/recovery operations anymore!

EMILY at swift water exercise in Texas
EMILY at swift water exercise in Texas

Grant Wilde and Gino Chacon observed the EMILY rescue boat, a new concept in disaster robotics. I have followed Tony Mulligan and his work with EMILY since 2012 and EMILY is really gaining acceptance. He demoed the boat this week during a swift water exercise (which our partners Austin Fire Department participated in- thanks Coitt for the directions!) The robot acts as a barrel-shaped life ring. An operator teleoperates EMILY to a victim in the water and the victim grabs it. The operator then uses the tether to pull EMILY and her cargo to safety.

Creator Tony Mulligan assembling EMILY
Creator Tony Mulligan assembling EMILY
We’ve been pinged many times by fire rescue teams about swift water rescue- though about the use of UAVs. Swift water has many challenges, one of which is that the rescue crew in their boat is at risk for being struck by debris speeding into them. The idea was the UAV could act as a spotter and help coordinate the safety of responders and victims alike. EMILY bypasses some of the challenge by eliminating the responder.
Grant and Gino are working a slightly smaller robot boat for shallow littoral areas, definitely not swift water, for radiation sampling for treaty verification, identifying flood vulnerable areas (with the Hazards Reduction and Recovery Center), and general environmental sampling. Some nuclear power plants or towns can be readily accessed by creeks even if roads are out. Dr. Josh Peschel, a former PhD student now at University of Illinois, is working with these marine platforms to better understand water management.
EMILY assembled and ready to go
EMILY assembled and ready to go
Gino filming EMILY
Gino filming EMILY

Update: Flooding disasters continue to mount casualties and challenge response efforts

We are working with our partner the Lone Star Unmanned Aerial System Center, one of the 6 FAA test centers, to help support the flooding response and recovery efforts for the Texas floods. The floods are a sad example of how flooding costs the U.S. more than 80 lives and $8 billion in damages each year. And that excludes storm surge from hurricanes.

Technology based solutions have advanced to where employing robotic assisted solutions can aide in crisis assessments by federal, state and local officials and emergency workers. Yet more data adds another layer of complexity and extra coordination of robot assisted efforts.

Accurate ground and aerial surveys can help decision makers choose where to deploy limited resources in the best way. By quickly identifying survivors, any follow on threats from the natural disaster, collecting measurements of debris fields, volume of water flows or similar data can be provided without endangering the emergency responders.

Water gone wild and what the Center for Robot-Assisted Search and Rescue (CRASAR) is working to do:

Texas A&M’s Center for Emergency Informatics (CEI) continues its two-year study of information technologies, including unmanned systems, in response to floods. The CEI brings together practitioners, academics, and industry experts to converge and test their knowledge hardware and software to respond to an array of water borne disasters.

CEI as the host for the Summer Institute on Flooding, slated for June 16-18, will set the stage for testing ideas, marry technologies and coordinate decision makers in real-time flood based scenarios

From the 2014 summer institute 42 representatives from 12 states tapping 14 agencies, 14 universities, and eight businesses were able to judge what information technology was mature enough, or which needs more development to assist with flooding response.

This year’s summer institute will include three exercises, each representing the key problem missions identified by the participating agencies:

  • Swift water rescue – a scenario has been developed where suddenly rising waters cut off helping responders to rescue people. Responders from Texas Task Force 1 using the characteristics of a campground flooding will be offered realistic challenges. The TEEX designed exercises focus on real missions rather than around technologies.
  • Life-saving response and immediate mitigation: identify best ways to assess where’s the flood, who is at risk?
  • Restoration and recovery– exercise decision making for resource allocation to restore roads, electricity, sewage, etc. And engaging insurance companies to conduct property damage assessment and projections of debris generation estimation to manage post flooding problems with decay, vermin and disease.

The growth of small UAVs and flooding response:

Small UAVs have been tasked for at least eight disasters with flooding or had flooding associated with it:

  • Hurricane Katrina 2005
  • Typhoon Morakot, Taiwan 2009
  • Thailand Floods 2011
  • Typhoon Haiyan, Philippines 2013
  • Boulder Colorado floods 2013
  • Oso Washington Mudslides 2014
  • Balkans flooding Serbia 2014
  • Cyclone Pamela Vanuatu 2014

From the several UAV platforms that went airborne each provided rapid and often timely information to officials and for responders in search and rescue missions as well as the difficulties with recovery.

General reconnaissance: addressing where’s the flooding, where are the people cut off by the flooding, and what roads are still passable.

Hydrological situation awareness: both real-time and post-processed. The flooding caused by the Oso mudslide was a real problem and rotorcraft could hover and stare at the river, letting the hydrologists estimate the flow rate in different areas. The biggest need remains surveying the amount of flooding. And with the addition of photogrammetric image software – UAVs provided details of the terrain and potential for additional flooding or the best place to put a dyke, channel or other mitigation. Our partners at FIT took it further and printed out a 3D model of the terrain to help everyone visualize the terrain.

The power of Unmanned Autonomous Vehicles:

  • Over watch for swift water rescue teams: our friends at South Carolina Task Force 1 have been pushing us to help create the protocols for using small UAVs. Seeing something such as a logjam that might be coming down river and pose a threat to life and limb as they work to rescue people. They’ll get their wish as this is one of the scenarios to be unfolded in our 2015 Summer Institute on Flooding in June hosted by CEI.
  • Debris estimation: both the debris directly from the flood and the indirect debris that follows on from people having to rip out sheet rock and toss water logged carpets. The advances in photogrammetrics make it possible to estimate the volume of debris. That is based on having the “before” survey of the area. To test it, we flew with PrecisionHawk at the Bennett Landfill superfund site in February 2015 to estimate the volume of toxic trash, which was on fire and needed to be safely removed. Part of the data analysis included estimating the content type, because vegetation and construction materials have to get handled and processed differently.
  • Delivery of supplies to isolated regions. We learned during last year’s summer institute that if the locals can hold their own for 72 hours, usually that was sufficient. In Texas, where breeding stock can represent hundreds of thousands of dollars of investment, someone to stay behind may make sense. Determining that they are not a victim is vital. Disasters can take people by surprise with a bridge being washed out or a vital need of medicines and other perishables. A group of Texas A&M aerospace students won 2nd place for their small, hefty fixed-wing UAV that could be used to drop off heavier/bigger bundles of supplies from further distances. Groups like Matternet, who like CRASAR are members of FIT, are looking at delivering medicine with rotorcraft.
  • Delivering lifelines, life jackets, and small things to people trapped on roofs: A note about delivering things with a rotorcraft– Using rotorcraft to carry a line or bottle to someone is complicated by the weight and distribution of them. Those factors usually make the UAV very sensitive to wind and control errors. As such, if an open rotor system is used, more distance than normal from a person is needed to account for such errors. One senior design project from last year’s Summer Institute created a two-way audio system for rotorcraft. Since rotors can generate about 85 dB of noise hanging a microphone and speaker on the UAV is ineffective. The Computer Engineering team used noise reduction algorithms from National Instruments LabVIEW to prototype a lightweight 2-way audio system impervious to noise.

Look for more to come from the 2015 Summer Institute for Flooding, June 16-18.

 

Flooding and small UAVs

At this time, we aren’t involved in the horrible Texas and Oklahoma flooding. But we’ve been studying flooding since our deployment with Hurricane Charley in 2004. This blog will give a short background on what we do here at A&M and the upcoming 2015 Summer Institute on Flooding, then the history of small UAVs for flooding, and the potential uses generated by experts at our 2014 Summer Institute on Flooding.

About CRASAR and flooding

The Center for Emergency Informatics (CEI) here at Texas A&M is in the middle of a two-year exploration of information technologies, including unmanned systems, for floods. The CEI is a “center of center” that brings together practitioners, academics, and industry to fuse and apply their knowledge to disasters.  CRASAR is the “response” arm for CEI (actually we do participatory research versus response- similar to what anthropologists like Margaret Mead do. Sometimes the only way to learn is by doing by embedding with the responders, we don’t do disaster tourism).

The CEI is hosting a Summer Institute on Flooding June 16-18, the second on flooding. The choice of flooding was motivated by the fact that flooding is costing the U.S. over 80 lives and $8 billion in damages each year, excluding storm surge from hurricanes. Last year’s summer institute brought together 42 representatives and 12 states from 14 agencies, 14 universities, and 8 companies to consider what information technology is mature enough, or needs a bit of encouragement, to assist with flooding response.

This year’s Summer Institute consists of three exercises, each representing the key problem missions identified by the agencies:

  1. Swift water rescue (June 16)- helping responders rescue people suddenly cut off by rising waters. The plan for the swift water rescue reads like the campground flooding- showing the advantage of having the responders from Texas Task Force 1 and TEEX design exercises around real missions rather than around technologies.
  2. Life-saving response and immediate mitigation (June 17)- where’s the flood, who is at risk?
  3. Restoration and recovery (June 18)- restoring roads, electricity, sewage, etc. as well as insurance companies conducting property damage assessment  and cities generating debris estimation so that they can keep the rats out.

About small UAVs and flooding

Small UAVs have been used at least 8 disasters from flooding or had flooding associated with it: Hurricane Katrina 2005, Typhoon Morakot, Taiwan 2009, Thailand Floods 2011, Typhoon Haiyan Philippines 2013, Boulder Colorado floods 2013, Oso Washington Mudslides 2014, Balkans flooding Serbia 2014, and Cyclone Pamela Vanuatu 2014.

Historically they have been used for:

  • General reconnaissance: where’s the flooding, where are the people cut off by the flooding, and what roads are still passable.
  • Hydrological situation awareness, both real-time and post-processed. The flooding caused by the Oso mudslide was a real problem and rotorcraft were considered because they could hover and stare at the river, letting the hydrologists estimate the flow rate in different areas. The biggest push was for surveying the amount of flooding and- with the addition of photogrammetric image software- the terrain and potential for additional flooding (or the best place to put a dyke, channel or other mitigation). Our partners at FIT took it further and printed out a 3D model of the terrain to help everyone visualize the terrain.

Potential uses for UAVs

  • Overwatch for swift water rescue teams: our friends at South Carolina Task Force 1 have been pushing us to help create the protocols for using small UAVs to help them see that a logjam is coming down river and going to wipe them and the people they are trying to rescue out. This is one of the scenarios we will be playing in our Summer Institute on Flooding in June
  • Debris estimation: both the debris directly from the flood and the indirect debris a few days or weeks later from people having to rip out sheet rock and carpets. The advances in photogrammetrics make it possible to estimate the volume of debris— if you have the “before” survey of the area; we flew with PrecisionHawk at the Bennett Landfill superfund site in February in order to estimate the volume of toxic trash (which was on fire) that needed to be safely removed. The next step is to estimate the content, because vegetation and construction materials have to get handled and processed differently.
  • Deliver supplies to cut off regions. We learned during last year’s summer institute that if the locals can hold out for 72 hours, usually that’s sufficient. Indeed, in Texas, where breeding stock can represent hundreds of thousands of dollars of investment, it may make sense for someone to stay behind. But sometimes people get taken by surprise or a bridge washes out unexpectedly and need diabetes medicine and other perishables. Groups like Matternet, who like CRASAR are members of FIT,  are looking at delivering medicine with rotorcraft. A group of Texas A&M aerospace students won 2nd place for their small but hefty fixed-wing UAV that could be used to drop off heavier/bigger bundles of supplies from further distances
  • Delivering lifelines, life jackets, and small things to people trapped on roofs. A note about delivering things with a rotorcraft- You can use rotorcraft to carry a line or bottle to someone but the weight and distribution usually makes the UAV very sensitive to wind and control errors— if you are using a open rotor system just be aware and maintain more distance than normal from a person if at all possible. One senior design project that resulted from last year’s Summer Institute was to create a 2-way audio system for rotorcraft. The rotors generate about 85 dB of noise so if you’re a responders and want to try to talk with a person by hanging a microphone and speaker on the UAV, it probably won’t work. The Computer Engineering design team used noise reduction algorithms from National Instruments LabVIEW to prototype a lightweight 2-way audio system impervious to noise.

 

 

 

 

Nepal: recommendations for small UAVs

As the tragedy in Nepal unfolds, the immediate rescue response has ended and now efforts are shifting to agencies working on the mitigation of the event and dealing with continuing cascade of consequences and hopefully to recovery as well as humanitarian relief.  We have not been asked to participate and cannot self deploy but to those planning to fly small UAVs, I recommend that you look over the range of uses of small UAVs in the past 8 earthquakes in the past blog (and in more detail in Disaster Robotics). Plus:

  • Be aware that the altitude may change the performance of your platform
  • Working in complex terrains such as mountains will impact any preplanned paths. We have found that imagery reconstructions from fixed-wings will do better with a series of flights “stair stepping” along a hill or mountain than trying to cover the entire area at one altitude. Also a flight at one altitude may violate any flight AGL restrictions because to be high enough to fly at the top of the mountain, you’ve almost certainly exceeded the AGL limits for the lowest part of the terrain. We have found that rotorcraft flight plans work better as a set of vertical planes.
  • If you are planning to conduct structural inspection missions, you will most likely need to fly with 3-10m of the structure. Be aware that this creates wind effects and can interfere with GPS and wireless connectivity. Also, our research with civil engineers indicates that no matter how much video or photos we try to take, having a specialist who knows exactly what to look for is critical.
  • Expect engineers and structural specialists to use the raw images. Our studies at Disaster City indicate that orthomosaics do not accurately show straight edges on buildings and have a slight bit of ghosting, regardless if from fixed or rotorcraft.
  • Be aware that the country may have a temporary flight restriction in order to protect manned helicopters working at low altitudes in the area- that applies to anything that flies, there are no hobbyist exemptions. The normal procedure is for ANY aircraft, manned or unmanned, to coordinate with the air traffic control so that the manned helicopters can continue to operate. Regardless, manned systems cannot see small UAVs and thus cannot avoid. Should they see a small UAV operating and are not briefed, they typically have to return to base because of the possibility of a collision. Sending someone to the Air Branch of the incident command can go a long way to making sure ad hoc flights don’t accidentally interrupt other activities.

 

Nepal: disaster robots and earthquakes- history and uses

Our thoughts and prayers to the victims, families, and responders in Nepal where CNN is reporting over 777 people killed. Here is some information about how disaster robots have and can be used.

Uses: the primary use of disaster robots in 8 previous earthquakes have been to give authorities and experts rapid understanding of the damage and general situation, the state of the infrastructure – especially underwater portions of bridges and ports which is key for transportation of responders and supplies, and the state of building collapses- especially where there is the indication of survivors or where the building must be inspected by experts but it appears to be too unsafe to inspect.

History: Small unmanned systems have been reported for use in response and mitigation of 8 earthquakes. The first reported use in 2004- an experimental ground robot from IRS exploring a house in the Niigati Chuestsu (Japan) earthquake with the Japanese equivalent of FEMA. In 2009 a small UAV was deployed in Italy by La Sapienza with the Italian Fire Department for the L’aquila Earthquake. At the Haiti Earthquake in 2010, the Navy MSDU used underwater ROVs to clear the port so that ships could bring in responders and supplies without running aground or collapsing the piers. The Haitian airspace was under a temporary flight restriction but there was a  small UAV that self-deployed and performed reconnaissance. A small UAV was tried for indoor inspection of the cathedral at the Christchurch earthquake (2011) but the structural specialists shifted to ground robots.  Underwater robots were used extensively by municipalities with some use of the ground robots and small UAVs for structural inspection at the Tohoku earthquake/tsunami in 2011.  Small ground and UAVs were used by the NifTI team with the Italian Fire Department at the Finale Emilia earthquake for structural inspection in 2012. The Chinese military used small UAVs at the 2013 Lushan China earthquake and the 2014 Yunnan China earthquake for rapid reconnaissance of hard to reach areas.

 

 

Vanuatu: How disaster robots have helped in 12 similar events and might help there

I’m here at the UN World Disaster Conference where word of the destruction in Vanuata is coming in and our thoughts and prayers go to out the victims and families. It sounds like the effort is on humanitarian relief.  I’m not seeing any discussion of mitigation/response/recovery of critical infrastructure, which is the historical focus of disaster robotics. Here’s some information on how robots have helped in 12 similar disasters.

Small unmanned aerial systems have been used by rescue authorities in 8 storm or flooding events: Hurricane Katrina, Hurricane Wilma, Typhoon Morakot (Taiwan),  Thailand floods, Typhoon Haiyan (Philippines), Boulder Colorado floods, SR530 mudslide and flood, Serbia floods. UAS are most often used for rapid reconnaissance and mapping of the extent of devastation, condition of transportation routes and what areas are cut off, power lines, and general hydrological and geological mitigation needed to predict, contain, or drain water, etc.

Small marine vehicles have been used by rescue authorities in 3 storm and flooding events: Hurricane Wilma, Hurricane Ike, Tohoku tsunami (Japan).  They are mostly used to identify the state of bridges and ports, debris that is blocking ports or polluting fishing areas, and for the recovery of victims that were washed away into the sea. Plus they were used at the Haiti earthquake to clear underwater debris from the port to allow humanitarian relief supplies to be shipped in (it’s hard to feed a country via planes into a single airport). In Japan, the use of marine vehicles by the IRS-CRASAR team was credited for re-opening the Minamisaniruku new port 6 months earlier than would have been possible with manual divers and in time for the key salmon fishing season.

Unmanned ground robots are almost never used because commercial buildings rarely collapse in these conditions, it is mostly individual homes.

Tohoku earthquake anniversity and UN World Conference on Disaster Risk Reduction

wcdrr-logo-desktop-v3.0It is the 4th anniversary of the Tohoku earthquake and I am en route to Sendai where the United Nations World Conference on Disaster Risk Reduction. It is sobering to think that over 18,000 people died and that the great nation of Japan is still recovering.  It makes it even more appropriate for Japan to host the conference and at Sendai. I look forward to seeing my colleagues from the International Rescue Systems Institute who we worked side-by-side with unmanned marine vehicles.

I will be giving an invited talk on the current state and achievement of  disaster robotics during the Public Forum. In looking back between 2011 and now, the biggest surprise is that unmanned marine vehicles are not being used as much as I would have thought. The tsunami response and recovery showed the efficacy of these tools and how the can do in 4 hours what it take divers weeks to do (if they are available). There’s no surprise in that unmanned systems are being used more frequently! I’ll post the my findings when I give my talk– no spoilers!