Archive for the ‘Research’ Category

#Mexico #Earthquake Overview of robots and earthquakes: background and how they can help

YouTube Preview Image  Our hearts go out to the victims, their families, and the responders in Mexico. CRASAR has not been contacted about robots but this blog may be of use in thinking about how to use robots.

 

Ground, aerial, and marine robots have been used in several earthquakes. A good overview of ground robots for structural collapse is at https://www.youtube.com/watch?v=5Cm2bGlUjbQ. It’s an older documentary but all the issues and gear are still the same.

 

Small unmanned aerial vehicles are probably everyone’s first thought for earthquakes, in part to map out the extent of the damage. They can also be used to help responders determine the shortest, most debris-free route to locations or interest.  For the reconstruction and recovery phases, UAS have been used to fly around and in large buildings that are suspected of being too dangerous for structural specialists to enter and assess the risk of further collapse- for example the cathedrals in Canterbury NZ, Mirandola ,Italy, and Amatrice, Italy.  After the Tohoku earthquake, many experts pointed out that UAVs should be used to determine the state of hospitals- both whether the hospital is still functional but also if it is being overwhelmed by patients.

 

Ground robots also have a role. An earthquake may cause buildings to completely collapse where there are no voids that a responder can get into. The general strategy is to use a canine team to determine if there are survivors in the rubble (dogs can tell if the person is still alive). Typically a boroscope or a camera on a wand is inserted to try to see if they can locate the survivors and also get a sense of the best was to remove material to get to them.  The boroscope or a camara on a wand can only go about 18 feet into the rubble, depending on how twisty the void is. In a major building collapse, survivors may be much further, which is why small, shoe-box sized robots such as an Inuktun VGTV may be used. The “cameras on tracks” robots can pull themselves into the rubble and also change shape to help get into tight spots.  These small robots will usually be tethered, with the tether acting as a belay line and the tether preventing loss of signal.

 

Some interesting robots are the snake robots being developed by Howie Choset at CMU and the Active Scope Camera caterpillar robot being developed by Olympus in conjunction with Japanese researchers led by Satoshi Tadokoro. The ASC was used at the Jacksonville building collapse in 2007.

 

Bigger ground robots, such as those used at Fukushima, can be used in bigger buildings but generally can’t see the ceilings, which is usually very important and happened the New Zealand earthquake. They can’t go into small voids and may be too heavy- they could cause delicately balanced rubble to further collapse and kill a survivor underneath.

 

Marine robots, especially ROVs, are important as was seen in the Haiti earthquake and Tohoku earthquake and tsunami. The earthquake may have changed the shipping channels, damaged bridges and ports, and put debris in unexpected places. Thus shipping is stopped until the shipping channels are cleared- and as was seen at Haiti it’s hard to feed a country with one airport. Shipping is extremely important for getting relief supplies in.

 

There are more details in the case studies in Disaster Robotics  and the Springer Handbook of Robotics on what robots have been used and particular strategies. All of us are happy to answer questions. We wish everyone the best on this terrible event.

Update on Hurricane Irma Unmanned Aerial Systems: new record of 247 flights for public officials

FSU's Mike McDaniel with Collier County official flying a DJI MavicA major update from our earlier post    CRASAR was at Hurricane Irma supporting Roboticists Without Borders’ member Florida State University Center for Disaster Risk Policy deployment to Collier County, Florida. The CDRP effort was led by David Merrick, CDRP director, with Justin Adams, Kovar and Associates, who led the CRASAR Harvey response serving as his deputy. Collier County is in southern Florida near Naples and Marco Island. This is familiar territory to CRASAR, who responded to Hurricane Wilma in 2005 at Marco Island, sending out the first known use of unmanned marine vehicles. The county was exposed to severe wind damage and flooding from Irma which passed over as a Category 3 hurricane on September 10. The teams demobilized on September 16, with last flights on September 15.  Six UAS pilots flew 247 imaging flights covering over 491 critical infrastructure targets, as well as provided overwatch for FL Task Force 8 and made multiple maps. The flights started on September 11 and surpassed the record of 119 mission flights for public officials during a disaster set at Hurricane Harvey. FSU and Texas A&M plan to offer a joint day-long short course on small UAS for emergency management on October 21, expanding the course that they have taught at the Florida Governors Hurricane Conference in May 2017 and for Los Angeles County Fire Rescue last week. Contact Robin Murphy for more information about the Irma deployment and the upcoming course.

 

The FSU CDRP team under the direction of David Merrick was a major component of CRASAR’s record setting small UAS Hurricane Harvey response, leaving Texas with two days to prepare and predeploy for Irma. The use of small UAS at Irma initially followed a similar pattern to Hurricane Harvey, with UAS being used to assist with rapidly conducting search and rescue operations and determining the best route for US&R teams to reach people at risk but now has shifted to determining the state of over 1,500 critical infrastructure targets in the county. These targets, which include bridges and waste water treating stations,  impact the restoration of services and the economic recovery of the regions. The use of small UAS has significantly sped up the process, provide more complete assessment of all sides of a target, and multiple targets can be examined on one flight.

 

The FSU CDRP-led teams consist of pilots, data managers, platforms, and the RESPOND-R mobile lab. The teams are from FSU, Kovar and Associates, and CRASAR. The teams have 7 pilots with a fleet of 20 platforms including the DJI Mavic, DJI M600 Pro, Inspire, Intel Falcon 8, Disco, Phantom 3, and PrecisionHawk Lancaster rev 5. The teams are using the lessons learned at Harvey to improve rapid sortie planning, team situation awareness, and streamlining data management. Data is being collected by CRASAR and FSU from both deployments to permit developers to build better UAS optimized for the wind and operations tempo in a disaster and user interfaces that facilitate the data-to-decision process enabling county emergency management experts to rapidly get the right information and make good decisions. The data will also support the creation of  new visualization tools for responders, enabling them to sort through terabytes of aerial imagery, and serve as a foundation for machine learning and computer vision algorithms to process tetrabytes of data.

 

 

The Irma response differs from the Harvey response in at least four key ways.

  • the hurricane posed primarily wind damage with flooding as a secondary impact, whereas Harvey in Fort Bend County was primarily flooding from rainfall and then the risk of additional river flooding. This changed the style of search and rescue operations, especially as cellular coverage was affected and residents could not always call for help. The UAS teams assisted Florida Task Force 8 as the aerial view helped  the searchers determine where to go, which flooded house to inspect next, and to better coordinate operations. The wind damage in Irma made route clearing operations more important as downed trees could unpredictably be blocking roads as compared to flooding with tended to inundate specific areas and predicted from flood maps.
  • the majority of missions are for critical infrastructure property damage assessment. These assessments were normally being done in person.  This is time consuming for driving to the site (including determining alternative routes) and then requires a person’s time to survey the target. In many cases, the inspector cannot see or get to all sides of a facility. While both manual and UAS inspection require the same amount of time to get to a target, the use of UAS is being shown to be faster than walking around and more complete as the UAS can fly around the target and also provide an birds’ eye view as well. For example, on Sept. 13, two 2-person UAS teams were able to document 97 infrastructure targets with 12 flights. Our initial look at our logs indicate an average of 16 minutes on-site for a mission– that’s stop the car, turn on the UAS, fly the mission, get back in the car. And a flight covers an average of 2 targets per flight. It’s hard to believe that a person can walk around a tank farm faster.
  • A single flight (or sortie) at Irma typically covers multiple targets (an average of 2 targets), while at Harvey a single flight covered only one direct mission objective (though the aerial imagery was used to inform additional multiple emergency support functions).
  • Operationally, the resident population and airspace traffic is less dense, with less manned assets flying in the region and the operations did not require a Temporary Flight Restriction for safety. The Irma teams have less challenges in planning flights to avoid flying over people and did not encounter self-deployed teams or hobbyists interfering with UAS flights for officials; these challenges were seen at Hurricane Harvey.

Hurricane Harvey Deployment for Fort Bend County OEM: 119 flights over 11 days

YouTube Preview Image
Group photo, missing USAA and Hydronalix teams.

Group photo with TAMU, FSU, Kovar & Associates, LSUASC, PrecisionHawk and Intel, missing USAA and Hydronalix teams.

The Texas A&M Engineering Experiment Station Center for Robot-Assisted Search and Rescue (CRASAR) coordinated the largest known deployment of unmanned aerial systems (UAS) by public officials for a federally declared disaster- both serving as Air Operations for manned and unmanned aircraft and deploying small UAS ranging in size from DJI Mavics to the Insitu ScanEagle. The deployment was for the Fort Bend County Office of Emergency Management with whom CRASAR had provided assets for previous floods. CRASAR flew 119 mission flights from August 25 (preparing for landfall) to September 4 (when the emergency life-saving response and restoration of services phases of the disaster were largely over), with a record 61 flights on one day. The deployment was led by Justin Adams, who served as Air Operations branch director for Fort Bend County manned unmanned ops and CRASAR Roboticists Without Borders coordinator. Videos are available on Fort Bend County OEM’s website in accordance with the county’s drone data policy; there may be a backlog of posting due to the size of the event.

The UAS flights were for

  • rapid spot checks of situation awareness of people in distress
  • the extent of flood and tornado damage
  • how many people had not evacuated, access routes to neighborhoods
  • projecting how long the neighborhoods would be cut off, throughout the county (which is very large and hard to get a handle on)- based on information coming to EOC, the county’s projections and knowledge from 2016 floods- not just easiest or most compelling for media to fly or waiting for complete coverage by manned assets
  • inform the public and dispel rumors- allowed County Judge Hebert  to immediately and directly address Citizens’ concerns and dispel rumors, e.g., postings to social media about a particular neighborhood, then tasked to fly that neighborhood to get eyes on to inform the EOC as to the situation and to show the community; water is coming over the Richmond railroad bridge
  • systematically document damage for federal disaster relief and future planning
  • project river flooding by monitoring the river and confirming river flood inundation models
  • monitor the river and condition of over 100 miles of levees through out the county

There were additional flights for videography, training, and some flights were actually multiple flights where UAS had to return home for a battery change before continuing. Each mission flight was to satisfy a request made by officials throughout the county following their incident command system. CRASAR also directed Air Operations for the county, coordinating all manned and unmanned assets.

The center was requested by Fort Bend County Office of Emergency Management on Aug 24 and activated the Roboticists Without Borders program, which consists of companies, universities, and individual experts who have been trained in disaster response; volunteer their time, travel, and equipment; and conform to Fort Bend County OEM data management policies. The RWB brought in 24 unmanned aerial vehicles and 2 unmanned marine vehicles in from five institutional members: Florida State University Center for Disaster Risk Policy (5), GroundVu (2), Hydronalix (3),  Kovar & Associated (1), Lone Star UAS FAA Center of Excellence (3),  PrecisionHawk (1), and USAA (2) plus Intel (1). RWB provided 13 UAS pilots, 3 UMV operators, and 4 data managers. The team members were from 6 states: Arizona, California, Florida, Indiana, Texas, and Washington State. In addition PrecisionHawk donated five Lancaster UAS and access to their DataMapper software, Boeing Insitu deployed their Scan Eagle through Lone Star UASC,  and Intel loaned a Falcon 8 UAS designed from structural inspection, for a total of 25 UAS platforms. The platforms were, in alphabetical order, AirRobot 200, AirRobot 180, DJI Inspire, DJI Mavic, DJI M600 Pro, DJI Phantom 3 Pro, DJI Phantom 4 Pro, Insitu Scan Eagle, Intel Falcon 8, Parrot Disco, PrecisionHawk M100, PrecisionHawk Lancaster 5, 3DR Solo, and UAUSA Tempest.

In addition, there were other significant UAS donations by other groups. Rocky Mountain Unmanned Systems loaned a Z30 camera high resolution and high zoom payload that increased the area of view. AirMap adapted their popular UAS flight app for use by CRASAR, allowing the members to see which of their UAS were in the air at the same time as well as other UAS who reported that they were flying in the area. PrecisionHawk’s LATAS tags for the Lancasters were very helpful as well. LSUASC loaned access to Harris’ RangeVue software, allowing AirOperations to see manned flights and alert UAS teams of approaching low flying aircraft.  Other donations and support came from ESRI, FireWhat, GroundView, RemoteGeo, Salamander Technologies, Sweetwater Video, and TAC Aero.

Two EMILY unmanned marine vehicles, small robot boats, were available. One was outfitted with a side scan sonar and used to attempt to determine flow rates of the river. The other was outfitted for swift water but was not needed during the rescue phase of operations.

Hurricane Harvey: some video of flights in Fort Bend County

UAVs We’ve certainly been busy flying for Fort Bend County Office of Emergency Management plus our member Justin Adams serving as AirBoss for manned and unmanned aircraft- see some videos. The graphic is a quick look at the platforms on tap for the response here. Big shout out to Rocky Mountain Unmanned Systems and PrecisionHawk for their donation of equipment plus Insitu coming out with a Scan Eagle as part of the resources contributed by Lone Star UAS Center!

FBC policy is to post all their drone video- this worked great during the April 2016 floods where the people manning the phones could tell worried family members to go look at a particular video of the river by a senior assisted living facility.  They are a bit behind in posting but if you are interested some of the videos are now available on the Fort Bend County Office of Emergency Management YouTube channel.  The missions range from situation awareness of neighborhoods (who’s still there, how severely and long are they going to be cut off before the waters recede) to bridge and dam inspection. Our FSU Center for Disaster Risk Policy team has been flying extensively and even flew off a flat bottom boat.

New Zealand: what can robots do for a tsunami and quake?

Our thoughts and prayers go out to the Kiwis and especially to our colleagues at the New Zealand Fire Service who have been diligently adopting robotics.

So when a tsunami strikes, what can robots do? As was shown at the 3/11 Japan tsunami, unmanned marine vehicles can accelerate economic recovery by inspecting critical underwater infrastructure

YouTube Preview Image As was shown by our Japan-US deployments at the invitation of two municipalities at the 3/11 Japan tsunami, unmanned marine vehicles  (UMV) can assist with the response and accelerate economic recovery by inspecting critical underwater infrastructure- the underwater portions of bridges, ports, and shipping channels that are vital for access by responders and for getting supplies to any cut off populations. Later, the UMVs can help with environmental remediation, finding fishing boats and cars leaking gas and oil into pristine fishing waters and identifying other sources of pollution or dangers to fishing and navigation.

UAVs could be used to assess the overall boundaries of the incident, though most of the damage is near the ground. Like flooding, this is hard to get the angles to accurately assess damage. In places such as New Zealand, the agencies (and news media) generally have enough resources to get a general aerial assessment.

 

Emergency Managers Find Small Unmanned Aerial Systems Effective for Flooding and Popular With Residents

A paper to be presented next week at the IEEE International Symposium on Safety Security and Rescue Robotics in Lausanne, Switzerland, details the use of small unmanned aerial systems in two recent Texas floods in Fort Bend County, a major Houston suburb and 10th largest populated county in Texas. The 21 flights over four days provided flood mapping and projection of impacts, helping the county prepare and respond to the floods. Surprisingly, the flights did not encounter public resistance and the videos became a popular and useful asset for informing the county residents as to the state of the flooding. A pre-print is available here.

The small unmanned aerial systems were deployed through the Roboticists Without Borders program of the Texas A&M Engineering Experiment Station Center for Robot-Assisted Search and Rescue for two flood events in April and May 2016. Both events were presidential declared disasters.  Experts from DataWing Global, CartoFusion Technologies, USAA, and Texas A&M embedded with the Fort Bend County Office of Emergency Management and the Fort Bend County Drainage District to fly low-cost DJI Phantoms and Inspires. The flights provided flood assessment including flood mapping and projection of impact in order to plan for emergency services and verification of flood inundation models, providing justification for future publicly accountable decisions on land use, development, and roads.

The paper, titled Two Case Studies and Gaps Analysis of Flood Assessment for Emergency Management with Small Unmanned Aerial Systems by Murphy,  Dufek, Sarmiento, Wilde, Xiao, Braun, Mullen, Smith, Allred, Adams, Wright, and Gingrich, documents the successful use of the small unmanned aerial systems for the two. It discusses the best practices that emerged but also identifies gaps in informatics, manpower, human-robot interaction, and cost-benefit analysis.

The annual IEEE International Symposium on Safety Security and Rescue Robotics was established in 2002 by the IEEE Robotics and Automation Society. It is the only conference dedicated to the use of ground, aerial, and marine robots for public safety applications. It typically attracts 60-150 researchers, industrialists, and agency representatives from North America, Europe, and Asia. This year’s conference will be held at Lausanne, Switzerland, see http://ssrrobotics.org/ for more information about the conference.

The TEES Center for Robot-Assisted Search and Rescue is the leader in documenting, deploying, and facilitating technology transfer of unmanned systems for disasters. It has inserted robots or advised on the use of robots at over two dozen events in 5 countries, starting with the 9/11 World Trade Center and including Hurricane Katrina and the Fukushima Daiichi nuclear accident.

For more information contact:

 

Justin Adams, US Datawing and UAS lead for Roboticists Without Borders, justin.adamas@datawinglobal.com , 832.653.1057

Dr. Robin Murphy, director for the Center for Robot-Assisted Search and Rescue, robin.r.murphy@tamu.edu, 813.503.9881

For Hurricane Matthew: Quick Guide For Agencies Flying small Unmanned Aerial Systems (SUAS) for Emergencies

The illustrated version in pdf is here.

This quick guide is aimed at helping emergency managers quickly determine how they can exploit small unmanned aerial systems (like quadcopters).  The guide covers our best understanding of who can fly?  where can they fly?, and  any additional considerations in planning. Our best practices series has other documents on what kind of data you can expect to get, flight duration, etc., but this guide is about how the new regulations impact emergency managers. It is based on our SUAS deployments since 2005 and lessons learned from deployments by our colleagues.

 

WHO CAN FLY?

 

If members of your agency own a small UAS or have friends with a small UAS, they cannot fly at the disaster- even if they aren’t asking for money. The FAA has repeatedly ruled that a) disasters are a business or government activity and  b) if the UAV flight is a donation to a business or government, it is the same thing as if the business or government agency flew directly.

 

Therefore, the only people/companies who can fly are those with a:

  • Part 107 license. The license is new and many people/companies don’t have these yet.
  • 333 exemption. Essentially a business license versus of the COA. Many hobbyist declared themselves a company to get a 333.
  • COA. Essentially a government or academic license.

 

Your agency does not have to have the 107, 333, or COA– just formally invite the group to fly on your behalf. If the group has one of the above, there are three important caveats.

 

1. Controlled airspace. They can fly at a disaster in uncontrolled airspace, but will need special permissions for controlled airspace. Keep in mind, many densely populated areas will be in controlled airspace.

 

2.  They have to obey all the flight restrictions for their license, including Temporary Flight Restrictions. Getting permission to fly under a Temporary Flight Restriction does not give them permission to change up the rules, it only means that they are now coordinated with the rest of the air traffic who will expect them to obey the same rules as in normal flights.

 

3. 24 hour notifications before flights may be required.  If the group is flying under a 333 or COA, they have to post an online notice of intention to fly in a specific area, called a NOTAM, 24 hours in advance. So if you think you are going to have a group fly, have them declare as soon as you know. There is no downside to filing a NOTAM and then not flying.

 

 

WHERE CAN THEY FLY?

 

For planning purposes there are 3 types of airspaces: uncontrolled, controlled, and TFRUncontrolled means they can fly anywhere that is not controlled according to their license. TFR was covered above. That leaves the controlled airspace.

 

You can quickly determine if an area you want a group to fly in is in controlled airspace by going to:

 

https://app.airmap.io/

 

and enter the nearest town, then click the appropriate boxes.  What is “Controlled airspace” and what you have to do to get permission to fly in it will depend on whether the group has a) a Part 107 license or b) a 333 exemption or COA.

 

a. Determining Part 107 controlled airspace.  If the group has a 107, click on the menu on the left that says Controlled Airspace and “all”. You will get something like this:

 

 

 

 

Anything in shade means that it is controlled airspace. This means that they can fly only IF they have an airspace authorization that they have applied for in advance online and gotten approval. Note: the FAA system is backlogged by weeks, so for Matthew, this may not make possible to get approval fast enough.

 

b. Determining 333 or COA airspace.

 

Clear airmap and instead click on “blanket COA”. You should get something like this:

 

 

 

Any area in orange means that the airspace is off limits without additional permissions- no matter what altitude you are flying at.  The controlled airspace is due to airports. A local group may already have permission to fly in those areas, but may not. If not, permission to fly in controlled airspace on short notice is handled through an Emergency COA, also called ECOA, process. The process takes about 1 hour to get through the FAA- assuming you have the GPS coordinates of where you want to fly, the COA number, etc.

 

The key is that the tower has to approve the flights (actually the approve the process of letting them know where you’re flying, when you take off, land, etc.) and the FAA has to agree to the temporary extension of the current license.

 

  • Note about 333 exemption. ECOAs are granted only to businesses or agencies, not individuals doing business as. Too many quasi-hobbyists were trying to fly at disasters without working with a response agency.

 

 

ARE THEY ANY OTHER CONSIDERATIONS?

 

There are three considerations:

 

  • Data. The data (images, video) really belongs to your agency and needs to be handled as such. It may have personal identifying information. Some groups may routinely post videos and images to the web or tweet, which might not be appropriate. Therefore, you may want to make clear what the data management policies are applicable to flights on your behalf.

 

  • Privacy, state laws, or other regulations plus the public perception.  There may be state or local rules that impact the use of SUAS. Regardless, if you have a group flying SUAS for disasters, the residents will need to be aware that they are legitimate- plus the teams will be magnets for residents asking for help or assistance. So you will probably want to plan to have an agency representative in uniform or vest with the team.

 

  • Some SUAS may be software disabled from flying in TFR areas. DJI Phantom 3 and Inspires, which are very common, are now disabled by the manufacturer when a TFR is in place. So that may be something to discuss with your SUAS team.  DJI does have a procedure that allows agencies to override the software and fly up to 1.5 nautical miles from an airport, trusting the group to have obtained permissions.

Missions, Choice of sUAS Platforms, and Manpower for Flying Floods: Lessons learned from our deployment with Fort Bend County May 30-31, 2016

YouTube Preview ImageCRASAR was in the field from Monday and Tuesday (May 30-31, 2016) at the request of Fort Bend County Office of Emergency Management, with Roboticists Without Borders member CartoFusion providing off-site data support.  We flew a small UAS (DJI Phantom 3 Pro) throughout the county to help them validate their flood inundation models, conduct hydrological forensics, and educate the public on why evacuations were necessary. This is our third response to flooding this year (Louisiana March, Fort Bend County April, and this event) and we continue to learn about missions, selection of sUAS, and crew organization and CONOPS building on my previous recommendations for flooding.  I am working on a formal cognitive work analysis using the shared roles model but some preliminary lessons that may be of use to other teams are discussed below. We’d like to especially thank Jeff Braun, Lach Mullen, Adam Wright, and Juling Bao from Fort Bend County. Traci Sarmiento served as VSO both days, Xiaosu Xiao and Grant Wilde took turns being data manager, and I was the pilot.

 

Check out some of the YouTube clips at (some of which were on the CBS news):
RWB member Hydronaulix also offered Texas Task Force 1 the use of five EMILY swiftwater rescue robots that we used in Greece (see NPR story here) but they weren’t needed.

Missions

Six distinct missions have emerged from working with emergency managers:
  • property damage assessment: can sUAS help document the number of houses damaged and the amount of damage in terms of height of water into the houses while there is still flooding in order to qualify the area for disaster assistance? Our experience in Louisiana suggests that UAS of any size are not a good fit because the UAS cannot see an 18 inch high water line on houses in a subdivision crowded with trees or cover much within LOS. This mission appears to be a better match for a robot boat which can zoom down the flooded streets.
  • flood mapping and projection of impact: can sUAS help document the extent of the flood, the impact on residents, roads, levees, etc.? sUAS appear to have advantages over manned aircraft for forested regions where the platform can operate safely at lower altitudes and hover and stare to detect flowing water in between trees. An expert can use the sUAS  in order to identify possible causes of unexpected flooding and mitigation.
  • verification of flood inundation models
  • flood monitoring over time: This is related to flood inundation modeling and one of the reasons why Fort Bend County had us fly multiple days.
  • justification for publicly accountable decisions: The documentation of flooding is useful for future land use planning. Fort Bend County was particularly interested in capturing compelling video of the floods in the western part of the county which were severely flooded to show residents in the eastern part of the county which had not been yet been flooded so that they could see why evacuations were mandated.
  • public information: Fort Bend County immediately posted the video to YouTube and began pointing citizens to the video to answer questions about their neighborhood. One of the neighborhoods filmed had an assisted living facility and relatives calling into the OEM were directed to look at the video and see that the flooding wasn’t going to impact their family member.
Flood mapping, verification of flood models, and flood monitoring are not new missions and have been known for some time. Property damage has always been a projected use of sUAS for floods, though we did not expect to encounter the practical problems introduced by trees, trees, and more trees. People like trees in their yards.

Choosing the type of sUAS

The majority of missions for sUAS (versus larger UAS that can fly higher and longer areas) are expert-in-the-loop missions (more formally called remote presence), where the expert wants to be able to view the video and then direct the sUAS to a better view. It is not clear that orthomosaics and digital elevation maps (DEM) are a priority. The emergency managers generally have DEM already (though they may be outdated) and the areathat they want to look at is so large that a sUAS team is unlikely to reach all of the areas if restricted to line of sight operations— this is where larger UAS or Civil Air Patrol can be of great value. Another problem is that the file size of orthomosaics is unwieldy for OEMs to handle, share, and post among themselves. Not that many mangers have laptops that can handle a 55GB file and the upload times are slow.

We have converged on quadcopters being the default platform because of expert-in-the-loop flying and because of the physical constraints of landing zones, though we don’t rule out fixed wing. In the field there is limited access to the area because of the flooding. Access points such as raised roads or levees also had high tension powerlines which can induce interference- indeed, we had one “whoa!” takeoff next to powerlines. (We won’t discuss the creepy horde of swarming insects making it unsafe to stand in one potential site, or the fire ant bites I am sporting from a misstep at another site.)  Flying near rivers or from residential areas is hard because of trees and power lines. Empty lots without trees are rare, especially in older and urban parts of town, though suburban areas may have soccer fields suitable for launching and recovering fixed-wing sUAS.

Manpower: Crew Organization and CONOPS

This is my area of research, so it is always of interest to me! As noted in previous blogs, my TED talk, and papers, it’s the data that is the barrier to adoption. We’ve converged on a 4 person field team plus dedicated data management team back at the base to handle the data. Gee, that sounds like a lot, doesn’t it? Well, it is better than accidentally writing over data or taking extra hours to get data products to the OEM.

 

Let me explain about the field team. I see four major roles of the field team which leads to 4 people:
  • pilot, who is in charge of the sUAS
  • visual safety officer, who is not allowed to look at the pilot’s display or do anything but eyes on the sUAS and sky (and given the number of manned aircraft zipping by at low altitudes, that is an important and full time job)
  • agency expert, who actually knows what to look for and to opportunistically direct the flight
  • data manager, who immediately backs up the data (hard lesson learned at the 9/11 World Trade Center robot deployment) and makes sure all data is logged and stored for immediate hand-off to the OEM (want to give them that thumb drive as soon as sneaker net permits)

 

The pilot and VSO have to be dedicated roles, held by different people. In the future, one person could share the pilot and agency expert roles but for now it seems unreasonable to expect a flood inundation engineer to also be proficient with sUAS. One person can’t share the agency expert and VSO roles because then the person is sometimes looking at the display and sometimes at the sUAS, which is not permitted by regulations and bazillion of  safety studies.

 

There’s room for debate for having an agency expert- RWB member David Kovar points out that requiring an agency expert can hold up getting to the field. But in my experience, if the flights are exploratory, require expertise, and opportunistic, it is more effective in the long run to have them there. But not all missions for all disasters are remote presence. But for the missions that have emerged for the floods so far, this appears to be reasonable.
By the way, having an agency expert is valuable to the agency— citizens come up and ask questions that we can’t answer and they also like seeing that their officials are out there doing proactive things. It’s also important because it also helps with legitimacy. As a woman with a woman safety officer, we look more like a team of news people trying to sneak in and get footage than engineers operating under the official request of the county.

 

Back to roles, we’ve typically tried to keep it to a 3 person team by having the person serve as the VSO also serve as the data manager. Doubling up on roles seems like a great idea as the VSO takes notes, then when the platform lands, can take the data, make sure the files aren’t corrupted, and as the team drives to the next site, make backups, put in folders with filenames more helpful than say “DCIM”, and so on.

 

Except it never happens that way- which could be me. The VSO starts the process then before everything is done has to stop in the middle because of a) carsickness, b ) too bumpy to type, c) we’re at the next site and the VSO has to do a safety check and help set up, or d) all of the above. The pace just exceeds capacity. In Louisiana, we wound up sorting through a wad of video and imagery, requiring about 3 hours of extra effort at the end of a very long day. You can’t dump this on the agency person (“look, you’re now the data manager, follow this to do list! And remember, we’re here to help you!”) because they are busy making their own notes, talking to the OEM about what they are seeing, and talking to residents.

 

Bringing a fourth person along to do nothing but to sit in the truck with the air conditioning on and manage the data was a huge win on this last deployment. But again as David points out, a well honed team can do this. I suspect that that teams that do not fly together or fly these types of rapid fire missions (called ad hoc teams in industrial psychology)  will need the fourth person and real high performing teams will be able to combine roles.

 

But if you are handling data in the field, why do you need a data management team back at the base? Well, someone needs to create visualizations such as SituMap and summaries of where the data is from, edit high value snippets, upload data over a faster internet connections, etc. The field data manager doesn’t have access to the faster internet, can barely keep up with pace as it is so doesn’t have time to do snippets, and needs to head right back out to the field. Noooo, the OEM doesn’t do this. They are too busy to handle the data themselves. Their solution is to data management is to shout, “pause it there!” and then whip out their cellphone to video the video that is playing on the screen so that they can mail it and post it internally (an Ewok approach to snipeting and reducing resolution). They are trying to USE the information, not process it themselves. So while having Xiaosu or Grant come with us in the field was great, they couldn’t get it all done and we did not even try to work in the visualization software or photogrammetrics.
Again in theory, agencies have people dealing with ESRI and Big Files, but I have encountered this capacity once, maybe twice in my deployments. And in that case, the need to use these products caused a huge problem between the tactical responders and engineers and the people back in the emergency operations center.  So I still see there is a need for “immediate and intermediate” data processing.

More flooding– recommendations for small UAVs

Flooding continues through the southeast and we are getting some preliminary requests– here’s a quick rundown of  previous blogs:

suggestions from our work at the Texas floods where we flew with Lone Star UASC

a history of use of robots at floods

why the flood of data may be the biggest problem in floods

and some suggestions on flying for floods

plus best practices:

Let’s hope the flooding is not too bad- a bit of the luck of the Irish in time for St. Patrick’s Day.

Common UAV Software May Not (Yet) Be Reliable for Building Safety or Damage Assessment

Reconstruction exhibiting all four types of anomalies

Sample reconstruction exhibiting all four types of anomalies

(Note: there is lots of great work going on worldwide and we look forward to working with all companies and researchers to help improve this vital technology)

Researchers at Texas A&M and University of Nebraska-Lincoln have found that popular software packages for creating photo mosaics of disasters from imagery taken by small unmanned aerial systems (UAS) may contain anomalies that prevent its use for reliably determining if a building is safe to enter or estimating the cost of damage. Small unmanned aerial systems can enable responders to collect imagery faster, cheaper, and at higher resolutions than from satellites or manned aircraft. While software packages are continuously improving, users need to be aware that current versions may not produce reliable results for all situations. The report is a step towards understanding the value of small unmanned aerial systems during the time- and resource-critical initial response phase.

“In general, responders and agencies are using a wide variety of general purpose small UAS such as fixed-wings or

quadrotors and then running the images through the software to get high resolution mosaics of the area. But the current state of the software suggests that they may not get always the reliability that they expect or need,” said Dr. Robin Murphy, director of the director of the Texas A&M Engineering Experiment Station Center for Robot-Assisted Search and Rescue and the research supervisor of the study.  “The alternative is to purchase small UAVs explicitly designed for photogrammetric data collection, which means agencies might have to buy a second general purpose UAV to handle the other missions. We’d like to encourage photogrammetric software development to continue to make advances in working with any set of geo-tagged images and being easier to tune and configure.”

In a late breaking report (see SSRR2015 LBR sarmiento duncan murphy ) released at the 13th annual IEEE International Symposium on Safety Security and Rescue Robotics held at Purdue University, researchers presented results showing that two different photogrammetric packages produced an average of 36 anomalies, or errors, per flight.  The researchers identified four types of anomalies impacting damage assessment and structural inspection in general.  Until this study, it does not appear that glitches or anomalies had been systematically characterized or discussed in terms of the impact on decision-making for disasters.  The team of researchers consisted of Traci Sarmiento, a PhD student at Texas A&M, Dr. Brittany Duncan an assistant professor at University of Nebraska- Lincoln who participated while a PhD student at Texas A&M, and Dr. Murphy.

The team applied two packages, Agisoft Photoscan, a standard industrial system, and Microsoft ICE, a popular free software package to the same set of imagery. Both software packages combine hundreds of images into a single high resolution image. They have been used for precision agriculture, pipeline inspection, and amateur photography and are now beginning to be used for structural inspection and disaster damage assessment. The dimensions and distances between objects in the image can be accurately measured within 4cm.  However, the objects themselves may have glitches or anomalies. created through the reconstruction process, making it difficult to tell if the object is seriously damaged.

The researchers collected images using an AirRobot 180, a quadrotor used by the US and Germany military, flying over seven disaster props representing different types of building collapses, a train derailment, and rubble at the Texas A&M Engineering Extension Service’s Disaster City®. The team flew five flights over 6 months. The resulting images for each of the five flights were processed with both packages, then inspected for anomalies using the four categories.