This weekend we performed a quantitative analysis to evaluate the accuracy of drone based photogrammetry results. For some time I’ve been very interested to see how well a photogrammetrically derived point cloud would identify the tops of trees or man-made objects, to see how useful this technology may actually be in the performance of obstruction surveys. I want to strongly emphasize this research project was not completed near an airport. Flying a UAV near an airfield is extremely dangerous. Our test site was carefully selected at a secluded location, miles from the nearest airfield.
We utilized a Geopro customized multi-rotor carrying a 12.4mp camera with 20mm focal length flying at an AGL of 100 meters. The 4000×3000 resolution photos resulted in a ground sample distance (GSD) of 4.3 cm (0.14 feet) per pixel. This height was selected due primarily to my comfort level; I wanted to stay below 400 feet and a GSD under 5cm seemed like a good idea.
We began by setting photogrammetric ground control around the site, consisting of eight points around the perimeter and center of the project. We collected the points using a Trimble R8 Model 3, receiving corrections from the Ohio VRS network. I assume the final positions are accurate within 2cm both horizontally and vertically, though we have not rigorously verified the positions. Work was completed in Ohio State Plane South using metric units of measurements. Geoid 12B was utilized to compute orthometric heights.
Next we established temporary survey points and utilized a recently calibrated 1” Trimble total station (5601) to measure the top of objects using Direct-Reflect technology. In all we shot the top of 36 trees, 6 posts, and two light poles. Some of the trees were in full leaf-on condition, others were dead or unifoliated.
After setting the points we launched our drone, the camera setting were manually applied:
- ISO 100 to get clean photos with minimal noise
- Aperture limit of f/2.8
- Shutter speed limit of 1/400 second
In total we collected 127 photos across the project area, flying two missions at 90 degrees to each other. The images are generally sharp with minimal motion blur.
The photos were then loaded into Agisoft Photoscan and processed on ultra-high settings to avoid any image quality reduction. The GCPs and photo alignment results in an average reference point error of just over 3cm.
The final point cloud had just under 79 million points. These points were exported as a .LAS for analysis. A height-field model was then produced for visual inspection:
The exported .LAS file and terrestrially measured points were then loaded into Cloud Compare. The area around each point was segmented into its own cloud for analysis using the height-field rasterization tool using a 2cm grid and projecting the maximum height.
Note, Cloud Compare required a coordinate shift during import in order to maintain precisions. A -200m offset was applied as a simple shift, 292.865 becomes 92.865.
The results of the comparisons can be categorized by object type:
Man-made objects typically were not accurate unless they have a cross sectional area substantially larger than the GSD. Not a single post (10cm diameter) was identified by the photogrammetric software, on average missing by 2.21 meters (7.25 feet), the height of the post. Street lights with their large (60cm) top housing were accurately measured within 0.25 meters (0.82 feet). It is clear that for an object to be accurately measured, it must have a cross-sectional diameter greater than 2 times the GSD. In this case the GSD was 4.27cm and a 10cm post was not detected. I may do more testing to try to confirm or better approximate this requirement but is seems the sampling theorem may generally still be true here.
The following table compiles the results:
|Trees (healthy)||30||0.43 m||0.03 m||1.56 m|
|Trees (no leaves)||6||2.72 m||2.30 m||3.50 m|
|Posts||6||2.21 m||2.11 m||2.31 m|
|Street Light||2||0.27 m||0.28 m||0.26 m|
Looking at the results in their entirety, without differentiation between object types, it is worth noting the average vertical error was 1.08 meters (3.36 ft), just slightly more than the FAA 1A accuracy code.
Note: No evergreen trees were mapped as part of this effort, but based on their geometry, I would guess they tend to be similar to unfoliated tree results and would also be unsuitable for this approach.
My conclusion is that while drone technology offers a lot of promise, it cannot be exclusively relied upon, especially when man-made objects or slender natural features exist. A power pole without lighting or an aerial wire, for example, would go completely undetected using his drone technique. Those very dangerous objects are critical and must be accurately located during an obstruction survey, but a drone only approach would likely miss them completely.
It is important that the drone imagery be further analyzed by trained photogrammetrists to identify and measure missing obstacles. Also critical is the quality control measures performed by ground-surveyors, to measure obstacles using terrestrial techniques and ensure that no errors occurred during the automated processing of drone images. Relying on the software alone to compute obstructions could be a potentially disastrous risk, based on what we’ve seen here. With lives on the line it is essential that specially trained and skilled professionals are performing the analysis and understand the limitations of the technology.
I’ve been hearing the term PhoDAR (photogrammetric detection and ranging) a lot lately. There are numerous automated “photogrammetry” software packages being offered today that create point clouds from photos. The software looks phenomenal and many boast of incredible accuracy. Wikipedia currently lists more than 60 such solutions, with big name vendors such as AutoDesk, Bentley, Microsoft, and Trimble on the list.
These software solutions utilize camera based techniques that rely strongly on a workflow called Structure from Motion (SfM). During the SfM process, 2-Dimensional photographs of a scene are obtained from many different perspectives, the software then analyze the photos looking for similar features and patterns. Using this redundancy the software estimates a camera calibration as well as transformations between the different camera views. Typically this is done in multiple steps with an initial “sparse processing” to determine camera and view parameters, and then a “dense reconstruction” is performed that processes the images in overlapping chunks. This technique is computationally intense, but uses statistics to estimate reality and come up with a point cloud. If the same pattern matches in multiple photos a level of confidence can be estimated that point actually exists… even if it doesn’t.
I’ve been exploring this technology for nearly a decade and it has been interesting to see the market develop. In one recent Whitepaper eBee (using Pix4D I’m guessing) claims they can achieve an accuracy of 3 cm (Horizontally) to 5cm (Vertically). My guess is the key word there is “can.” Just because it can be done, doesn’t mean it is done. (How’s that for a sentence?)
If you’ve ever tried out this type of software you’ll notice glaring issues if you look at the data closely. An example of this is provided below. The far left is a photograph of the actual pole, the middle data was collected with our Faro terrestrial scanner, and the far right was computed using a very popular (and expensive) software SfM/PhoDAR/Photogrammetry package. Reality and LiDAR are very clearly in agreement, but the limitations of structure-from-motion based solutions are apparent. This point cloud was computed using 30+ images at an altitude of 150 feet (more to come on this example later).
It is interesting technology and those high-altitude demo videos I keep seeing are impressive, but I don’t think it’s quite a substitute for professional grade mapping. As this software becomes more accessible, there’s a potential danger to the public when used by drone owners who may not know the limitations. Be careful out there and stay informed, check back here often for more on this topic as we continue to explore new geospatial technologies.
Please comment below and share your opinion on PhoDAR and other automated photogrammetry solutions. Does automated photogrammetry pose a risk to the public?
Over our last decade of providing aeronautical surveying services, we have often been asked to provide training to government agencies and airport authority personnel on how to properly conduct and aeronautical survey. In response to this common request, Geopro has developed a training program available to airport authorities who wish to better understand the concepts and correct approach to conducting an AGIS-style survey.
Our survey training program focuses on the five major areas of an aeronautical survey: geodetic control, runway surveying, navigational-aid surveying, obstacles, and data reductions and reporting for ICAO specifications. The training provides real-world experience and is performed at the airfield of the client’s choosing. It also includes numerous hours of office training, where trainees are first introduced to the proper procedure for all phases of the survey process, from determining the correct runway end locations to identifying survey points for common navigational aids.
Though some training programs exist for surveying, ours is the only program that focuses on all parts of the aeronautical survey process. Another unique benefit of our program is the combined classroom and field survey training with certification exam. We offer two levels of training and certification: 40 and 80 hour programs that will take a novice airport surveyor to the certified and efficient status within two weeks.
An application process is required to be admitted to the training and not all candidates will qualify. Basic knowledge of survey instrument operation, GPS, and data processing software is required. Students who successfully complete the program will be awarded a certificate of completion for the course.
Geopro Consultants has established a worldwide network of procedure designers skilled in all aspects of airspace analysis and procedure development, including RNP Procedures to improve access and safety into often remote, but highly desirable airfields. Our teams routinely deploy to sites across the globe to perform high-accuracy data surveys to support procedure development. Your airport’s location has value; make the most of it by improving your approach procedures and maximizing your current investment.
Our team has also worked with airports within the United States to optimize existing procedures in conjunction with the FAA. Many of our procedure designing partners are ex-FAA officials with an intimate understanding of the FAA process and can spend the time necessary to understand the issues impacting your airport. A better understanding of the facts previously considered by the FAA combined with knowing what changes could improve procedures often leads to more efficient usage of available tree clearing funds.
If your airport desires updated, improved procedures and reduced minimums, please contact us today to learn how our services may help your airport grow.
Geopro Consultants’ aeronautical team has been working with the FAA Advisory Circular standards since they were first introduced in 2007. We have extensive experience completing aeronautical surveys to all iterations of the FAA AGIS Advisory Circulars, 150/5300-16, 17, 18, and 19. Our team members have been involved in hundreds of aeronautical surveys, including WAAS, LPV, RNAV, and FAR Part-77, throughout the nation. We have expertise that exceeds virtually all other survey companies in the nation when it comes to meeting AGIS requirements. If your project requires AGIS compliance, look no further than the survey team at Geopro Consultants.
Geopro staff are specially trained and skilled at performing aeronautical surveys and regularly travel throughout the country to perform critical airfield measurements. Our staff holds FAA Certification from the Airports GIS Integrated Distance Learning Environment (IDLE) for performing aeronautical surveys to the Advisory Circular 150/5300-16, 17, and 18 and has performed hundreds of aeronautical surveys to specifications including FAA 405, FAA AGIS, and DOD SDSFIE. Our team members have a thorough understanding of airfield procedures and equipment required to complete projects both on time and under budget.
Geopro can also provide data conversion services to make your current data compliant with the FAA GIS standards. Our track record of working within the new requirements ensures that existing data is converted correctly and accepted by the FAA the first time.
Geopro can help airports to understand the obstacles impacting their operations as part of a cost-effective onsite evaluation developed in response to the FAA Interim Policy from November 18, 2013. In that Policy, the FAA issued guidance regarding penetrations of the 20:1 visual area surface for instrument approach procedures. This guidance became effective on January 6, 2014, and the FAA has cautioned that they will be taking immediate action on 20:1 penetrations when identified, often resulting in increased minimums and elimination of night operations for airports.
Quoting the FAA, they “highly recommend Airport Sponsors to take a proactive approach by reviewing all approach surfaces in advance of any flight check schedule to ensure they are clear, including any planned approaches depicted on the Airport Layout Plan (ALP),” noting that, “if an airport is part of an [FAA] review that uncovers obstacle penetrations, there is a limited amount of time to act before procedures are impacted.” To get a general sense of when your airport’s procedures are scheduled for review, the FAA has published its procedure review schedule. Although the schedule does not specifically state when the FAA will begin its reviews, they typically commence 30 to 60 days prior to the “Due Date” listed in the table.
The 20:1 visual area surface is described in Section 3.3.2.c of FAA Order 8260.3B, United States Standard for Terminal Instrument Procedures (TERPS). The purpose of the surface is to protect aircraft during the last stages of the approach procedures when pilots transition from instruments to visual guidance. Objects penetrating the surface must be lowered or illuminated to ensure that pilots of approaching aircraft can see them. If they cannot be seen, the visibility minimums associated with the approach may need to be increased or nighttime use of the procedure may be disallowed.
Geopro will visit your airport and after measuring the runways and determining the latitude, longitude, and MSL altitude of the runways ends, will model a 20:1 Visual Surface in CAD-based environment with aerial photography as a base map. The team will then set up our high accuracy total-station equipment so that the 20:1 surface is aligned with the system optics. This way, objects penetrating the surface will be visible and can be located. Crews will then use our proprietary software and survey methodology to locate the obstacles and generate 3D coordinates at the obstacle top. Attributes of the obstacle, such as type, height, and lighted condition will also be documented. A final plot of the survey results will then be generated to share with airport stakeholders.
Although visual area surfaces have been defined in TERPS for many years, the FAA began reviewing these surfaces more systematically in early 2015. Initially, the FAA would take immediate action—raising visibility minimums and/or disallowing night operations—via NOTAMs to address obstacle penetrations that were identified. Unfortunately, several affected airport operators found that the obstacle data FAA used to make its determinations were erroneous (e.g., obstacles that had already been removed, obstacles that were depicted in incorrect locations). Other airports expressed concern that there was limited advance and post-NOTAM coordination between the airport and the FAA regarding 20:1 penetrations.
In response to these and other concerns from airport operators, the FAA developed established interim policy guidance to address penetrations of the 20:1 Visual Area Surface of instrument approach procedures. The interim policy incorporates a validation step during which airport operators can address erroneous obstacle data before NOTAMs are issued and provides a risk-based framework to address obstacle mitigation actions and timeframes.
Geopro Consultants provides obstruction validation services to correct the FAA’s erroneous obstacle data and identify legitimate threats to your airspace safety. Contact us today to find out more about potential risks to your approach minimums and nighttime operations.
Geopro Consultants specializes in the development of ICAO mandated aeronautical charts for obstacles, including Type A and B, and the Aerodrome Grid Chart. We also offer charting management services for airport authorities with a need for cost-effective yet highly accurate charting solutions.
By using a combination of previously existing data, satellite based aerial imagery, photogrammetry, and GIS software, our team can rapidly map, chart, and deliver ICAO compliant charts, maps, and accurate graphic depictions of the latest conditions of your airports.
Geopro Consultants offers complete e-TOD (Electronic Terrain and Obstacle Database) production and management for airports to ensure compliance with international quality requirements for terrain and obstacle data including ICAO Annex 15 up to Amendments 33, 34 and 36 and ICAO doc 9881. Using a combination of ground-based surveying techniques and photogrammetric mapping, Geopro collects the terrain and obstacle data after developing the ICAO Annex 14 surfaces.
Geopro can produce a variety of data products in support of your e-TOD challenges, including Digital Terrain Models (DTM, DSM, DEM), and digital orthophotography. Geopro is also currently working on an automated methods to produce the requisite ICAO Charts (Type A and B, Aerodrome Grid) from your current AIXM-compliant data or GIS database.
Geopro Consultants’ geodetic control experts have extensive experience establishing geodetic control stations for a variety of clients, including airports, counties, and statewide geodetic control projects. Geodetic control serves as a common reference system for establishing coordinate positions for all spatial projects on your site. Traditionally, geodetic control points are established as permanent physical monuments placed in the ground and precisely marked, located, and documented. Our team can develop and install customized geodetic control monuments including your organization’s logo for your site to create unique and prominent permanent monumentation.
A geodetic control network is a very high priority for professional surveyors, GIS professionals, and other spatial data gatherers. Control surveys establish precise horizontal and vertical positions of geodetic monuments. These serve as the basis for originating or checking subordinate surveys for projects such as topographic mapping, boundary surveys, construction planning, and design and layout. They are also essential as a reference framework for giving locations of data entered in Geographic Information Systems (GIS). Specifying locations of features relative to geodetic control makes it possible to assess the locational accuracy of these features.
Whether you need 2 monuments or 200, Geopro can help your organization to create geodetic control networks, including planning, establishment, surveying, and final reporting.