How Much Testing is Required

Performance testing of the building enclosure is a common requirement on new construction projects, especially larger projects in the commercial and government sectors. Testing is one of several ways to enact a quality control process to prevent or mitigate the potential for issues after the building is completed and occupied. One of the more common issues that building owners, contractors, and designers hope to avoid is water penetration through the building enclosure to the interior; therefore, performance testing typically includes some form of water penetration resistance testing, such as ASTM E1105-15, Standard Test Method for Field Determination of Water Penetration of Installed Exterior Windows, Skylights, Doors, and Curtain Walls, by Uniform or Cyclic Static Air Pressure Difference.1 While the ASTM E1105-15 test method, which is shown from the building exterior in Fig. 1, can be readily performed and understood, one aspect of performance testing that is markedly not as well understood can be summarized by the following question:
How much water penetration performance testing should be required on a new construction project?
The short answer to this question is that currently there is no industry-wide consensus or agreement for the extent or quantity of field testing of water penetration performance required for a single project. The lack of a consensus, along with existing and vague guidelines in frequently referenced documents, creates a good deal of confusion and misunderstanding for specifiers and contractors. To clarify the existing confusion, it is first necessary to examine how ASTM E1105 testing is typically specified and what existing trade association documents require for testing quantities. Note that the examination will be limited to quantities for ASTM E1105 testing of windows since,
Figure 1. Typical ASTM E1105 test in progress.
Photo by Matthias Heil on Unsplash
18 • IIBEC Interface February 2022
This paper was originally presented at the 2021 IIBEC Building Enclosure Symposium.
in this author’s experience, this testing is the
most common water performance test for new
construction. An examination of quantities
for other types of building enclosure tests is
beyond the scope of this paper.
SPECIFICATION
TEST REQUIREMENTS
One of the biggest misconceptions with
respect to ASTM E1105 testing is that simply
including ASTM E1105 test requirements in
the specifications is sufficient, and no other
information is required by the specifying
authority. This author has been on multiple
projects where the designer’s response to questions
such as “How many E1105 tests should be
performed?” or “What is the minimum test area
size?” was to reference the ASTM E1105 standard.
The problem with this response is that
ASTM E1105 does not include guidelines that
explicitly define testing quantities or extents.
This is not a fault in the ASTM standard, as the
general intent of the standard is to clearly define
the test procedure, provide requirements for the
testing equipment, and state information that
should be included in a test report. In fact, the
standard states that “the specifying authority
shall supply the following information…Test
Specimen sampling, selection, adjustment, and
identification.” The standard purposely does
not address test quantities so that the method
can be applied to a variety of buildings, from
small multifamily projects to large commercial
condos such as the one shown in Fig. 2.
Given the clear guidelines in ASTM E1105
with respect to the need for the specifying
authority to define testing quantities, a reasonable
question to ask is why confusion persists
among project teams about required testing
quantities. The answer to that question leads to
another commonly specified trade association
document, AAMA 502, Voluntary Specification
for Field Testing of Newly Installed Fenestration
Products.2
AAMA 502 is a specification created and
maintained by the American Architectural
Manufacturers Association (AAMA), a trade
association that is now part of the Fenestration
& Glazing Industry Alliance (FGIA). The
AAMA 502 specification includes requirements
for field air leakage and water penetration
testing of newly installed fenestration
products. For the water penetration testing,
the AAMA specification basically requires
testing per ASTM E1105. Specification writers
frequently include field testing per AAMA
502 because, unlike ASTM E1105, the AAMA
specification includes guidelines for both test
pressures and, more importantly with respect
to this paper, information for testing quantities.
The short-form field-testing section in
AAMA 502 includes the following language,
which is intended to be inserted into a Division
08 specification for windows:
Test three (unless otherwise specified)
of the fenestration product specimens
after the products have been completely
installed for air leakage resistance and
water penetration resistance as specified.
As previously noted, AAMA 502 requires
that water penetration resistance testing should
be performed per ASTM E1105. For any project
where AAMA 502 testing is specified and no
other testing requirements are included, three
water penetration resistance tests should be
performed.
However, the three default tests are not
applicable to every project since the specifier
notes in AAMA 502 include the following language:
“The number of specimens to be selected
for testing on a project should be determined
after careful consideration of the following
factors.” The specifier note then goes on to list
factors including, but not limited, to cost, project
size, complexity, and prior experience with
the fenestrations. In addition to these factors,
the specifier note also includes the following
language, which is both helpful and confusing:
“ASTM E122 provides guidance on how to
establish the number of test specimens in order
to estimate a measure of quality of a production
lot with prescribed precision.” This language is
helpful to specification writers because when
questions arise about test quantities, the writer
can point back to AAMA 502 and state that test
quantities can be determined per ASTM E122.3
At the same time, the language is confusing
because now another ASTM standard must be
reviewed and understood to establish the number
of ASTM E1105 tests required for a project.
ASTM E122 is where the test quantification
process gets even more muddled.
ASTM E122 GUIDELINES
ASTM E122-17, Standard Practice
for Calculating Sample Size to Estimate,
with Specified Precision, the Average for a
Characteristic of a Lot or Process, includes multiple
methods for determining how large a
sample size needs to be to provide a precise
estimate of quality for all units in a lot of material
or process. The methods in ASTM E122
are mostly applicable to material properties of
production lots, as evidenced by the examples
in the standard that discuss average transverse
strength of a lot of bricks. However, the standard
does include a section that relates to the
fraction that does not conform to a process, and
this section is how we will attempt to examine
how to calculate the quantity of ASTM E1105
tests required for a project.
February 2022 IIBEC Interface • 19
Figure 2. Commercial building that could require more water penetration resistance testing.
Given the complexity of calculating sample
size with ASTM E122, we must first frame and
define the language and terminology in ASTM
E122 in terms of fenestration installation and
ASTM E1105 testing. The terms and symbols
in Eq. (1) will be needed to calculate a sample
size for an example building. Each term or
symbol is first defined by the general definition
provided in ASTM E122 and then discussed
in terms of ASTM E1105 testing for a new
construction project. Note that the equation is
taken directly from ASTM E122.
Equation (1) will be critical for calculating
the number of ASTM E1105 tests required,
and the calculation process and challenges are
best illustrated and explained with a practical
example.
ASTM E122 EXAMPLE CALCULATION
Before we provide any information for our
example new construction project, we should
note that Eq. (1) is not actually affected by our
example building. If E is prescribed and previous
data are available that allow for a good
estimate of the window failure rate p0 when
subjected to ASTM E1105 testing, the sample
size or number of windows that should be
tested can be readily calculated. Realistically,
however, prescribing an E value is difficult,
and a good estimate of p0 is likely not available;
therefore, an example building is needed to
understand how we can discuss and assume
these values.
For our example new construction project,
let us say that 500 identical windows will be
installed. This example building is obviously a
simplification when compared to any new construction
process, but simplification is needed if
we are to understand the ASTM E122 calculation
process and demonstrate the assumptions
needed to determine our sample size.
Determining Window Failure Rate p0
The first piece of information that is
required is p0, which, as previously defined, is
an estimate of the fraction of installed windows
on the project that would fail ASTM E1105
testing, or the failure rate of the windows. The
best way to estimate window failure rate is
to examine empirical window failure rates of
similar processes. Unfortunately, such empirical
data are likely not available since reliable
window failure rate calculations would require
that 100% of windows on similar projects were
tested per ASTM E1105. In this author’s experience,
few, if any, projects exist where 100% of
the windows have been tested.
Another approach for estimating failure
rate is to calculate and average the percentage of
ASTM E1105 failures on similar projects where
some, but not all, of the windows were subject
to ASTM E1105 testing. While this approach
seems advantageous, it is not a reliable estimate
of failure rate for at least two reasons: (1) failure
rates of small sample sizes on similar projects
are not necessarily equivalent to failure rates
of the entire lot or process; and (2) data from
previous projects are not likely applicable to our
example project since several factors (including,
but not limited to, flashing methods, crew experience,
and nature of installation) are not the
same from one project to the next.
Given the limitations of methods for estimating
p0 from empirical data, the author suggests
an alternate strategy to define this value.
This strategy involves defining an acceptable
failure rate of the windows for our building
in terms of what the project team is willing to
accept. For this example, let us say that the project
team is willing to accept a failure rate of 4%,
meaning 20 windows could fail ASTM E1105
testing. Such a strategy is obviously problematic
since most project teams will likely demand a
0% failure rate; however, any demand for zero
failures would necessitate testing and repairing
all windows, which is even less likely than
accepting a failure rate greater than 0%.
Prescribing Acceptable Difference E
Like our failure rate process, determining
an acceptable difference between the failure
rate of the sample and the failure rate of the
entire process is not a straightforward process.
ASTM E122 actually requires that
Equation 1.
Equation 2.
n = size of a sample taken from a lot or process
Sample size is the number of installed fenestrations—specifically windows—that will
be subject to ASTM E1105 testing. The lot or process is best described as the total number
of installed windows on a new construction project. Note that for this paper, we will use the
terms “fenestrations” and “windows” interchangeably. And for the example, we will assume
all the windows on the building are the same size. This simplification is not ideal since the
term “fenestration” does apply to more than windows; however, simplification is required
to demonstrate the calculation procedure.
E = maximum acceptable difference between the true average or fraction
nonconforming of the lot or process and the sample average or
fraction nonconforming
This value is the difference, expressed as a decimal, between the fraction or percentage
of all fenestrations on the building that fail or would fail ASTM E1105 testing and the fraction
or percentage of fenestrations in the sample size that fail ASTM E1105 testing. Note
that this value must be prescribed or assumed, which can be difficult and will be discussed
in the example.
pO = advanced estimate of the fraction of a lot or process whose units
have the nonconforming characteristic under investigation
The nonconforming characteristic is ASTM E1105 test failure of any installed fenestration
or window. This value is best described as the fraction or percentage of windows on
the new construction project that would experience ASTM E1105 test failure, or the failure
rate of the installed windows. The fraction nonconforming estimate should be based on
knowledge of similar lots or processes. Estimating this value is difficult, as will be discussed
in the example.
20 • IIBEC Interface February 2022
the approximate precision desired for the estimate must be prescribed. That is, it must be decided what maximum deviation, E, can be tolerated between the estimate to be made from the sample and the result that would be obtained by measuring every unit in the lot or process.
In the author’s opinion, the acceptable precision or maximum deviation value should be provided by the specifying authority. Realistically, the entire reason we are calculating a sample size via ASTM E122 is because the specifying authority is not likely to provide or even understand this value, so we will again need to assume this value.
Returning to our value for p0, we know that the project team accepted a failure rate of 4%, so let us define the acceptable difference E between the sample size failure rate and process sample rate as 3.5%, or 0.035. This E value means that a sample failure rate of anywhere between 0.5% and 7.5% is acceptable and is a reliable indicator of the failure rate of the entire process.
Calculating Sample Size n
Since the values for p0 and E have been established, calculating the sample size, or the number of windows that should be tested per ASTM E1105, is now straightforward as we only need to substitute these values into Eq. (1), as shown in Eq. (2).
The n value represents the number of installed windows that should be tested per ASTM E1105 to ensure that the fraction nonconforming, or failure rate, of the sample is a precise measurement of the fraction nonconforming of the entire process of installed windows on our example building. Since our example building has 500 windows, testing 282 of those windows equates to testing approximately 56% of all windows. Testing this many windows would be a substantial effort for any project team, so the sample size calculation warrants further analysis and discussion.
February 2022 IIBEC Interface • 21
ASTRUTExtruded Aluminum Pipe Support protects roofs.And pipes.l One-piece designl Integrated shallow strut allows use of any standard accessoryl Multiple, custom lengths— cut-to-order from 6″ to 20″ longl 150 lb. load; 5 lb. per square inch compressive strengthInnovative rooftop supports since 1998www.MAPAproducts.com(903) 781-6996The new A-Strut Aluminum Pipe Supports from MAPA elevate pipes above roofs, protecting the roof from abrasion and punctures, and the pipes from deterioration caused by movement and rough roofing material.You’re protected from headaches and expenses.TM®Model MA-6F4Patent Pending
Since databases and statistical education and acceptance do not yet exist, the determination of testing quantities should continue to be a collaborative process where all parties involved discuss, debate, and ultimately settle on a testing regimen.
SAMPLE SIZE DISCUSSION
As shown in our example, using ASTM E122 to calculate a sample size can result in large sample sizes relative to the size of the entire lot or process. The resulting sample size is a function of both the fraction nonconforming and acceptable difference values. Both values can vary based on empirical data, project team requirements, and assumptions, so it is worth noting how variations in these values will affect the resulting sample size, as shown in Fig. 3.
The key pattern in Fig. 3 is that the sample size increases from top to bottom and from right to left. This pattern means that as the fraction of nonconforming windows increases and the acceptable difference decreases, the resulting sample size increases. This relationship between sample size, fraction of nonconforming windows, and acceptable difference is shown graphically in Fig. 4 and 5.
The wide variety in possible sample sizes based on the variability of the fraction of nonconforming windows and acceptable difference values makes it difficult to routinely calculate a sample size for a given project per ASTM E122 guidelines. For comparison’s sake, let us assume that the values for our example building are reasonable for any given project and that testing approximately 56% (282 out of 500) windows is the norm for a new construction project. How would this value compare with typical testing rates for actual projects?
Testing rates vary widely from one project to the next; however, in this author’s experience and based on limited published information, testing anywhere from 1% to 2% of the total number of installed fenestrations is not uncommon. Returning to our example project, testing 1% to 2% would require 5 to 10 tests, and this range is much smaller than testing 282 windows. Again, we should note that testing 282 windows per ASTM E122 is only one of many possible sample sizes, but if a wide variety of sample sizes are possible per ASTM E122, then the ASTM E122 guidelines may be of little value when attempting to implement and specify a testing regimen for any new construction project.
SUMMARY
ASTM E122, which is referenced in AAMA 502 as containing guidelines for establishing the number of test specimens for a project, does not provide simple, repeatable procedures for establishing sample sizes for building enclosure field performance testing. The ASTM E122 guidelines require the use of empirical data that are not readily available for common water penetration testing, such as ASTM E1105 testing. Without this empirical data, the assumptions
Figure 3. Sample size variations with highlighted result from the example project.
Figure 4. Sample size versus acceptable difference at selected fraction nonconforming values.
Figure 5. Sample size versus fraction nonconforming at selected acceptable difference values.
22 • IIBEC Interface February 2022
that are needed to complete the ASTM E122
calculations have too much impact and result
in a wide variation in potential sample sizes for
performance testing.
In lieu of the ASTM E122 guidelines, for
the time being, this author suggests that a reasonable
strategy is to water test 1% to 2% of all
windows installed on a project. The 1% to 2%
guideline may be viewed by the project team
as too much testing. However, if nothing else,
the guideline can be used as a starting point for
project team discussions, and testing quantities
can be adjusted based on the needs and constraints
of the individual project. Returning to
our example one last time, testing 1% to 2% of
the windows would result in 5 to 10 water tests,
which, again in this author’s experience, can be
a reasonable testing quantity.
If ASTM E122 continues to be referenced
as a guideline for determining sample sizes,
databases will need to be established and maintained
that contain results from previous enclosure
testing so these results can be used to
estimate fraction nonconforming values for
future projects. Additionally, the industry will
need to embrace statistical analysis as a means
for calculating sample sizes so that the demand
for zero nonconforming items on a project can
be avoided.
Since databases and statistical education
and acceptance do not yet exist, the determination
of testing quantities should continue to be a
collaborative process where all parties involved
discuss, debate, and ultimately settle on a testing
regimen. The testing regimen should be based
on numerous factors such as budget, product
complexity, building risk profile, and project
team experience. All these factors, among others,
should be weighed and reviewed to arrive at
a testing quantity for the building enclosure that
provides the appropriate level of quality control
and assurance.
REFERENCES
1. ASTM International. 2015. Standard
Test Method for Field Determination of
Water Penetration of Installed Exterior
Windows, Skylights, Doors, and Curtain
Walls, by Uniform or Cyclic Static Air
Pressure Difference. ASTM E1105-
15, West Conshohocken, PA: ASTM
International.
2. American Architectural Manufacturers
Association (AAMA). 2012. Voluntary
Specification for Field Testing of Newly
Installed Fenestration Products. AAMA
502-12. Schaumburg, IL: AAMA.
3. ASTM International. 2017. Standard
Practice for Calculating Sample Size
to Estimate, With Specified Precision,
the Average for a Characteristic
of a Lot or Process. ASTM E122-17.
West Conshohocken, PA: ASTM
International.
February 2022 IIBEC Interface • 23
Jeff Crowe, PE,
has been with
Pie Consulting &
Engineering since
2010. As a member
of and technical
director for Pie’s
Building Science
Group, he is part of
a team that regularly
reviews building
enclosure drawings
and details and frequently
performs building enclosure field testing.
He has provided consultation for architects,
contractors, and owners for appropriate testing
and has also presented on several technical topics
with respect to building enclosure testing. He was
a contributing author to the U.S. Army Corps
of Engineers Air Leakage Test Protocol for
Building Envelopes.
Jeff Crowe, PE
What It Takes to Get Back to Work
As 2022 begins, COVID-19 uncertainty remains, as does the staffing shortage that
the pandemic has exacerbated.
The Building Owners and Managers Association International asked attendees at
one of its recent meetings about what kept them up at night. Leading the field: staffing
concerns, which extend to other industries.
“The talent shortage was the top concern across the board, and it in turn connects
directly to other issues, such as staff burnout, operational efficiency and growth,” John
Salustri wrote in Looking to 2022: Property Professionals Accelerate on Their Positives
(boma.org/DeepDives). “A recent Bisnow report puts the overall staffing shortfall as
impacting 60 to 70 percent of all commercial real estate firms, and certainly all disciplines.”
What can building enclosure professionals do to attract and keep talent? One
senior executive advised moving away from passive hiring techniques, such as posting
job listings, and toward greater emphasis on the promotion of company resources and
training programs.
But the building itself and the working environment could be just as important. “In
an era when that talent can pick and choose where they want to work, the building itself
plays a vital role in attraction and retention,” Salustri wrote. “‘The way to distinguish
a building from its peers is how much more it offers employees,’ said one contributor.
People make employment decisions based, at least in part, on the physical environment.”
S P E C I A L I N T E R E S T
Photo by Chris Gray on Unsplash