Factory Acceptance Testing (FAT) is a term used in engineering, particularly when manufacturing equipment or machinery, to make sure that a product is built and operational in accordance with the appropriate standards before it goes to the customer. I have been responsible for Factory Acceptance Testing at a large multinational oil and gas equipment supplier for a number of years, both from the point of view of the design engineer on multi-million pound projects, and of the Engineering Test Manager responsible for every project which leaves the factory. I would like to share my thoughts on what makes a good Factory Acceptance Test, and how to do it well.
Failure in operation is almost always more damaging than failure during factory acceptance testing. In the Oil & Gas industry a failure of a £5000 product during installation of a £3 billion gas drilling platform can cost installation contractors upwards of £250,000 per day in additional installation vessel costs. Worse, if a failure occurs in service, at the very least a vessel must investigate and correct the problem using ROV at a cost of many hundreds of thousands of pounds. In some cases an entire pipeline many kilometres in length may need to be re-lifted and the offending product replaced. In the worst case an environmental disaster may ensue, costing the operator, the contractors, and the local government many millions of pounds.
The risk to the manufacturer of the faulty product is in lawsuits of negligence if an appropriate FAT was not completed, and so it is vitally important that the plan is agreed from the start.
Factory Acceptance Test Structure: What does a good FAT look like?
A good factory acceptance test is very important to plan up front. If possible, an FAT should be planned alongside the product design, as this way the same calculations and design decisions can be used to determine the parameters of the FAT. Often, however, there are schedule constraints which make this impractical, as well contractual requirements which may prevent work from proceeding on the FAT procedure until the design is approved by the customer. In any event, a well-planned FAT can save time and money for both the manufacturer and the customer, so is vital to get right.
1. The FAT should follow relevant standards
Factory acceptance testing evaluates the performance of the equipment, so it is vital to have a clear plan of what the equipment is intended to do, and what standards it must adhere to. These standards may be global standards, or they may be internal standards. Either way, you should have a specification which outlines the exact requirements of the product, and what this can be assessed against.
2. The FAT should have clear limits
As well as knowing what it is supposed to do, there should be clear limits of acceptability. It is common to have a specification with exact values, or perhaps more general guidelines, such as “it should deflect 185mm at 3 tonnes of bending load”, or “the carriage should be painted red”. The first one is too exact, and needs to have a range of acceptance. Perhaps 182-191mm is appropriate; this would need to be based on technical analysis and have some reason to be set as such. The second is too vague; how red is red – is burgundy acceptable? Is pink acceptable?
3. The FAT should have clear procedures
Thirdly a method of acceptance testing should be developed and agreed upon. This method should be as simple a procedure as will reliably prove the conformance of the product in each test. There may be international standards which describe the method, but usually it is up to the manufacturer to make a proposal and the customer to agree or otherwise.
4. The FAT should recognise non-conformance
Finally a process should be in place to deal with non-conformance. Ideally each test will have criteria which are sensibly placed and non-conformance with these will have specific responses. These responses may be to correct the fault, to adjust a process, to add additional equipment to mitigate the fault, or to deviate from the specification. It is far better to determine the likely responses prior to the test than to justify them later, particularly if there are commercial implications.
5. The FAT should have a clear report format
Once the factory acceptance test is completed, the information needs to be made available to both the customer and the internal quality assurance department. The information in the report must be a complete record of the FAT, including details of the product being tested, the tests completed, the protocols followed, results of the tests, and any discrepancies and how they were dealt with.
6. Witnessing the FAT
In a lot of cases the customer will want to witness the FAT, or send a third-party inspector to watch it. This allows the customer to get feedback sooner as to the results of the FAT, as well as being able to influence the testing. This is sometimes a good thing, as the FAT engineer may not know exactly how the customer intends to use the product and can ask their advice. It can also be a bad thing as this is very conducive to ‘scope-creep’.
Sometimes the customer will ask for additional testing or extra processing of results. While the manufacturer is usually keen to keep a customer happy, this can have unforeseen effects in delaying the FAT completion. It also moves the testing away from the approved test plan, and can result in confusion over what should be completed. It is vital to manage the witness expectations while conducting the FAT.
Factory Acceptance Test Parameters: How to decide what to include in an FAT?
The factory acceptance test plan – sometimes called an Inspection and Test Plan (ITP) – should be based on factual information, usually a combination of the customer’s specification and the manufacturer’s design documentation. The decision of what to include is usually a matter of negotiation
1. Customer Specification
A well-written specification for the product would help greatly in the first draft of the FAT, as it would contain all of the requirements of the product which the customer finds important. This is the primary commercial aim of the FAT; to satisfy the customer that the product is fit for purpose.
Quite often the specification will include properties that the manufacturer does not feel are important the the functionality of the product, such as the colour, any cost-saving devices, or properties which have been included by the customer as a general requirement, but which don’t feature in the design calculations for this specific product. These features will nevertheless need to be checked to ensure that the customer specification is fulfilled.
2. Manufacturer Design
There is another, more important purpose of the factory acceptance testing; to ensure that the product is fit for purpose from the designer’s perspective, and that it will not fail during installation or operation. The manufacturer knows the product better than the customer in most cases, and will have access to data and design calculations that the customer does not. More importantly, the designer will know which aspects of the design are closest to the design envelope, and are at more risk of failure or would have a greater impact on the product if failure did occur.
There is a tendency from customers to require testing which is more than is required, and for manufacturers to propose less than would prove the product. On the one hand the customer may have no prior knowledge of this product except for this project. On the other hand the manufacturer is likely to have experience in the design, and may have already qualified design techniques and manufacturing processes in the past. It is always going to be a negotiation between the two parties, but it is the responsibility of the manufacturer to have the design of the product foremost in their mind.
For example, one product family may require a particular range of testing due to its unique design characteristics resulting in potential failure modes at two extremes of fit, but a similar product family has different characteristics which would only result in failure at one extreme. If the customer has experience of one type but not the other, he is likely to assume that the same testing applies. The manufacturer has the upper hand here in understanding the products and their differences, and should provide a technical explanation of the actual requirements. It should never be acceptable to only partially prove a product is fit just because a customer didn’t request a particular test. Likewise it should never be accepted that a test is performed which has no bearing on the product’s fitness, just because one party is ill-informed as to the design characteristics.
3. DFMEA: Design failure mode and effect analysis
One way of determining the requirements of the factory acceptance test is to conduct a DFMEA, or Design Failure Mode and Effect Analysis. The DFMEA is a collaborative approach to determining all of the potential failure modes of a product and calculating an objective risk / consequence of failure score. Depending on the results of this analysis, some aspects of the design may be changed to remove the failure modes, and some mitigation processes may be put in place to capture failures as early in the process as possible. FAT is one of these mitigations which should already be in place. An effective DFMEA will dictate the risks inherent in the design which need to be tested.
Factory Acceptance Tests: What type of FATs are there?
As well as deciding what parameters need to be assessed, the actual tests which will check these need to be determined. There are a large variety of types of tests which could be chosen, depending on the industry the product is intended for. Factory acceptance testing is not intended to show off the product and demonstrate its capabilities, but rather to prove the parameters decided upon above. The most basic test which will appropriately prove the acceptance should be chosen.
1. Visual inspection
Some aspects of the design or specification are fairly obvious from a visual observation. For example, if the customer requested that the product is red, there is a very simple test to be carried out which can be conducted visually (using a sample colour card with an acceptable range of colours). It may be specified that there are lifting points at two locations – a quick visual scan of the product should confirm that there are two locations with marked lifting points. There may be a requirement for a maximum number of paint defects in a painted surface – these may be checked visually. These may be lumped together into a single test with a number of acceptance criteria to cover each aspect that can be checked visually.
2. Dimensional survey
Dimensional surveys are commonly used to confirm that a fabrication or machined part conforms geometrically with the design. Be careful with these, as not all dimensions are as easy to measure as you may expect. It is normal for a designer to expect that a machining drawing, suitable as it is to program a CAD/CAM machine to produce a part, could also be used to inspect the part afterwards. However, when the sharp corners have been radiused to 5mm, and the datum faces have been ground away, there may be no physical points to place a vernier caliper against!
It is also generally more complex to evaluate the acceptance criteria of a complex component, with a build-up of tolerances on two or more parts which should be evaluated to determine the min and max dimensions which would result in failure of the machine to be assembled or to operate correctly. It is often far simpler and more effective to just carry out a fit-up of the parts in question, perhaps using mock-ups of the interfacing parts at the minimum and maximum dimensions to represent other customer equipment which isn’t available.
3. Mechanical property tests
Material properties are a large set of parameters which can be tested for. Sometimes lab testing on the actual product or (more commonly) on a companion sample is carried out. Sometimes the material properties of a large portion of the product need to be assessed. It is then the effect of the properties which is being tested, and not the properties themselves in most cases. Proof loading is a common test to confirm that the materials are as strong as they are required to be.
4. Output
Output – particularly from a piece of manufacturing machinery which will carry out some form of work to a product – is something that can be assessed. A machine developed to print wallpaper may well be strong enough to stay in one piece, operate perfectly well without jamming, and fulfill all safety requirements, but if it does not keep the repeating pattern at the same distance each time, will be difficult to justify as fit for purpose. In fact, often the end-result of a piece of machinery is more important than any individual aspect of the design. Does it actually matter that the two rollers inside are 595mm long, if the wallpaper being produced is perfectly aligned? If the wallpaper is correct, then it is highly likely that everything inside is correct, and if it is not then the design may be over-conservative.
5. Proof loading
A structural piece of equipment is likely to have to hold a minimum load, whether in bending, tension, compression, or any other manner. It is normal to proof load this type of equipment in each of the load cases to ensure that failures won’t occur in service. A proof load test would normally go slightly over the design limit (remember the design limits will include a healthy safety margin) by probably 10%. If the product can withstand this load then it will survive in service.
6. Pressure Testing
Similar to proof loading, some products need to operate in hazardous environments. One of these is at high pressure, so simple pressure tests can be conducted to ensure that the products are still operational. These can be completed inside hydrostatic pressure vessels, capable of internal high pressures.
Alternatively, the machine may have pneumatic or hydraulic lines which require pressure testing from inside. In this case the system is pressurised to the required design pressure plus a safety factor.
7. Continuity
Wiring looms may be inspected for continuity at the appropriate points. Cathodic protection may be checked for electrical continuity between the anodes and the cathodic materials.
8. Controls
Any control devices on the product should be checked for correct operation. They should also be checked for incorrect operation, and their use in unusual scenarios should be assessed.
For example, a series of five buttons is included on a control panel. Each will begin the next phase of a manufacturing process. Each stage must follow the one before, and so the designer has included electrical circuits which will sense the product progress. These will only allow the button press to start the next stage when the one before is complete. A sensible dual-safety system which should be user-friendly and fool-proof.
A sensible test is to check that each button operates the next stage when the previous one is complete. This should work. A logical next test is to confirm that if the previous stage is not complete, then the button won’t operate. A third test, which may be overlooked, is to use the equipment in a non-logical manner. Perhaps pushing button 5 after button 2 will result in some flawed logic? Perhaps the electrical switch for button 5 doesn’t register that stage 4 hasn’t finished because it hasn’t started yet?
Controls can be a very difficult parameter to fully test, and a lot of time can be spent trying to ‘break’ a complex system. It is very important to ensure that a comprehensive plan of controls testing is written beforehand, as this can greatly save time and resources by finding the most efficient order of testing.
9. Safety
A number of the tests already include safety. Proof load tests are a confirmation that the products won’t physically fail. Visual inspection would be appropriate for ensuring that safety signage is included in the appropriate position and conveys the correct message. Continuity testing may be a check that earths are properly connected. A full safety check of the product should be carried out and a checklist against the appropriate standards completed. The last thing a manufacturer wants is for one of its products to hurt somebody.
10. Documentation
Processes, health and safety information, material datasheets, maintenance schedules, product material traceability data, and a bunch of other information will need to be compiled and supplied with the product. A check of this documentation should be included in the factory acceptance test.
The final plan should include a clear list of tests which can be used to assess the function of the product and the conformance to the customer specification. It should be made up of tests which are practical and useful, and which have clear acceptance criteria. Where possible a simpler test should be used in preference to a more complex one.
Factory Acceptance Test Criteria: Setting Acceptance Limits
It is all too common to become lazy when setting acceptance criteria. Even worse than those tests with no tolerance on acceptance (for example – break load should be 1.8 tonnes – is this a minimum, a maximum, or an exact figure which is required?) are those tests with arbitrary critera. The problem with this is two-fold. Firstly, if the figure is arbitrary (for example, a dimensional survey with a +/-2mm on all dimensions) then who’s to say if a product at the extents of this will be suitable at all? Secondly, the procedure in the event of test failure needs to be considered.
If the process in the event of failure is that first of all a concession from the customer will be sought, then it will need to be proven that the product is fit for purpose. If the original acceptance criteria was too restrictive, and a more open limit is still appropriate, then the manufacturer first needs to convince the customer that their own original plan was too harsh and that a looser set of limits will still work. This is far more difficult than having an appropriately engineered set of limits at the start of the test, which should mean that the test is more easily passed.
For example, a product is intended to hold back a load of 200kg – this is the specification from the customer, and must be kept to. The product is designed with a little more scope; perhaps it was simpler to re-use existing designs or mould tools. It can withstand a minimum of 350kg of load according to the designer. An obvious test parameter is the load limit, and a simple proof load test will confirm this – but the engineer sets the test limits based on the limit of the design, and requires that the product is proof loaded to a minimum of 350 x 1.1 (a small safety margin) – 385kg minimum.
The test is carried out and the product collapses at 376kg. Is this a failure? Yes it is! Now the manufacturer needs to check if this is a problem. In fact, it is not – it failed at almost double the specification. But now the customer is asking questions. “Why did it fail? Is there a problem with the material? How can you know that the other products don’t have the same defects? Why did you design a test which was unsuitable? Prove to us that this won’t happen again.”
The manufacturer is absolutely confident that this isn’t a problem, but has created more problems by over-testing the product. The test should have been conducted at a limit of 200 x 1.1 = 220kg.
The lesson learned? Always give as loose an acceptance limit as you can justify. That means that you must justify every criteria, but that shouldn’t be difficult if you have designed your product correctly.
Non-Conformance: How the Factory Acceptance Test should Respond
A major part of a factory acceptance test is non-conformance and how to deal with it. Any FAT engineer who carries out these tests each day will come across numerous examples of non-conformance, particularly with new designs. A lot of these will be simple to fix, and may be a wire which isn’t connected, or a piece of software which has a bug in a variable name. Some may be a design error, and will need to be referred back up to the designer. There may be major faults in a product which will need a lot of time and effort to resolve. There may even be errors in the FAT process itself, which always muddies the waters between design flaws and test flaws.
The important thing is to have a robust process in place for recording, reporting, and resolving these non-conformances. Every fault should be logged, no matter how small. If a bolt has been missed out of this particular sample product, then perhaps it has been missed out of every product? A record of the faults found will prompt an investigation into other products of the same type. Likewise, the non-conformance needs to be reported to an appropriate person or department immediately. There may well be other concerns such as shipping which needs to be cancelled, or a larger sample of the product which need quarantining.
Resolving the non-conformance should be a matter for a small group of relevant people. Lessons should be learned from every non-conformance. This may be relating to the product design, the manufacture, or the test design. The results of any resolution should be included in the FAT report and communicated to the customer. This promotes open communication and trust in both directions, and allows both parties to understand the full picture.
Factory Acceptance Test Reporting
On conclusion of the factory acceptance test a report should be written and provided to the customer. This should include a full record of all testing which took place, including any non-conformances. If there are non-conformances then a record of the corrective action taken should be included, and a repeat of the FAT for that portion of the test.
It should also include any test equipment certificates, such as calibration certificates. This allows the customer to trace back any measurements to recognised standards, and ensures accuracy of testing. The FAT report is a record of evidence that the product is fit for purpose, and the customer will approve this document before allowing the equipment to be delivered. This document should be retained as a record for the manufacturer as well.
Conclusion
A good factory acceptance test proves the parameters of the product to the satisfaction of both the customer and the manufacturer. It is planned ahead of time, preferably alongside the design. It is clear, and concise, using the simplest practical test for the parameter in question. It has a clear reporting format, and a clear process for non-conformance. If your factory acceptance test meets this description, then it should do its job admirably, and there will be no complaints from your customer or your colleagues.
A good factory acceptance test will save both the manufacturer and the customer time and money in the long term, even if in the short term it finds errors with the product which take a little time to correct.