Even if test and production environments are closely matched, there are other reasons issues can be missed during testing. The following list contains additional considerations.
-
Custom-developed applications and integrations are often not tested for performance impact, memory stability, or session and license leakage—just correctness.
-
Environmental changes such as applying patches, changing components, and business process changes can cause test coverage to become outdated.
-
Volume of data. Test systems may not have the hardware specifications to replicate the complete production environment. This may prevent you from observing issues due to the size of the database or the quantity of documents being ingested.
-
Velocity of data. Test systems often have limited resources to generate the quantity of simultaneous transactions that will occur in production. For example, testing a Workflow action that runs through a limited set of transactions may prove that all operations can occur, but it may not prove that the software will handle the quantity of transactions that may bombard an entry queue used by hundreds of production users. Sometimes, obscure issues require precise timing of events to occur. A greater load in production increases the likelihood that a specific timing oversight could occur.
-
Number of users. Test environments may have a limited testing team that does not accurately reflect the number of concurrent users of modules in production.
-
Concurrent access. It is difficult to replicate real world concurrency. Issues have occurred that were caused by simultaneous usage of different modules. These issues are difficult to diagnose in production, and occur infrequently.
-
Historical data. Issues are sometimes related to the states or status of transactions already in the system. When a test system doesn't reflect historical data that will interact with transactions, a possible interaction may not be observed during testing.
Testing can greatly reduce risk in an upgrade, particularly when planned well, but even the most thorough testing cannot eliminate all risk. Testing by itself is not enough to sufficiently manage risk.
For information on creating an OnBase test environment, see the Creating an OnBase Test Environment document found in the Resources section of the OnBase Upgrades Technical Community on the Hyland Community site.