NS Structured Testing
According to Normalized Systems Theory, it is imperative to avoid the multiplicative explosion of combinations for any task that needs to be performed or artifact that needs to be maintained. This of course, is very relevant for software testing. Suppose that you aggregate or sequence 3 consecutive tasks, and want to distinguish and test 10 possible scenarios for every task. In case you want to emulate and test all possible scenarios on the level of the aggregated or complete flow, you have to consider 1000 scenarios. In order to avoid this multiplicative combination of scenarios, one needs to adopt an hierarchical testing strategy. This implies a clear separation of the three tasks, and a thorough testing of the 10 scenarios on each task. On the aggregation level, one needs to identify a number of representative aggregation scenarios with corresponding test data. Suppose a number of 10 aggregation scenarios as well, this would correspond to 40 test scenarios in total.
It is important to realize that the multiplicative combination of scenarios will, in reality, not be limited to 1000, but will easily become unbounded. Indeed, the number of artifacts or tasks, will in general be much larger, and serves as the power to which the number of scenarios is elevated. This means that a complete or nearly complete test coverage without a hierarchical test strategy is simply impossible. Therefore, thorough testing has to be based on an hierarchical strategy, on some kind of testing pyramid. Now, the Normalized Systems software approach provides such a hierarchical structure, and should be leveraged to create such a systematic hierarchical testing pyramid, based on the NS structure and its elements.
The same style of reasoning should be applied to the generated or expanded code. Though it would be possible to generate standard testing code for most of the generated code, such as the CRUDS operations of the various data elements, this would lead to a huge amount of testing code. Therefore, the hierarchical testing strategy should be extended to include the expanded code. For every version of the NS expanders, extensive testing should be performed for all basic operations of the various elements, for all possible data types and all supported technology frameworks. This requires the creation of a structured set of test data, featuring elements containing all possible data types and settings selecting all supported technologies. However, such a structured approach would suppress the need for testing all basic operations on all elements of the expanded code.
Within the NS development teams, a number of standardized test classes have been and are being developed. This is work in progress, but is included here: not as a reference guide, but as a collaboration platform.