Specifications include, but are not limited to: Integration Testing • Testing of higher-level system components as modules and other solution units are integrated by the development team (e.g., Customer Relationship Management solution). System Testing • Comprehensive testing of the complete integrated system release. Parallel Testing • Uses a common set of inputs to assess the compatibility of outputs from the new system with the outputs from the legacy system. External Interface Testing • Ensures proper function of interfaces with systems owned and managed by external organization outside of NYCERS that relate to the solution (e.g., employer data exchanges). • Includes closely coordinating and collaborating with staff of external organizations to the extent directed by NYCERS. Performance Testing • Determines how the system (including computers, networks, software programs or devices) performs in terms of responsiveness and stability under different workloads primarily to help measure, validate or verify quality attributes such as scalability, reliability, interoperability and resource usage, as well as locate bottlenecks. • Includes load testing and stress testing. • Verifies testing-related SI SLAs will be met. Backup, Failover, Roll-back and Recovery Testing • Ensures the solution meets all requirements for backup, failover, recovery, disaster recovery, and business continuity. • Includes testing the ability to roll-back the solution from its current release to a previous release, and to legacy systems. • Note: NYCERS may conduct backup, failover, roll-back and recovery testing independent of the project as a course of normal business. The vendor will be required to participate in such testing, including helping NYCERS to develop a strategy and plan for execution of the testing and providing feedback on test progress. The vendor may also need to provide staff augmentation during test execution.