Tool Requirements: Functional Requirements: Data validation and standardization processes. Feature scaling to Standardize and Normalize data across a variety of datasets from various sources (range of values, gender, age groups, etc.) (L) Record matching algorithms and accuracy expectations. Algorithm must evaluate datasets and match records based on probabilistic methods. Algorithm must learn and predict based on supervised, unsupervised learning models. Algorithm must provide means to measure accuracy of output such as Precision, Recall metrics. Need a mechanism to determine the accuracy of the match, statistical methods used (Example: precision, recall, confusion matrix) (H) Integration capabilities with existing systems. Ability to ingest data from across a variety of data sources (JSON, CSV, Tab delimited, and pipe delimited). Ability to address different file encodings: utf-8, utf-16, and latin-1 (M) Need a configuration driven approach that can adjust and manage acceptable probability match % and filter records that match the criteria. Must support data siphoned to different queues such as manual review, pass, fail, etc. (H)...
