A multidisciplinary group of judges recommended winners after evaluating submissions based on the following equally weighted criteria. Entrants also considered and addressed target performance metrics in their submissions.
Evaluation criteria
Phase 1 submissions were evaluated by judges against the six official evaluation criteria described below.
Phase 1 evaluation criteria
Analytical performance
The degree to which the proposed solution indicates potential to accurately and reliably detect low blood lead levels at the point of care. The degree to which the proposed solution includes a clear and evidence-based explanation for the scientific principle(s) underlying the solution, as well as specific and well-supported quantitative targets for analytical performance.
Error mitigation
The degree to which the proposed solution comprehensively identifies potential pre-analytical, analytical, and post-analytical sources of error and how the solution design mitigates the most significant error sources (e.g., contamination). The degree to which the proposed solution includes credible quantitative targets for accuracy and replicability, and actionable strategies to reduce overall error in future development.
User-centered design
The degree to which the proposed solution demonstrates a clear understanding of user needs, given identified point-of-care design target(s) and associated clinical workflows and environments. The degree to which the proposed solution includes user-centered rationale for design decisions to date, rationale for solution performance in the context of user needs (and considering overall challenge target performance metrics), and plans to engage clinical and patient stakeholders in future development.
Accessibility
The degree to which the proposed solution shows potential for broad adoption to increase rates of testing and equity of access to testing. The degree to which the proposed solution includes clear plans for addressing affordability, scalability, and other factors that would enable implementation across a wide range of geographic and socioeconomic target populations — particularly children who are currently underserved.
Development plan
The degree to which the submission describes an ambitious but achievable development plan for Phase 2 of the challenge and beyond, including a clear plan for prototyping, iteration, testing/validation, and evaluation of the test; reasonable estimates of budget requirements and resourcing opportunities beyond the challenge; and consideration of potential risks to deliver on that plan.
Team
The degree to which the team demonstrates relevant expertise, recognizes gaps, and proposes approaches to mitigate gaps to further develop the proposed solution.
Phase 2 submissions were evaluated by judges against the six official evaluation criteria described below.
Phase 2 evaluation criteria
Analytical performance
The degree to which the submission demonstrates through laboratory-based proof of principle, or other testing of critical technical assumptions, that the solution is able to accurately and reliably detect low blood lead levels using non-invasive or minimally invasive sample collection. The degree to which the submission includes evidence that the solution is currently able, or will be able in the near term, to achieve the target performance metrics’ operating parameters in point-of-care settings.
Error mitigation
The degree to which the submission demonstrates how the solution design will mitigate the most significant sources of pre-analytical, analytical, and post-analytical error (e.g., contamination, use of quality control checks, electronic transfer of test results with an associated sample ID) in point-of-care settings and is “fail-safe.” The degree to which the submission demonstrates measurement accuracy and operates across the reportable range, as defined in the target performance metrics, and articulates actionable strategies to reduce overall sample collection error in current or future development.
User-centered design
The degree to which the submission demonstrates the solution addresses user needs and usability by healthcare personnel in a CLIA-waived setting (i.e., without training or use of moderate or high complexity laboratory skills or equipment), as specified in the target performance metrics. The degree to which the submission includes user-centered rationale for design decisions to date, rationale for solution performance in the context of user needs, and a description of clinical and patient stakeholder engagement throughout current and planned development.
Accessibility
The degree to which the submission shows feasibility for broad solution adoption across a wide range of geographic and socioeconomic target populations — particularly caregivers and children who are currently underserved. The degree to which the submission demonstrates the solution may achieve target cost, analysis time, and reporting time, as specified in the target performance metrics, that would enable broad implementation.
Development plan
The degree to which the submission describes an ambitious but achievable plan for continued development, addressing each of the target performance metrics and including a clear plan for prototype refinement, testing and validation, and evaluation of the solution; comprehensive consideration of regulatory requirements to achieve a “CLIA-waived” status; reasonable estimates of budget requirements and resourcing needs beyond the challenge; and consideration of potential risks to deliver on that plan.
Team
The degree to which the team demonstrates relevant expertise, has recognized and is taking steps to fill gaps, and proposes approaches to mitigate remaining gaps throughout further solution development.
Target performance metrics
In Phase 1, the target performance metrics served as guidance for what would be required of a future, hypothetical FDA-cleared and CLIA-waived product. In Phase 2, judges considered the following metrics when evaluating submissions. Entrants addressed how their solutions meet, or will meet, each of these metrics. Alternatively, entrants could articulate in their submissions why different performance metrics are more appropriate for their specific solutions.
Operating parameters
Lower reporting limit
Solutions should seek a lower reporting limit (i.e., limit of quantitation) at or below 1.5 μg/dL with an accuracy of ±2 ug/dL (i.e., total error ≤ 2 ug/dL). Limit of quantitation and limit of detection claims should be demonstrated using standardized approaches such as described by the Clinical Laboratory Standards Institute document EP17-A2.
Reportable range
Solutions should demonstrate the potential to test blood lead concentrations spanning the reportable range that includes important decision points of 3.5 μg/dL, 20 μg/dL, and 45 μg/dL. Precision should be adequate to support the claimed measurement range.
Measurement accuracy
Solutions should indicate the ability to achieve accuracy within ±2 μg/dL or ±10%, whichever is greater, of the true blood lead concentration for 80% of samples analyzed across the reportable range.
Reporting time
Solutions should indicate the feasibility of the method to collect, process, and deliver test results in not more than 15 minutes.
Result reporting
Solutions should be capable of electronically displaying a quantitative result and electronically transferring the test result along with the associated sample ID to an electronic device for incorporation into electronic health information systems (e.g., systems that use HL7 interoperability standards) without the need for manual transcription.
Sample collection
Solutions should minimize the required sample volume, using no more than 150 µL, and must use a capillary blood draw that includes steps to minimize contamination.
Development parameters
Ease of use
Solutions should be usable by non-laboratory personnel without training, meeting the FDA criteria of waived tests as defined in 42 CFR § 493.15b.
Cost
Solutions should indicate consideration of manufacturing and per-test costs required for broad adoption.