On January 17, 2024, the New York State Department of Financial Services (“NYDFS”) issued a proposed circular letter for comment regarding the “Use of Artificial Intelligence Systems and External Consumer Data and Information Sources in Insurance Underwriting and Pricing” (the “Circular Letter”). The Circular Letter details NYDFS’ expectations and guidelines for the use of artificial intelligence systems (“AIS”) and external consumer data and information sources (“ECDIS”) by “all insurers authorized to write insurance in New York State, licensed fraternal benefit societies, and the New York State Insurance Fund (collectively, “Insurers”). While NYDFS notes that AIS and ECDIS can provide certain benefits to consumers and Insurers, they flag the unique risks – “the self-learning behavior of AIS increases the risks of inaccurate, arbitrary, capricious, or unfairly discriminatory outcomes,” and that those outcomes can disproportionately affect marginalized or vulnerable communities and individuals.
The Circular Letter tracks many of the broader themes taking shape in AI regulation and governance in financial services and across industries generally. Right at its outset, the Circular Letter emphasizes it is “critical” for Insurers that use AI to “establish a proper governance and risk management framework.” This reflects a growing trend of U.S. regulators (discussed in our prior advisories) to focus regulatory approaches on governance and risk management. As a fairly recent example, the National Institutes of Standards and Technology (NIST) finalized its AI Risk Management Framework. NIST’s framework, like the Circular Letter, requires governance, risk identification, and ongoing risk management in AI development and deployment.
More specific to AI regulation in the insurance industry, the Circular Letter tracks key themes of the NAIC Model Bulletin: Use of Artificial Intelligence Systems by Insurers, and recent Colorado regulations on the use of algorithms and predictive models by life insurers, focusing on three key principles – fairness, governance and risk management, and transparency.
NYDFS is accepting comments, which should be submitted to innovation@dfs.ny.gov, through March 17, 2024.
Fairness: The Circular Letter sets out core fairness principles that Insurers must adhere to if they use AIS or ECDIS. Insurers are already subject to legal and regulatory requirements to ensure they are not unfairly discriminating against protected classes. Accordingly, NYDFS expects Insurers to first validate and be able to demonstrate that the ECDIS tools they use are “supported by generally accepted actuarial standards of practice.” This includes being based on “actual or reasonably anticipated experience, including, but not limited to, statistical studies, predictive modeling, and risk assessments.”
In addition to showing that the ECDIS used are supported by generally accepted actuarial standards, Insurers must be able to demonstrate that the ECDIS are not prohibited by the New York Insurance Law or associated regulations, and that they do not serve as a proxy for any protected classes. NYDFS’ “must be able to demonstrate” language is similar to accountability concepts in other privacy and consumer protection regulations (as set out in the FTC’s blog post on AI, where accountability is a core principle) and the NAIC model bulletin. It suggests that NYDFS may see Insurers as bearing the burden of proactively putting together documentation that ECDIS tools are not discriminatory as a condition of using them, as opposed to a regulator needing to make a case that they are discriminatory.
Further, NYDFS emphasizes the importance of validation and testing, specifically of ECDIS, which may come from sources that are not subject to regulatory oversight. This can heighten the risk that ECDIS may contain or reflect biases and discrimination that will affect the output of any AIS that uses the data. Accordingly, the Circular Letter states that Insurers may not rely on vendor’s claims of non-discrimination, and instead must conduct their own validation and testing. As part of this validation and testing, Insurers should ensure that the ECDIS used does not collect or use criteria that are prohibited under anti-discrimination or unfair trade practices law. This includes not using ECDIS or AIS to collect data that the Insurer would be prohibited from collecting directly.
The letter also states that Insurers should conduct a comprehensive assessment before using ECDIS or AIS, to ensure that they are not unfairly or unlawfully discriminatory. This assessment should include:
- Assessing whether the use of ECDIS or AIS produces disproportionate adverse effects on a protected class or similarly situated insureds;
- If there is a prima facie showing of disproportionate adverse effects, the Insurer must further assess whether there is a legitimate, lawful, and fair rationale for the differential effect; and
- If there is a legitimate, lawful, and fair rationale for the differential effect, the Insurer must conduct and document a search for a less discriminatory alternative that would meet the Insurer’s reasonable business needs.
This testing should be conducted prior to implementation and regularly thereafter, and the procedures and results of this testing should be documented. The testing should also include both qualitative statistical metrics as well as a quantitative analysis of unfair or unlawful discrimination.
Potentially helpfully, the Circular Letter does not merely tell companies to ‘conduct quantitative testing’ – it instead provides prescriptive guidance, listing qualitative testing methods that NYDFS may consider acceptable for compliance purposes. These include statistical methods such as “adverse impact ratio,” “standardized mean differences,” and “drivers of disparity.”
Governance and Risk Management: The Circular Letter mandates that companies establish a corporate governance framework that starts at the top, with the board of directors and flows down to senior management. The board (or other governing body) shall oversee the Insurer’s use of ECDIS and AIS and monitor the Insurer’s risk appetite. While the board or governing body may delegate certain oversight responsibilities, quarterly reports on the material activities and risks associated with the Insurer’s use of ECDIS and AIS to the board is still required.
Senior management is generally responsible for the daily implementation and management of ECDIS and AIS, ensuring it is consistent with the strategic vision and risk appetite. Senior management must ensure that the Insurer establishes written policies and procedures, documenting roles and responsibilities and training for relevant personnel. Senior management shall also ensure qualified staff are assigned to manage ECDIS and AIS, engage with internal audits to review their findings, and ensure prompt remediation of any findings as necessary. Importantly, the Insurer must develop a set of policies, procedures, and standards if it intends to use ECDIS or AIS from a third-party vendor.
Transparency: Lastly, the Circular Letter emphasizes the importance of transparency. The failure to “adequately disclose to the insured or potential insured any other specific reason or reasons for refusal, limitation, or rate differential may be deemed to be an unfair or deceptive act and practice in the conduct of the business of insurance and may be deemed to be a trade practice constituting a “determined violation” of Insurance Law.
NYDFS makes clear that its transparency requirements apply not just to final decisions about whether an applicant is approved for coverage, or the level of premiums an applicant will have to pay. They also apply to upstream AI decisions about whether an applicant can apply for insurance via an “accelerated process,” or whether the applicant will need to submit to “the traditional underwriting process.” In NYDFS’ words, “if the accelerated process determines that an applicant will not be approved for insurance under the accelerated process and can only obtain insurance by submitting to the traditional underwriting process, the applicant has the right to know why.” The Circular Letter describes this as a “clarification” of prior NYDFS circulars. The insurance industry may view it as an expansion of transparency requirements.
The Circular Letter notes that Insurers may not rely on the proprietary nature of a third-party vendor’s algorithmic process to justify a lack of specificity in a required disclosure, and the failure to adequately disclose both the material elements of an AIS and the ECDIS upon which it relies may constitute an unfair trade practice under Insurance Law Article 24.
This largely aligns with the Consumer Financial Protection Board’s (CFPB) guidance on complying with the Fair Credit Reporting Act (FCRA) when using AI. The CFPB also emphasized that FCRA’s requirement to provide specific reasons for an adverse action remains even if financial services companies use AI for credit decisions, and that it will not accept the “black box” nature of an AI tool as a reason for not providing specific reasons.
AI Auditing: While all governance approaches emphasize ongoing monitoring of AI tools, NYDFS is uniquely specific in noting that it expects Insurer’s internal audit functions to bring AI tools into their auditing remit. Per NYDFS, New York insurance laws already “require an insurer to have an internal audit function to provide general and specific audits, reviews, and tests necessary to … evaluate compliance with policies and regulations.” Thus, Insurers should “ensure the internal audit function is appropriately engaged with the Insurer’s use of ECDIS and AIS” as part of overall risk management.
The Circular Letter goes further and identifies specific audit objectives that, if applicable to an Insurer, it may expect to see fulfilled. These include:
- Assessing supporting operational systems and evaluating the accuracy, reliability, and integrity of ECDIS and other data used by AIS;
- Assessing potential biases in the ECDIS or other data that may result in unfair or unlawful discrimination against insureds or potential insureds; and
- Assessing whether there is sufficient reporting to the board.
We will continue to track and report on the final version of the Circular Letter, along with other federal and state regulations on the use of AI in the insurance sector.
