로고

• Planning and Design3
  • Risk management plan for AI system and execution of the plan (2)
    • Have you analyzed risk factors that may arise throughout the life cycle of the AI system? (1)
      • Have you identified the risk factors of the AI system and the ripple effect?
    • Have you prepared measures to remove and prevent risk factors or mitigate the effects? (1)
      • Have you developed measures to remove risk factors and confirmed if the ripple effects were mitigated?
    Organization of an AI governance system (4)
    • Have you established guidelines and policies on AI governance? (1)
      • Have you prepared internal guidelines and policies on AI governance?
    • Have you formed an AI governance group and reviewed the composition of the group? (2)
      • Is the AI governance group composed of adequately trained members?
      • Have you formed an AI governance group?
    • Is the AI governance being supervised to ensure proper implementation? (1)
      • Is compliance with internal guidelines and policies on AI governance being overseen?
    • Has the AI governance group reviewed the differences between the new and previous systems? (1)
      • Have you analyzed if the system can be implemented by improving, integrating, or abolishing other infrequently used systems?
    Development of a plan to test trustworthiness in the AI system (2)
    • Have you designed a test environment in consideration of the AI system’s features? (2)
      • Have you considered the operating environment of the AI system when determining the test environment?
      • Have you obtained a simulator if the AI system needs a virtual test environment?
    • Have you organized a negotiation system to design the test for the AI system? (2)
      • Have you organized a negotiation system to determine the expected output of the AI system?
      • Have you organized a user review group to check if the AI system is explainable and interpretable?

• Data Collection and Processing3
  • Provision of detailed information for data utilization (2)
    • Is there detailed information to support the accurate comprehension and utilization of data? (4)
      • Have you explained the data attributes before and after cleansing?
      • Have you sorted data into training data and metadata and is there a specification document for each of them?
      • Have you explained the reason for selecting the protected attributes and whether they were reflected?
      • Were data labelers trained and have you provided them with work instructions?
    • Is data provenance documented and managed? (2)
      • Is the dataset provided by a trustworthy provenance?
      • Have you clearly stated the provenance when using an open-source dataset?
    Inspection of abnormal data to ensure data robustness (2)
    • Have you inspected the detection of abnormal data and their normality? (2)
      • Have you checked any possible errors by visualizing the overall training data distribution?
      • Have you implemented techniques to detect outliers in training data?
    • Have you devised measures to defend against data-oriented attacks? (1)
      • Have you prepared measures to defend against poisoning and evasion attacks?
    Removal of bias in the collected and processed training data (4)
    • Have you prepared measures to mitigate bias due to human and physical elements in data collection? (3)
      • Have you implemented procedural and technical measures to eliminate human bias?
      • Have you used a heterogeneous device to ensure data diversity?
      • Have you examined bias in data that may occur due to hardware?
    • Have you analyzed features used in training and prepared selection criteria? (3)
      • Have you made a thorough analysis when selecting the protected attributes?
      • Have you mitigated the impact of features that may create bias?
      • Have you reviewed whether features were removed excessively during data pre-processing?
    • Have you checked and prevented potential biases in data labeling? (3)
      • Have you clearly established the data labeling standards and provided them to labelers?
      • Have you made an effort to recruit diverse data labelers?
      • Have you made an effort to recruit diverse reviewers for labeled data?
    • Have you conducted sampling to prevent bias in data? (1)
      • Have you implemented a sampling method to prevent bias?

• AI Model Development4
  • Ensuring security and compatibility of the open-source library (2)
    • Have you confirmed the stability of the open-source library? (1)
      • Have you used an active open-source library?
    • Are you managing the risk factors of the open-source library? (2)
      • Have you fulfilled the license compliance of the open-source library in use?
      • Have you confirmed the compatibility and vulnerability of the open-source library in use?
    Removal of bias in the AI model (1)
    • Have you implemented techniques to remove bias in the AI model? (2)
      • Have you chosen a bias removal technique appropriate to the model to be developed?
      • Have you selected quantitative indicators to evaluate and monitor bias and are you managing them?
    Establishment of defensive measures for AI model attacks (2)
    • Do you have a defense technique in place against model extraction attacks? (1)
      • Have you implemented a defense technique to prepare for model extraction attacks?
    • Do you have a defense technique in place against model evasion attacks? (1)
      • Have you implemented a defense technique to prepare for model evasion attacks?
    Explanation of AI model specifications and the inference results (3)
    • Do you provide evidence for users to accept the generation process of the model’s inference results? (2)
      • If XAI is not applicable, have you prepared measures other than the application of the technique?
      • If XAI is applicable, have you reviewed the application of the technique to explain the inference results of the AI model?
    • Have you transparently provided the specification of the model on the AI model specification document? (1)
      • Have you prepared a document that describes the details of the system development process and model operation method?
    • When needed, do you provide an explanation about the inference results of the AI model? (2)
      • Have you reviewed whether an explanation of the model’s inference result is needed?
      • Have you provided an explanation to users about the inference results of the AI model?

• System Implementation3
  • Removal of potential bias in the implementation of the AI system (1)
    • Have you made an effort to remove bias due to source code and user interface? (2)
      • Have you examined the possibility of bias in the source code, such as the implementation process of the data access method?
      • Have you examined bias due to the user interface and interaction method?
    Safe mode of AI system and establishment of a process for notification of problems (2)
    • Have you implemented a safe mode that can respond to problems such as attacks, low performance, and social issues? (4)
      • Have you prepared an exception handling policy for such problems?
      • Have you implemented a security technique to reinforce the security of the AI system?
      • Have you considered human intervention if there is a significant ripple effect and high uncertainty due to the AI system’s decision-making?
      • Are guidance and action on handling expected user error provided?
    • Does the system perform the function of alerting the operator if a problem occurs in the AI system? (2)
      • Have you established a notification process for ethical issues such as prejudice and discrimination?
      • Have you established a notification process by developing procedures and indicators to evaluate the low system performance?
    Improvement of users’ comprehension of the explanation of the AI system (2)
    • Have you analyzed user characteristics and constraints in the AI system? (1)
      • Have you analyzed specific considerations according to the user characteristics?
    • Have you provided a thorough explanation based on user characteristics? (5)
      • Have you established criteria for the evaluation of explanation according to user characteristics?
      • Have you refrained from using technical terms that are difficult for users to understand?
      • Have you used accurate expressions to lead users to specific behaviors and comprehension?
      • Are the location and timing where an explanation is needed appropriate?
      • Have you utilized various user survey techniques to evaluate user experience?

• Operation and Monitoring2
  • Ensuring traceability and modification history of the AI system (2)
    • Have you established measures to track the AI system’s decision-making? (3)
      • Have you developed measures to track the contribution to the AI system’s decision-making?
      • Have you put in place the log collection function to track the AI system’s decision-making?
      • Do you collect and manage user logs to continuously monitor user experience?
    • Have you obtained the modification history of training data and managed the impact of data modifications? (5)
      • Have you prepared measures to track the data flow and lineage?
      • Have you developed measures to monitor modifications in the data source?
      • Have you managed the versions during data change?
      • Do you provide information to stakeholders when data change?
      • When new data have been collected, do you reevaluate the performance of the AI model?
    Explanation about the scope of services provided and the subject of interactions (2)
    • Do you provide an explanation to encourage proper usage of the AI service? (2)
      • Do you provide an explanation about the goal and objective of the AI service?
      • Do you provide an explanation about the limitation and scope of the AI service?
    • Do you accurately explain the subject of the interaction? (1)
      • Have you accurately explained to users that they are interacting with the AI?