An Inside Look at NTT DATA’s Test Automation Service for Snowflake (TASS)

We know that data accuracy and quality are crucial to making business decisions. During the ETL/ELT process, it’s necessary to keep an eye on data quality. Why? The manual efforts and time required for testing data quality can be very costly. While automation tools exist in the market, moving data from the Snowflake environment to conduct data quality testing has the potential to raise security concerns, especially when it comes to sensitive data.

As your Trusted Global Innovator, NTT DATA understands that data security is a priority for a variety of businesses and industries. To help us put your data first, we built a framework, known as Test Automation Service for Snowflake (TASS), that conducts data testing without moving data out of the Snowflake environment and without the need for manual efforts. Below, figure 1 lays out a high-level block diagram of this framework, the Test Automation Service for Snowflake, which includes two major building blocks—the test automation environment and execution environment. The test automation environment includes user management, test case management, test result dashboards, and a scheduler module. The execution environment includes Snowflake and Snowpark API.

Fig.1. High-level block diagram of Test Automation Service for Snowflake

User Management Module

The user management module within the Test Automation Service for Snowflake is responsible for authenticating users. This module relies on Azure Active Directory for reading credentials to execute test cases on a schedule. Test Case Management Module

The test case management module, pictured below, provides a user interface that allows you to add new a new test case or review any existing test cases. These test cases are stored in a Snowflake environment.

Fig. 2. User Interface for adding new test cases

The following test case types are currently supported:

• Expect column values to be unique
• Expect column values to not be null
• Expect column values to be null
• Expect column value lengths to be between
• Expect column value lengths to equal
• Expect column values to not match regex
• Expect column values to match regex
• Expect column values to match string time format
• Expect column values to be date parseable
• Expect column values to be JSON parseable

If you’re interested in a particular test case that isn’t listed, reach out to us! We are always looking to add test cases quickly according to customers’ needs:

Scheduler Module

The scheduler module is responsible for the execution of test cases on a predefined schedule .

Dashboards Module

The dashboard module is responsible for displaying a visualization of the test results. This module also provides the ability to explore test results that are based on Test IDs or table names. The test results dashboard shows test results from the last 10 executions. Sometimes, we find it useful to take a deeper dive by looking at the pattern of test results over a period of time and correlating it with deployment time. The test results dashboard shows the last 10 test results in a tabular format, as well as a line chart, as shown below:

The Test Automation Service for Snowflake’s dashboard, and user interface, were designed using Streamlit libraries. The Test Automation Service for Snowflake utilizes Snowpark data frames to execute the queries that are required for the dashboards and for executing Python test libraries. Snowpark helps to push SQL and Python codes to the Snowflake environment to avoid any data movement out of Snowflake.

If you’re concerned about the security of your organization’s data, NTT DATA has you covered. The Test Automation Service for Snowflake is particularly helpful for those looking to maintain the security of data in Snowflake’s platform, and it provides valuable insights to help you make the right decision for your data.

Do you want to learn more about the Test Automation Service for Snowflake (TASS) or see what NTT DATA & Snowflake can help you achieve with your data? Connect with us for more information.