Skip to main content

Quality Monitoring

Build and manage data quality rules to monitor data integrity, completeness, and accuracy across your database tables.

VIDEO TUTORIAL⏱️ 3 mins

📹 Building Quality Rules

Learn how to create data quality rules manually and automatically using AI to monitor your data quality.

Overview​

The Data Quality Agent enables you to define and manage rules that validate data quality across your connected databases. Rules can be created manually with specific parameters or automatically generated by AI based on your data's metadata and profiling.

Accessing Data Quality​

Dashboard Widget​

The Data Quality widget on the dashboard provides a quick overview of:

  • Overall Quality: Percentage score (e.g., 73%)
  • Rules Status: Number of passed and failed rules
  • Quality Dimensions: Scores for:
    • Completeness
    • Accuracy
    • Consistency
    • Timeliness

Data Quality Agent View​

Access the full Data Quality Agent from:

  1. Navigate to your data source
  2. Click on Data Quality in the agent list
  3. Or click View details from the dashboard widget

The agent view has three tabs:

  • Dashboard: Quality scores and trends
  • Rules: Manage and create quality rules
  • Results: View validation results

Creating Quality Rules​

Method 1: Manual Rule Creation​

Manually attach quality rules through a 3-step wizard:

  1. Click "Attach New Rule" button in the Rules tab
  2. Step 1 - Select Table:
    • Choose database
    • Choose schema
    • Select table
  3. Step 2 - Select Rule:
    • Search for rules by name
    • Filter by dimension (dropdown)
    • Select from available rule cards
  4. Step 3 - Configure Parameters:
    • Set rule-specific parameters
    • Configure columns and thresholds
    • Click "Attach Rule" to save

Method 2: AI-Generated Rules​

Let the AI agent automatically generate rules based on your data:

  1. Click "Build Rules" button in the Rules tab
  2. The agent analyzes:
    • Table metadata and structure
    • Data profiling results
    • Column data types and patterns
  3. Rules are created as AI suggestions
  4. Rules are created with status "Pending Review"
  5. Click "Activate All" to activate pending AI suggestions

Rule Categories​

Rules are organized into seven categories for better management:

1. Data Integrity​

  • Purpose: Ensure primary keys, foreign keys, and uniqueness constraints
  • Examples:
    • Column values must be unique
    • Compound columns uniqueness
    • Valid UUID formats
  • Icon: 🔑

2. Completeness​

  • Purpose: Verify required fields and non-null constraints
  • Examples:
    • Column values not null
    • Column values not null or empty
    • Required field validation
  • Icon: ✅

3. Format Validation​

  • Purpose: Validate data formats and patterns
  • Examples:
    • Currency format (e.g., USD, GBP, EUR)
    • Account number formats
    • Date formats
    • Email/phone validation
    • Regular expression patterns
  • Icon: 📝

4. Value Constraints​

  • Purpose: Check value ranges and specific constraints
  • Examples:
    • Values between min/max range
    • Values in specific set
    • Column length constraints
    • Statistical thresholds
  • Icon: 🎯

5. Business Rules​

  • Purpose: Domain-specific logic validations
  • Examples:
    • No future dates for historical data
    • Cross-field validations
    • Custom business logic
  • Icon: 💼

6. Consistency​

  • Purpose: Cross-table and referential integrity
  • Examples:
    • Cross-table consistency checks
    • Referential integrity beyond foreign keys
    • Value consistency across systems
  • Icon: 🔄

7. Performance​

  • Purpose: Monitor table size and growth patterns
  • Examples:
    • Table row count monitoring
    • Growth pattern analysis
    • Performance metric tracking
  • Icon: 📊

Available Rule Types​

Common Rule Types​

Rule TypeDescriptionParameters
Column ExistsVerify column exists in tableColumn name
Not NullValues must not be nullColumn name
Unique ValuesAll values must be uniqueColumn name
Value RangeValues within min/max rangeColumn, Min, Max
Match RegexValues match patternColumn, Pattern
In SetValues in predefined setColumn, Value list
Type CheckValues match data typeColumn, Data type
Length CheckString length constraintsColumn, Min/Max length

Advanced Rule Types​

Rule TypeDescriptionUse Case
Cross-Column ValidationCompare values between columnsStart date < End date
Referential IntegrityValues exist in reference tableForeign key validation
Statistical RulesMean/median/stdev thresholdsOutlier detection
Business RulesDomain-specific logicCustom validations

Rule Configuration​

Regular Expression Example​

For currency format validation:

^[A-Z]{3}$

This validates 3-letter currency codes like USD, EUR, GBP.

Range Configuration​

For numeric value constraints:

  • Minimum: Lower bound (inclusive)
  • Maximum: Upper bound (inclusive)
  • Allow Null: Whether null values pass validation

Viewing Rules​

Hierarchical View (Default)​

Rules are displayed in a tree structure organized by table and category:

├── Table Name
│   ├── Value Constraints
│   │   ├── Balance Range Check
│   │   └── Account Length Validation
│   ├── Format Validations
│   │   ├── Currency Format
│   │   └── Account Number Pattern
│   ├── Completeness
│   │   └── Required Fields Check
│   └── Data Integrity
│       └── Account ID Uniqueness

Each category shows:

  • Number of rules
  • Number of enabled rules
  • Columns being checked

Flat List View​

Switch to a flat list to see all rules at once with:

  • Rule name and description
  • Table and column assignments
  • Enabled/disabled status
  • Confidence score (for AI-generated rules)
  • Quick actions menu

Rule Management​

Enable/Disable Rules​

  • Toggle rules on/off without deletion
  • Bulk enable/disable by category
  • Temporarily disable for maintenance

Edit Rules​

  • Modify rule parameters
  • Change severity levels
  • Update descriptions

Delete Rules​

  • Remove individual rules
  • Bulk delete by selection
  • Clear all rules for fresh start

Rule Status Indicators​

For Manual Rules:​

  • Active: Rule is enabled and running
  • Inactive: Rule is disabled

For AI-Generated Rules:​

  • Pending Review: Blue badge - Awaiting activation
  • Active: Green badge - Approved and running
  • Rejected: Red badge - Dismissed by user

Executing Quality Rules​

Running Validation​

Quality validation can be triggered from:

From Dashboard Widget:​

  • Click "Run quality analysis" link when no data exists

From Agent View:​

  • Quality checks run automatically when rules are built
  • Validation runs are stored and viewable in the Results tab

Understanding Validation Runs​

  • Each run processes all enabled rules
  • Results show passed/failed rule counts
  • Quality scores are calculated for dimensions

Viewing Results​

The Results tab contains:

Left Panel - Validation Runs List:​

  • Time range selector (24h, 7d, 30d)
  • List of validation runs with scores
  • Auto-selects latest run

Right Panel - Run Details:​

  • Overview Tab: Summary metrics and scores
  • Rule Results Tab:
    • Individual rule execution results
    • Pass/Warning/Failed status
    • Click failures to see sample data
  • Issues Tab: Quality issues detected

Best Practices​

Rule Creation Strategy​

  1. Start with Build Rules: Let AI create baseline rules
  2. Review AI Suggestions: Check suggested rules before activating
  3. Activate Selectively: Enable rules that match your requirements
  4. Add Custom Rules: Use "Attach Rule" for specific validations

Maintenance​

  • Regular Reviews: Audit rules quarterly
  • Update Thresholds: Adjust ranges as data evolves
  • Monitor False Positives: Fine-tune overly strict rules
  • Document Custom Rules: Add descriptions for complex logic

Performance Considerations​

  • Limit Active Rules: Too many rules impact scan performance
  • Optimize Complex Patterns: Simplify regex where possible
  • Schedule Wisely: Run comprehensive checks during off-peak hours

Integration with Other Features​

Data Profiling​

  • Rules can be generated based on profiling insights
  • Statistical rules use profiling metrics
  • Anomaly detection from historical profiles

Quality Dashboard​

  • Rule results feed into quality scores
  • Trend analysis shows improvement over time
  • Drill-down from scores to specific rule failures

Alerting​

  • Configure alerts for critical rule failures
  • Email notifications for quality degradation
  • Integration with monitoring systems

Troubleshooting​

Common Issues​

IssueCauseSolution
Rules not generatingNo profiling dataRun Data Profiler first
Too many false positivesOverly strict thresholdsAdjust rule parameters
Slow rule executionComplex regex patternsSimplify or optimize patterns
Missing expected rulesLimited AI analysisManually add specific rules

API Access​

Programmatically manage quality rules:

  • Create/update/delete rules via API
  • Trigger rule execution
  • Query rule results
  • Export rule configurations

Next Steps​

After creating quality rules:

  1. View quality results
  2. Review data profiling
  3. Browse data catalog
  4. Check sensitivity classifications