Abstract

Data quality observability has matured from a niche concern into a core data platform capability through 2025, driven by the downstream costs of silent data degradation in machine learning features, financial reporting, and customer-facing products. Commercial platforms like Monte Carlo and open-source frameworks including Great Expectations and Soda Core have each staked out distinct positions in the market. This article examines how these tools operate technically, how they differ in their detection approaches, and the architectural patterns teams use to integrate quality monitoring without duplicating pipeline complexity.

Detection Approaches

Monte Carlo’s approach is based on automated anomaly detection over table-level statistics - row counts, null rates, schema change events, and freshness timestamps - collected continuously from warehouse query logs and information schema tables. It requires no explicit rule authoring for its baseline monitors; instead, it learns normal statistical behavior over a rolling window and alerts on deviations. This unsupervised approach catches unexpected regressions without requiring engineers to anticipate every failure mode, but produces false positives during intentional schema migrations or seasonal volume shifts that fall outside learned norms.

Great Expectations takes the opposite philosophy: quality is defined by explicit expectations authored by the team and versioned in a data docs store. An expectation suite for a transactions table might assert that amount is always positive, user_id is never null, and row counts fall between 10,000 and 500,000. Suites run as validation steps in Airflow DAGs, dbt tests, or Spark jobs, and results are written to a configured store (S3, local filesystem, or a database). The approach requires upfront investment in expectation authoring but produces deterministic, auditable quality gates with no false positives from learned baselines.

Soda Core occupies a middle ground, using a YAML-based check definition language (SodaCL) that is more concise than Great Expectations’ Python API and supports a growing set of built-in check types including freshness checks, schema drift detection, and cross-table referential integrity assertions. Soda Cloud provides centralized results dashboards, while Soda Core remains open-source and runnable standalone.

Integration Architecture

The most common production pattern pairs automated anomaly detection for broad coverage with explicit rule-based checks at known critical points. Monte Carlo or a similar platform monitors the entire warehouse for unexpected table-level changes, providing a safety net across hundreds of tables without per-table configuration. Great Expectations or Soda suites guard the output of specific high-stakes transformations - the revenue fact table, the ML feature store snapshot, the daily customer cohort - where the business cost of a silent error is highest.

dbt’s built-in tests block covers the most common cases (not-null, unique, accepted values, referential integrity) and runs as part of every dbt build. Teams with dbt-centric stacks often use dbt tests as their primary explicit check layer and reserve Great Expectations for checks that require more complex logic or cross-source validation that dbt cannot express natively.

Lineage Integration

Quality observability without lineage produces alerts that are difficult to act on. Knowing that a downstream revenue table has an unexpected null rate is less useful than knowing which upstream source table and transformation introduced the nulls. Monte Carlo, Atlan, and DataHub each maintain lineage graphs that connect quality events to their source tables, allowing on-call engineers to trace incidents upstream rather than inspecting every table in the dependency chain.

Column-level lineage - tracking which specific columns flow from source to target through transformation logic - is the current frontier. It enables quality rules to propagate automatically: if a source column gains a not-null rule, derived columns in downstream tables can inherit the assertion. As of mid-2025, column-level lineage is available in Monte Carlo and DataHub for SQL-based transformations, with coverage for Spark DataFrame operations still maturing.

Practical Recommendations

Teams starting a data quality program should resist the urge to write exhaustive expectation suites for every table immediately. A practical starting point is freshness monitoring on all tables (trivially automated), not-null and unique constraints on primary key columns, and row-count anomaly detection on high-volume ingestion tables. These three check types catch the majority of real incidents and can be deployed in a day. Deeper semantic checks - cross-table consistency, business rule validation, distribution drift - are high-value additions for tables that directly feed dashboards or model training, and should be prioritized by downstream business impact rather than implemented uniformly.