Skip to main content
General Validation is a hosted application with customer-owned Azure execution. You connect an Azure Data Factory or Microsoft Fabric catalog, import metadata, pair a source with a target, and reconcile the two with fixed validation functions. Your Azure Data Factory runtime reads the data and performs the comparison; the hosted service stores configuration and result metadata.

​
What you can connect and discover

Discovery reads metadata, not source or target row bodies. A catalog can contain more dataset types than the validation runtime supports, so catalog visibility alone does not mean a dataset can be paired or run.

​
Data you can validate

Each supported item that meets the requirements below can be the source or target of a Dataset Pair. The two sides can use different supported formats and can come from different registered catalogs. Both sides must be ready for validation before you can submit a run. Open an imported Dataset to check for Ready for validation; the dataset inventory uses Ready to pair for pairing readiness. The dataset readiness state is the definitive answer for an imported object. Missing schema, unsupported storage, nested shape, access, or connector settings keep that object visible while disabling pair, test, or run actions. General Validation reports the reason and remediation instead of attempting a partial validation. Parquet folders are qualified across their immediate parts, including declared schemas. Adding or changing parts requires preparation again; a schema check does not pin the source files against later changes. Folders above the 16-part limit must be reduced to a supported shape. For Azure SQL, a customer database administrator must grant the execution factory identity read access to the imported table. Password, service-principal, and user-assigned managed-identity linked services are outside the current Azure SQL execution-access contract. The hosted application does not automate these database grants.

​
Catalog visibility is not execution support

General Validation imports unsupported ADF dataset types so you can reconcile its catalog with ADF Studio without overstating what can run. If another connector contains data you need to validate, materialize a supported flat Parquet, Delta, or CSV dataset in ADLS Gen2, or use a supported Azure SQL Database or Fabric table.

​
Supported field shapes and types

The current runtime accepts flat root-level scalar fields:
  • numeric: Short, Integer, Long, Decimal, Float / Single, and Double
  • text: String
  • logical: Boolean
  • temporal: Date and Timestamp
  • bytes: Binary
Arrays, maps, structs, and other nested containers are not runnable. A connector can impose a narrower field contract: for example, typed CSV supports Boolean, Date, Single, Double, Int32, Int64, String, millisecond DateTime, and Decimal(18,2). The authoring UI uses the imported schema and dataset readiness status to show only eligible fields. Direct Parquet Timestamp fields are available only when Azure Data Factory identifies the field as a native Timestamp. If that type information is missing or changes, the field is unavailable for tests, while row count and other supported fields remain usable. SQL Server timestamp and rowversion columns are treated as Binary bytes, not dates or times.

​
The 12 validation functions

Function availability follows the selected field types:
  • COUNT and COUNT_DISTINCT accept every admitted scalar type.
  • SUM and AVG are numeric-only.
  • MIN and MAX accept every admitted scalar type. Numeric, String, Date, and Timestamp results support equality and ordering. Boolean and Binary results support only equality and inequality.
  • VALUE and OUTER_VALUE accept compatible scalar fields and require configured join keys. Numeric, String, Date, and Timestamp values support equality and ordering; Boolean and Binary support equality and inequality.
  • Set functions accept admitted scalar fields and use exact set membership or equality. They do not use a tolerance.

​
Comparison and safety rules

  • No implicit cross-kind coercion. Numeric fields can compare with other numeric fields when the operation is safe. String compares with String, Date with Date, Timestamp with Timestamp, Boolean with Boolean, and Binary with Binary. Date and Timestamp are not treated as interchangeable.
  • Intentional representation changes are explicit. VALUE, OUTER_VALUE, and Pair join keys can store a supported transform when you deliberately need to align representations. Aggregate and set fields remain in their authoritative types. Every applied transform is retained with the Test and Result.
  • Numeric equality uses an absolute threshold, not a percentage. With =, a zero threshold requires exact equality and a positive threshold of 5 accepts an absolute difference up to 5. With <>, the test passes only when the difference is greater than the threshold. With <=, the source value may be up to the threshold above the target; with >=, it may be up to the threshold below the target. Strict < and > comparisons use the values directly and ignore the threshold. Non-numeric comparisons require a zero threshold.
  • Native numeric promotion remains guarded. Compatible numeric fields use Azure Data Factory’s native promotion. If that makes distinct join keys collide, duplicate-key detection fails the test closed. Positive-tolerance arithmetic is admitted only when it can be preserved safely; General Validation does not silently round or coerce values.
  • Join keys must identify rows. A Pair supports up to five join-key columns. Duplicate normalized keys fail a joined-value test instead of letting the runtime guess which rows belong together.
  • Comparison options are disclosed. Value and set tests can opt into case-insensitive text comparison, trailing-space trimming, and explicit null behavior. These choices are off unless selected, except null-equals-null, which is on by default.
  • Filters are structured and typed. Source and target filters are validated against the imported schema. General Validation does not expose a raw ADF-expression mode for Pair filters.

​
Results and evidence

Aggregate tests produce summary results. Value and set tests can retain a configurable, capped set of affected records in customer-controlled Azure. Pass/fail tolerance and the evidence capture limit are independent: allowing a small number of discrepancies does not increase how many records are retained, and reducing the evidence cap does not change the verdict. Exact values, deltas, and captured rows remain in your Azure Storage. An authorized browser reads them directly from that storage when you open a result. The hosted service receives verdicts, counts, diagnostics, and artifact references. See Runs, results & evidence. Unsupported combinations fail before execution with a named reason. For a source-specific fit question, plan a first validation.