Skip to content

Data Validation Overview and Responsibilities

Every year, University of California – Office of the President receives hundreds of files from within the UC system that must be validated and processed into the UC Data Warehouse. This information is vital for the functioning and promotion of the university in the media, in state budget discussions, and in every day operations. As this information covers such diverse topics as students enrolled across the system, degrees awarded, and contracts and grants for each university, to name just a few, it’s vital that data be of the highest quality to ensure that the achievements of each campus are represented fairly and accurately. 

There are three basic groups involved in processing each and every file:

  1. UC Campus Business Units. Different business units across the campus are responsible for providing accurate information that a) Meets the provided file specification and b) is available for UCOP processing by the file submission deadline.
  2. UCOP – IRAP (Institutional Research and Academic Planning). Responsible for certifying that input files meet the minimum requirements to be loaded into the data warehouse.
  3. UCOP – Data Services. Responsible for providing validation reports and importing certified data into data warehouse.

Data Validation Process

The Data Validation Process designed and implemented by the Institutional Research and Academic Planning (IRAP) unit and the Data Services department is aimed at streamlining the validation and certification of data provided for the UC Data Warehouse (UCDW) by the campuses.

This process involves the collaboration of the UC Campuses, UCOP IRAP and UCOP Data Services.

Roles & Responsibilities:

UC Campuses

  1. Campuses create various input files with campus data based on pre-defined file specifications. Supplied input files include the Contracts and Grants, Undergraduate and Graduate admissions, Student Registration and Health Science Residents, Financial Aid and Degree files.
  2. Currently these input files are placed on the UCOP VSFTP server on or before the agreed file submission deadlines.
  3. Once files are processed into a preview area of the UC Data Warehouse, campuses are will be notified to validate their campus data and certify it to be correct.
  4. Campuses will provide IRAP with consent to process supplied data.
  5. If validation reveals that a campus file is erroneous, campus will resubmit a new corrected file.

UCOP Data Services

  1. UCOP Data Services retrieves campus input files from the VSFTP Server as they come in.
  2. UCOP Data Services processes these files and loads them into a preview area where validation reports are generated.
  3. UCOP Data Services will rigorously verify using unit testing and quality assurance processes (QA) that the transformation rules were correctly applied to the supplied campus input files.
  4. UCOP Data Services will provide results of all quality checks in a format that UCOP IRAP will understand. If there are variances between the source and target systems, UCOP Data Services will provide reasons for these variances.
  5. UCOP Data Services will notify both UCOP IRAP and UC campuses that validation reports are ready to be viewed.
  6. Once the data is deemed correct and is certified, UCOP Data Services loads the data into the reporting layer of the UC Data Warehouse.

UCOP IRAP

  1. UCOP IRAP will review all validation reports for all UC campus sites for input file validity only.
  2. If there are any data issues with supplied campus files, UCOP IRAP will work with campuses to get files corrected and resubmitted for processing.
  3. UCOP IRAP will not be responsible for verifying the accuracy of transformation rules applied by UCOP Data Services between staging and base data layers.
  4. UCOP will continue to perform user acceptance testing (UAT) of the Cognos Validation Reports to enable continuous improvement of the said reports.
  5. Based on the review of the quality checks performed by UCOP Data Services as well as the verification that the input files provided by the campuses are good, UCOP IRAP indicate consent to load the data into the reporting layer of the UC Data Warehouse.
  6. UCOP IRAP will develop a finite list of representative test cases (using production and/or published reports) to verify that the data within the reporting layer are accurate.

For any questions about the data validation process, please contact irap@ucop.edu.