Challenges
Many data warehouse ETL and analytic environments need to perform custom transformations on data to handle special business logic the exceeds the scope of off the shelf functions. Unfortunately, their coding typically entails special script components, SQL procedures, or assemblies.
In addition to their learning and maintenance curves, custom data manipulation techniques may not operate at the field level, serialze as rules for re-use, run in the same I/O pass with other transformations, or leverage the APIs of best-of-breed software.
You may also face data reengineering requirements that result from data profiling ... but then find that there is not enough time or expertise available. How will you write and maintain the custom scanning, cleansing, transformation, translation, or standardization routines that address your business requirements?
You may also need to apply special data protection(s) -- based on specific security circumstances -- that fall outside the scope of your application. Can you be sure the procedure you need to integrate will satisfy your business rules and comply with applicable data privacy laws?
Solutions
The SortCL program in the IRI CoSort package and IRI Voracity platform integrates and transforms data in multiple sources. While it simultaneously manipulates, reports on, and secures data at the field level, the opportunity for custom transforms is also at hand.
Plug-in complex field-level transforms into SortCL jobs. Run them with your other data processing, presentation, and protection tasks.
Specify custom transforms on the same field twice in one job script, if necessary; i.e., in the pre-action layout or target phases. This allows two transforms per field within the same I/O pass.
Examples of custom transforms in SortCL include data quality routines written by Trillium and Melissa Data. Most of the data masking and encryption routines that gave rise to IRI FieldShield, and the national ID test data generation functions in IRI RowGen, are also custom transforms.
Data transformation customization in the IRI envrionment means you can "roll your own" transforms with a simple C interface. That is, you can write your own custom transformation tools as functions that perform special lookups, and/or otherwise manipulate, mask, munge, mine, format or synthesize data as you see fit.
Alternatively, you can leverage custom transformation services from IRI Professional Services. Our engineers can design, test, deliver and or run field-level transforms that meet your exact requirements.
