IRI RowGen is a state-of-the-art, fit-for-purpose software product designed by database and DataVault experts for the rapid, reliable creation and management of safe, intelligent test data for multiple targets, big and small, on-premise or in the cloud. RowGen delivers test data for demanding stakeholders in DevOps, DB and DW ETL prototypes, benchmarking and demonstration.
RowGen is front-ended in IRI Workbench, the free and familiar Eclipse™ IDE for designing, modifying, and running test data jobs in script or graphical form. RowGen is available as a standalone product, bundled into a multi-product IRI Data Manager suite purchase, or automatically included in all IRI Voracity data mangement platform subscriptions.
Test Database Targets
Most users are interested in RowGen's ability to create and load structurally and referentially correct test data for very large relational databases. Rather than relying on production data access and masking operations (available to IRI FieldShield and Voracity users), RowGen uses existing or client-provided DDLs to generate realistic data from scratch for:
- Oracle
- Sybase
- DB2 UDB
- PostgreSQL
- MySQL
- Snowflake
- SQL Server
- Teradata
and packaged apps (e.g., Dynamics, PeopleSoft, Informatica, SAP ECC) using the above RDBs. It also supports the generation of synthetic subsets where you select (and save) the include or omit criteria for each target. If you use Voracity, there is also a DB subsetting wizard, which supports masking as an alternative approach.
Use the 'Create DB Test Data' wizard in IRI Workbench for RowGen to do the following automatically:
Parse
Table descriptions and integrity constraints are translated to test data generation definitions that reflect source structures, data types, and dependencies.
Generate
Pre-sorted flat files containing the test data are created for loading, while the definition scripts remain for easy modification and re-use.
Populate
Target tables are rapidly bulk loaded in the order necessary to maintain the same table relationships (referential integrity) that exist in production.
Test File Targets
RowGen gives you complete control over the generation of target test formats.
Us the 'Create Custom Test Data' wizard in the IRI Workbench GUI for RowGen, or work directly in the RowGen Control Language (RCL) job scripts that the wizard produces to combine data generation and formatting.
RCL scripts use the data and job definition syntax of IRI's SortCL program to describe precise layout, size, and content of the target file(s).
Specify the size, position, separator, and data type of each field element. Define conditional selection criteria or ranges for set file values. See the discussion of field value customization below.
Create one or more test files at a time in these formats:
- CSV (or other-delimited)
- ELF (W3C Extended Log Format)
- JSON
- LDIF (LDAP)
- Line sequential
- Mainframe variable block
- MF I-SAM & (Acucobol) Vision
- MF Variable Length (small/large)
- Record sequential
- Text, fixed or delimited
- Variable sSequential
- XML
Generate as many records per file, and as many files, as you specify. Format each output in custom field, record, and report layouts. Perform transformations like sorting, aggregation, and cross-calculation during the generation. See Intelligent Column (Field) Values below.
*You can write and modify RCL job scripts with any text editor, in the RowGen GUI's syntax-aware script editor, or through GUI dialogs that interact with the script.
Test Report Targets
Create custom reports with test data inside. RowGen has the same reporting features for test data that SortCL has for real data:
IRI Solutions > Business Intelligence > Embedded BI > Reporting Functions
RowGen is also ideal for franchising (preparing transformed) test data for specialized BI tools, and for providing better, faster, and bigger test data to applications like SAP and SalesForce.
Application Integration
RowGen users will find many different points of integration into their current operations.
RowGen jobs easily fit into applications and batch streams. RowGen Control Language (.RCL) specifications are in text scripts called from the command line. The jobs are simple to edit and will run anywhere RowGen is licensed.
Integrating RowGen is also easier because of the self-documenting nature and cross-compatibility of its metadata. The data definition file (.DDF) format RowGen uses is common to all IRI software tools, and is supported throughout the IRI Workbench GUI, built on Eclipse™. Metadata discovery and conversion functions produce new DDFs that RowGen and other IRI tools can use*, and free version control plug-ins in Eclipse support for data lineage and metadata impact analysis.
The universality of metadata throughout the IRI product line also allows you to easily transition between test data generation and real data processing. Jobs in CoSort SortCL (data transformation and reporting), FieldShield, and NextForm use the same DDF.
RowGen DDF is also supported in AnalytiX DS Mapping Manager and Meta Integration Technology Inc.'s Meta Integration Model Bridge (MIMB). Users of those packages can automatically convert the metadata in their DB, ETL, modeling, and related applications to RowGen DDF. This makes it easier to populate existing applications with test tables, files, and reports used in production, and with even broader value ranges for stress testing.
Use RowGen's built-in data transformation functions, such as expressions and aggregation, to test the functionality of your applications. For example, compare the results of your program's aggregation routine with the summary values that RowGen produced while creating the test data. See this case study as well.
Test targets can be files, named or unnamed pipes (stdout), one or more custom output procedures, and ODBC. Direct file output to database loaders (pre-sorted!) or your applications. Again, RowGen outputs can range from simple, one-field flat files, to EDW databases, to elaborately-formatted HTML reports complete with real-time system data.
*RowGen DDFs are also created by IRI\'s Fast Extract (FACT) tool, and by IRI-supplied metadata converters for COBOL copybooks, CSV file headers, SQL*Loader control files and ELF web log files.
Intelligent Values
Generate any number of rows with any number of columns, in fixed or delimited position. Choose the data type to generate for each field from these groups:
- ASCII & EBCDIC Characters
- Numeric, Whole, Currency, IP Address
- Alpha & EBCDIC Digits
- RM and MF COBOL Numerics
- Other Binary Numerics
- US, Europe, ISO, Japan Timestamps
- Unicode & Multi-byte
Define new data types with the compound (composite) data value wizard in the IRI Workbench GUI for RowGen. Specify, or randomly select, field values from one or more real set files, or literal values and ranges. Select a frequency distribution and preview the pattern in gnuplot.
When using set files, choose from several random selection methods (including inter-column permutation). This delivers both realism and de-identification when sets contain real names.
See field-level rules for customizing and applying field-level intelligence to the test data. You can also combine random generation and selection to fine tune the test data. In fact, RowGen has many ways to enhance realism without using real data. See:
IRI Blog > Test Data > Making Test Data Realistic Without Taking It from Production
Field-Level Rules
By default, RowGen randomly generates field (column) values based on the data type provided, without regard to range (though that can be controlled by condition statements). That is where things only begin, however.
For structured (or semi-structured) targets, you can also define the output size, field separators, and index values. And importantly, you can easily apply and modify field-level rules to one or more target tables or files that:
- randomly select data from set files (custom, single-column, tab-delimited, or database-column-extracted)
- contain specified string data (literal values)
- draw data from any numeric range in provided (linear, normal, percent) or custom distributions
- display custom 'compound data values' created one component at a time (e.g., phone number format)
Apply these same generation rules to individual table, flat-file, and custom report targets. Use the RowGen GUI's 'Create Custom Test Data' wizard, or script the field definitions directly.
Whether created by the GUI or by hand, RowGen jobs are available for re-use, integration, and batch execution.
