ETL & Beyond
Does your data integration platform do everything you need it to do, and does it work seamlessly with other critical data management activities?
- If so, and you're not happy with its performance, price, or complexity
- If not, and you need to modernize your data integration, governance, or wrangling strategy
then you may wish to reconsider your data integration solutions strategy, and examine the big data ETL tools in the IRI Voracity platform in particular.
All the features/functions listed below are supported in the Voracity data management platform and its included IRI Data Manager and IRI Data Protector suite products.
GUI refers to the IRI Workbench Graphical User Interface for Voracity. IRI Workbench is the widely adopted, user-friendly Integrated Development Environment (IDE) built on Eclipse™ for integrating and transforming data in Voracity.
DTP refers to the Data Tools Plugin (and Data Source Explorer) in the IRI Workbench. DDF refers to Data Definition Files, the simple, open metadata for source and target data layouts.
| Operation | Description |
|---|---|
|
Discover data in pattern, fuzzy, and dictionary searches through DBs, files, or "dark data" documents. Perform traditional DB profiling and E-R diagramming on connected tables. Auto-classify data into groups and match them to transformation, protection, and other rules. |
|
|
Create and modify jobs in multiple ways: a visual workflow palette, end-to-end wizards, GUI dialogs, batchable 4GL scripts, and even a metadata API that are all modeled and outlined in the GUI's syntax-aware editor ... even work your flow and task scripts in any external text editor |
|
|
Connectors |
Manage your data assets (including RDBs, LDIF, CSV, XML, COBOL, and other files) from the Eclipse project explorer, data source explorer, and remote systems explorer. Support is also available for mainframe index files, unstructured data data file formats, ASN.1-compatible CDRs, multiple legacy/proprietary formats, and big data and cloud/SaaS platforms; see the complete list here. |
|
Job Wizards |
Automate the generation of your ETL or standalone unload, transform, or load jobs, plus slowly changing dimension, change data capture, pivoting, subsetting, data masking, data migration / replication, and test data generation / population jobs. |
|
High performance, standalone or combined ETL operations in Voracity, i.e.,
If you have single sources >10TB, Voracity can also run many CoSort (SortCL) transformation, reformatting, and masking jobs seamlessly in Hadoop MapReduce2, Spark, Spark Stream, Storm, or Tez through the VGrid gateway to your (Cloudera, HortonWorks, MapR, or generic Apache) distribution. |
|
|
High performance "E" and pre-sorted "L". Design/manage "T" in Voracity (above) or integrated SQL operations |
|
|
Leverage embedded BI or data wrangling options in Voracity; i.e.:
Learn why DW industry guru Dr. Barry Devlin named Voracity a Production Analytic Platform |
|
|
Encrypt, redact, pseudonymize, hash, randomize, tokenize, or otherwise de-identify PII seamlessly; i.e., data masking on the fly in the same job script and I/O pass with all the ETL, cleansing, migration, and analytic / reporting functions listed on this page. The true 'magic' and value of Voracity is this very kind of task consolidation. |
|
|
Improve data quality with a variety of data scrubbing and standardization techniques |
|
|
Acquire, filter, subset, re-map and/or copy data from old to new data stores |
|
|
Team Share |
Update, check-in, manage, and share metadata and jobs in GIT, CVS, SVN, DataSwitch, MIMB, Quest (Erwin / AnalytiX DS) Mapping Manager, etc. |
|
Repositories |
Save, share, and re-use DDF metadata, master data dictionaries, business glossaries, set (lookup) files, rules, flow, and job scripts |
|
Data Views |
See and work directly with your source and target data in files and tables in custom editor and cell displays |
|
Schema |
Use static, create dynamic, or convert schemas via target mapping and table creation options |
|
Lineage |
Free Eclipse plug-ins support manual - and Erwin Mapping Manager supports visual - data lineage and impact analysis. Track and compare metadata and other resources (scripts, rules, templates) in version control hubs. |
|
Job Fragments |
Save, reference, and re-use job and metadata subsets in standalone, portable .DDF files, rule libraries, and other open artifacts |
|
Transpose rows to columns and columns to rows to de-normalize or normalize your data efficiently through an easy wizard |
|
|
Compare files or tables to identify, report on, and feed updates for smaller, real-time ETL using an intuitive job wizard |
|
|
Report on values from "fuzzy" lookup logic where they satisfy 'other than equal' criteria in all the common types from one wizard |
|
|
Windowed Aggregates |
Perform aggregation within specified row ranges for fair cost accounting and other apps |
|
Rules |
Define, store, and re-use field-level business rules for data transformation, protection, and test data generation |
|
Prototype & Test |
Generate and load safe, realistic, and referentially correct test data in file or table targets -- without real data -- for an entire EDW in Voracity's built-in IRI RowGen wizard(s). Or, use Voracity's built-in DB subsetting wizard to filter and mask referentially correct DB test sets. Or, preview the output of ETL and other workflow tasks with real data, or immediately simulated test data in the same format. |
