Using Selection to Reduce Data Bulk (and Improve Data…
One of the best ways to speed up big data processing operations is to not process so much data in the first place; i.e. to eliminate unnecessary data ahead of time. Read More
Informatica Pushdown Optimization with CoSort
Big data integration activities can happen outside the database in an extract, transform, load (ETL) environment, or inside the database in ELT:
http://www.iri.com/blog/data-transformation2/etl-vs-elt-we-posit-you-judge/
One example of an ELT operation would be Informatica’s Pushdown Optimization option, in which users transform data in a relational database like Oracle, or in Teradata. Read More
ETL, ELT & IRI in Between
The IRI data management platform Voracity, as well as its constituent tools, can perform and speed big data warehouse extract, transform, load (ETL) operations, delaying the need for new hardware or expensive proprietary appliances: http://www.iri.com/blog/data-transformation2/a-big-data-quandary-hardware-or-software-appliances-or-cosort/ Read More
Big Data Transformations with CoSort (Structured Data)
In 1992, Digital Equipment Corporation (DEC, long since acquired) asked IRI to develop a 4GL interface to CoSort in the syntax of the VAX VMS sort/merge utility. Read More
Sort Demonstration [video]
This demonstration shows how to set up a sort job for CoSort using the IRI Workbench. The sort is accomplished using the SortCL language. This video takes a CSV input file, shows how to define the sort keys and options, and demonstrates how to define the targets for output. Read More
Performing a Summary Aggregation with CoSort and the IRI…
This demonstration shows how to use the IRI Workbench to create an aggregation job using sums. Workbench is used to create the job script in the SortCL language. Read More
Speeding and Building on Unix bin/sort
The sort included with each Unix-based operating system is a standard command line program that prints lines of input or specified input files in the specified sorted order. Read More
A Big Data Quandary: Hardware or Software … Appliances…
Big Data Problem Big data volumes are growing exponentially. This phenomenon had been happening for years, but its pace began a more rapid acceleration in 2012. Read More
Pivot Transformations
Changing between column and row formats is an online analytical processing (OLAP) activity that optimizes data structures for queries and reports.
A pivot transformation turns multiple rows of data into one, denormalizing a data set into a more compact version by rotating the input data on a column value. Read More
Support for Star Schema
Star schema is the simplest and most common database modelling structure used in traditional data warehouse paradigms. The schema resembles a constellation of stars — generally several bright stars (facts) surrounded by dimmer ones (dimensions) where one or more fact tables reference different dimension tables. Read More
Introduction to IRI NextForm Data Migration [video]
IRI NextForm converts, replicates, federates, and reports on data in multiple sources and targets. There are six editions that specialize on different migration goals: Lite Edition – basic file conversions, COBOL edition – migrates MF COBOL, MF-ISAM, Acucobol, DBMS edition – database migration, Legacy edition – extracts data from hundreds of legacy sources, Modern edition – addresses cloud, SaaS, and big data sources, and Premium edition – which customizes your migrations for any of these data types, sources, and targets. Read More