What is business intelligence, and why is its rapid creation and delivery important?

Business intelligence, or BI, refers to computer-based techniques used in identifying, extracting and analyzing business data. BI includes the technologies of reporting, data mining, analytics and key performance indicators (KPIs, or benchmarks).

The ultimate purpose of BI is to create meaningful information from raw data so that business decisions can be made. Because of the large and growing volumes of business data sourced from web logs, points of sale, call detail records, and so on, an efficient BI system requires that all this data — stored in very large files and database tables — are prepared and analyzed quickly.

The faster the data can be analyzed and presented for decision making, the faster business decisions can be made. The timeliness of this information has competitive consequences for: pricing and advertising, product features and launch schedules, public relations and political campaigns, responses to security threats, web promotions, airline and bank fee determinations, just-in-time supply chains, and so on.

The processes of transforming and integrating (i.e. preparing) operational data for BI and analytics is called data wrangling or  data franchising. This refers to the staging of data for hand-offs to BI and data presentation tools like BIRT, Cognos, Excel, and Microstrategy, and newer analytic tools like QlikView, R, Splunk, Spotfire, and Tableau. The preparatory data transformation activities can include: sorting, joining, aggregating, filtering, cross-calculating, cleansing, sub-setting, and reformatting.

While many business intelligence tools incorporate these data integration capabilities, or interact with databases to provide these services, the transformation of big data in the BI or database layer is inefficient. These tools are not designed to manipulate or integrate massive amounts of raw data, and faced with it, they would either take too long to deliver the requested reports, or perhaps even crash. Imagine requiring Excel to open a 30GB CSV file …

How do you speed Business Intelligence?

Use a fast data wrangling tool like IRI CoSort (or the IRI Voracity data management platform powered by CoSort) to externalize and combine big data transformations in the file system, leaving your databases and BI tools to do what they do best: query and report. See this section of the IRI website for benchmarks with various BI and analytic tools.

In many high-volume data warehouse, data mart, and analytic settings, data are staged in sequential (flat) files where preparatory manipulations can occur rapidly, utilizing the multiple threads, I/O and memory resources available in the file (or operating) system.

CoSort was designed specifically to leverage this environment, and to perform multiple transformations on data from many sources at once. Multiple targets in multiple formats can be generated — including joined, aggregated, and filtered subsets in CSV and XML format — so that BI tools can use that ‘boiled down’ data right away. Data matching, masking, and migration (as well as replication and federation) are also available CoSort functions in this context.

In addition to producing reports or hand-offs to reporting tools, CoSort is also integrated at the ODA driver level with the free Business Intelligence Reporting Tool (BIRT) in Eclipse.

Click to see a 3-minute video of IRI approaches to faster BI:

At the end of day, ask whether you are you managing big data for BI or analytics efficiently, or is it managing you? Good solution architects work with business users to automate and manage many of the tasks involved in handling large volumes of information, so the best business decisions can be made in less time, while still preserving the benefits (like data quality, organized metadata, and centralized data) of a proper data management infrastructure.