Data is only going to continue to grow and with that, so must data processing capabilities.
Many of the more traditional methods of data processing have struggled to keep with the immense surge in data over the past few years. A recent webinar covering key solutions to speed up data processing was presented byKevin Petrie, senior director, Attunity; and David Friedland, senior vice president of business development with IRI. “In short, we now have a mandate to have to be able to process terabytes and even petabytes of data faster,” stated Petrie.
There are three challenges that present themselves to organizations that are having data processing issues: growth in data, platform complexity, and unknown data usage. Attunity proposes a few steps to help speed up data processing for organizations today: integrate faster across platforms, automate data warehousing, and profile usage to optimize placement.
To integrate across platforms faster Attunity proposes its software that will speed up the ETL process by not copying old already processed data and only processing new data. Another aspect that will speed up data processing is data automation. The traditional process of preparing a data warehouse is intensive and time consuming. Attunity’s software automates many of the processes and speeds it up for the user.
Friedland listed many of the common challenges when it comes to processing data: volume, velocity, variety, veracity, and value. To combat these challenges Friedland suggested three methods: embedded BI, CoSort-BIRT integration, and data preparation for your tool.
“There is going to be a fourth way as well, through a shard processing query tool that will be brought to the market next year,” noted Friedland. The most common method is the third, which is using CoSort to accelerate commercial BI and analytic tools. According to Friedland, in the company's comparisons of tools with and without CoSort and the speed-ups range from 2-16 times faster when using CoSort ahead of the BI tools.
To view a replay of this webinar, go here.
Image courtesy of Shutterstock