The rate of speed at which technology is changing is matched only by the speed of new applications. From high-frequency trading (HFT) to the Internet of Things (IoT), sophisticated processing capabilities mean that we just keep receiving and using data faster.
But is faster always better? HFT could be the cautionary tale in all this need for speed. Have you read the book Flash Boys by Michael Lewis, in which he investigates the phenomenon of HFT? If so, you might remember he said that the IT people who created the technology didn’t actually understand how the bankers made money with it.
Having worked for Apama since 2005, I witnessed this HFT revolution close up, as Apama was one of the leading complex event processing (CEP) engines that fueled the HFT trading revolution.
For more articles on big data trends, access the BIG DATA SOURCEBOOK
I can personally vouch that very few of us understood—in depth—how the traders made their money (probably the ex-traders we hired to sell to traders were the only ones who understood). Of course, we comprehended in broad terms how things worked, and we all knew the standard strategies such as slicing and pair trading—and we knew what the features were in the engine that made all the difference.
But, as to how the banks really made their money—their particular advanced strategy that made the difference—that was a highly guarded secret. And vice versa was true as well; the bankers really didn’t understand how it was possible that an engine could so quickly (we are talking microseconds) analyze very complicated patterns in space and time—they just took advantage of it.
Lewis was surprised and couldn’t understand why IT people didn’t know how the traders were making money with HFT—and he was once a trader. If you look around, you probably see that type of situation in your life too. Take manufacturing and how, thanks to Adam Smith and the division of labor, we have created an advanced economy with such highly specialized tasks that only a select few really understand the whole production process in detail.
Today, as IT and OT (operational technology) come together, knowledge domains that were separated in nature now have to learn each other’s secrets. In my opinion, this is the main challenge of the Industrial IoT (IIoT) revolution.
And it seems that history repeats itself. In order to explain why this is the main challenge of IIoT, we have to fast-forward to 2018. CEP engines have evolved and are now part of the analytical arsenal of most IoT platforms. For example, my previous company Apama was bought by Software AG and is now buzzing at the very core of its IoT platform. Instead of talking to bankers, I now talk to machine builders, but the challenges are remarkably alike. Let me explain this by recounting a recent conversation.
Bread-Making and IoT
A family-owned company specializes in building bread-baking machines. It delivers everything from the machines that raise the dough to the ovens, and the family told me its business mainly comes from providing the equipment to bakeries that bake bread on an industrial scale.
Its challenges are three-fold. One, its customers (the bakeries) are under pressure from their customers (the supermarkets) to deliver larger and larger quantities under more demanding quality conditions. For instance, every day a buyer may require 60,000 loaves of bread of X size and weight in different varieties.