Friday, 7 March 2014

the front lines of the data-analytics revolution


At a unique gathering of data-analytics leaders, new solutions began emerging to vexing privacy, talent, organizational, and frontline-adoption challenges.

This past October, eight executives from companies that are leaders in data analytics got together to share perspectives on their biggest challenges. All were the most senior executives with data-analytics responsibility in their companies, which included AIG, American Express, Samsung Mobile, Siemens Healthcare, TD Bank, and Wal-Mart Stores. Their backgrounds varied, with chief information officers, a chief data officer, a chief marketing officer, a chief risk officer, and a chief science officer all represented.1 We had seeded the discussion by asking each of them in advance about the burning issues they were facing.
For these executives, the top five questions were:
  • Are data and analytics overhyped?
  • Do privacy issues threaten progress?
  • Is talent acquisition slowing strategy?
  • What organizational models work best?
  • What’s the best way to assure adoption?
Here is a synthesis of the discussion.

1. Data and analytics aren’t overhyped—but they’re oversimplified

Participants all agreed that the expectations of senior management are a real issue. Big-data analytics are delivering an economic impact in the organization, but too often senior leaders’ hopes for benefits are divorced from the realities of frontline application. That leaves them ill prepared for the challenges that inevitably arise and quickly breed skepticism.
The focus on applications helps companies to move away from “the helicopter view,” noted one participant, in which “it all looks the same.” The reality of where and how data analytics can improve performance varies dramatically by company and industry.
Customer-facing activities. In some industries, such as telecommunications, this is where the greatest opportunities lie. Here, companies benefit most when they focus on analytics models that optimize pricing of services across consumer life cycles, maximize marketing spending by predicting areas where product promotions will be most effective, and identify tactics for customer retention.
Internal applications. In other industries, such as transportation services, models will focus on process efficiencies—optimizing routes, for example, or scheduling crews given variations in worker availability and demand.
Hybrid applications. Other industries need a balance of both. Retailers, for example, can harness data to influence next-product-to-buy decisions and to optimize location choices for new stores or to map product flows through supply chains. Insurers, similarly, want to predict features that will help them extend product lines and assess emerging areas of portfolio risk. Establishing priorities wisely and with a realistic sense of the associated challenges lies at the heart of a successful data-analytics strategy.
Companies need to operate along two horizons: capturing quick wins to build momentum while keeping sight of longer-term, ground-breaking applications. Although, as one executive noted, “We carefully measure our near-term impact and generate internal ‘buzz’ around these results,” there was also a strong belief in the room that the journey crosses several horizons. “We are just seeing the tip of the iceberg,” said one participant. Many believed that the real prize lies in reimagining existing businesses or launching entirely new ones based on the data companies possess.
New opportunities will continue to open up. For example, there was a growing awareness, among participants, of the potential of tapping swelling reservoirs of external data—sometimes known as open data—and combining them with existing proprietary data to improve models and business outcomes. (See “What executives should know about open data.”) Hedge funds have been among the first to exploit a flood of newly accessible government data, correlating that information with stock-price movements to spot short-term investment opportunities. Corporations with longer investment time horizons will need a different playbook for open data, but few participants doubted the value of developing one.

2. Privacy concerns must be addressed—and giving consumers control can help

Privacy has become the third rail in the public discussion of big data, as media accounts have rightly pointed out excesses in some data-gathering methods. Little wonder that consumer wariness has risen. (Data concerns seem smaller in the business-to-business realm.) The flip side is that data analytics increasingly provides consumers, not to mention companies and governments, with a raft of benefits, such as improved health-care outcomes, new products precisely reflecting consumer preferences, or more useful and meaningful digital experiences resulting from a greater ability to customize information. These benefits, by necessity, rest upon the collection, storage, and analysis of large, granular data sets that describe real people.
Our analytics leaders were unanimous in their view that placing more control of information in the hands of consumers, along with building their trust, is the right path forward.
Opt-in models. A first step is allowing consumers to opt in or opt out of the collection, sharing, and use of their data. As one example, data aggregator Acxiom recently launched a website (aboutthedata .com) that allows consumers to review, edit, and limit the distribution of the data the company has collected about them. Consumers, for instance, may choose to limit the sharing of their data for use in targeted Internet ads. They control the trade-off between targeted (but less private) ads and nontargeted ones (potentially offering less value).
Company behavior. Our panelists presume that in the data-collection arena, the motives of companies are good and organizations will act responsibly. But they must earn this trust continually; recovering from a single privacy breach or misjudgment could take years. Installing internal practices that reinforce good data stewardship, while also communicating the benefits of data analytics to customers, is of paramount importance. In the words of one participant: “Consumers will trust companies that are true to their value proposition. If we focus on delivering that, consumers will be delighted. If we stray, we’re in problem territory.”

3. Talent challenges are stimulating innovative approaches—but more is needed

Talent is a hot issue for everyone. It extends far beyond the notoriously short supply of IT and analytics professionals. Even companies that are starting to crack the skill problem through creative recruiting and compensation strategies are finding themselves shorthanded in another area: they need more “translators”—people whose talents bridge the disciplines of IT and data, analytics, and business decision making. These translators can drive the design and execution of the overall data-analytics strategy while linking IT, analytics, and business-unit teams. Without such employees, the impact of new data strategies, tools, and methodologies, no matter how advanced, is disappointing.
The amalgam is rare, however. In a more likely talent scenario, companies find individuals who combine two of the three needed skills. The data strategists’combination of IT knowledge and experience making business decisions makes them well suited to define the data requirements for high-value business analytics.Data scientists combine deep analytics expertise with IT know-how to develop sophisticated models and algorithms. Analytic consultants combine practical business knowledge with analytics experience to zero in on high-impact opportunities for analytics.
A widespread observation among participants was that the usual sources of talent—elite universities and MBA programs—are falling short. Few are developing the courses needed to turn out people with these combinations of skills. To compensate, and to get more individuals grounded in business and quantitative skills, some companies are luring data scientists from leading Internet companies; others are looking offshore.
The management and retention of these special individuals requires changes in mind-set and culture. Job one: provide space and freedom to stimulate exploration of new approaches and insights. “At times, you may not know exactly what they”—data scientists— “will find,” one executive noted in describing the company’s efforts to provide more latitude for innovation. (So far, these efforts are boosting retention rates.) Another priority: create a vibrant environment so top talent feels it’s at t

The Awesome Ways Big Data Is Used Today To Change Our World




The term ‘Big Data’ is a massive buzzword at the moment and many say big data is all talk and no action. This couldn’t be further from the truth. With this post, I want to show how big data is used today to add real value.
Eventually, every aspect of our lives will be affected by big data. However, there are some areas where big data is already making a real difference today. I have categorized the application of big data into 10 areas where I see the most widespread use as well as the highest benefits [For those of you who would like to take a step back here and understand, in simple terms, what big data is, check out the posts in my Big Data Guru column].
1. Understanding and Targeting Customers
This is one of the biggest and most publicized areas of big data use today. Here, big data is used to better understand customers and their behaviors and preferences. Companies are keen to expand their traditional data sets with social media data, browser logs as well as text analytics and sensor data to get a more complete picture of their customers. The big objective, in many cases, is to create predictive models. You might remember the example of U.S. retailer Target, who is now able to very accurately predict when one of their customers will expect a baby. Using big data, Telecom companies can now better predict customer churn; Wal-Mart can predict what products will sell, and car insurance companies understand how well their customers actually drive. Even government election campaigns can be optimized using big data analytics. Some believe, Obama’s win after the 2012 presidential election campaign was due to his team’s superior ability to use big data analytics.
2. Understanding and Optimizing Business Processes
Big data is also increasingly used to optimize business processes. Retailers are able to optimize their stock based on predictions generated from social media data, web search trends and weather forecasts. One particular business process that is seeing a lot of big data analytics is supply chain or delivery route optimization. Here, geographic positioning and radio frequency identification sensors are used to track goods or delivery vehicles and optimize routes by integrating live traffic data, etc. HR business processes are also being improved using big data analytics. This includes the optimization of talent acquisition – Moneyball style, as well as the measurement of company culture and staff engagement using big data tools.
3. Personal Quantification and Performance Optimization
Big data is not just for companies and governments but also for all of us individually. We can now benefit from the data generated from wearable devices such as smart watches or smart bracelets. Take the Up band from Jawbone as an example: the armband collects data on our calorie consumption, activity levels, and our sleep patterns. While it gives individuals rich insights, the real value is in analyzing the collective data. In Jawbone’s case, the company now collects 60 years worth of sleep data every night. Analyzing such volumes of data will bring entirely new insights that it can feed back to individual users. The other area where we benefit from big data analytics is finding love - online this is. Most online dating sites apply big data tools and algorithms to find us the most appropriate matches.
4. Improving Healthcare and Public Health
The computing power of big data analytics enables us to decode entire DNA strings in minutes and will allow us to find new cures and better understand and predict disease patterns. Just think of what happens when all the individual data from smart watches and wearable devices can be used to apply it to millions of people and their various diseases. The clinical trials of the future won’t be limited by small sample sizes but could potentially include everyone! Big data techniques are already being used to monitor babies in a specialist premature and sick baby unit. By recording and analyzing every heart beat and breathing pattern of every baby, the unit was able to develop algorithms that can now predict infections 24 hours before any physical symptoms appear. That way, the team can intervene early and save fragile babies in an environment where every hour counts. What’s more, big data analytics allow us to monitor and predict the developments of epidemics and disease outbreaks. Integrating data from medical records with social media analytics enables us to monitor flu outbreaks in real-time, simply by listening to what people are saying, i.e. “Feeling rubbish today - in bed with a cold”.
5. Improving Sports Performance
Most elite sports have now embraced big data analytics. We have the IBM SlamTracker tool for tennis tournaments; we use video analytics that track the performance of every player in a football or baseball game, and sensor technology in sports equipment such as basket balls or golf clubs allows us to get feedback (via smart phones and cloud servers) on our game and how to improve it. Many elite sports teams also track athletes outside of the sporting environment – using smart technology to track nutrition and sleep, as well as social media conversations to monitor emotional wellbeing.
6. Improving Science and Research
Science and research is currently being transformed by the new possibilities big data brings. Take, for example, CERN, the Swiss nuclear physics lab with its Large Hadron Collider, the world’s largest and most powerful particle accelerator. Experiments to unlock the secrets of our universe – how it started and works - generate huge amounts of data. The CERN data center has 65,000 processors to analyze its 30 petabytes of data. However, it uses the computing powers of thousands of computers distributed across 150 data centers worldwide to analyze the data. Such computing powers can be leveraged to transform so many other areas of science and research.
7. Optimizing Machine and Device Performance
Big data analytics help machines and devices become smarter and more autonomous. For example, big data tools are used to operate Google’s self-driving car. The Toyota Prius is fitted with cameras, GPS as well as powerful computers and sensors to safely drive on the road without the intervention of human beings. Big data tools are also used to optimize energy grids using data from smart meters. We can even use big data tools to optimize the performance of computers and data warehouses.
8. Improving Security and Law Enforcement.
Big data is applied heavily in improving security and enabling law enforcement. I am sure you are aware of the revelations that the National Security Agency (NSA) in the U.S. uses big data analytics to foil terrorist plots (and maybe spy on us). Others use big data techniques to detect and prevent cyber attacks. Police forces use big data tools to catch criminals and even predict criminal activity and credit card companies use big data use it to detect fraudulent transactions.
9. Improving and Optimizing Cities and Countries
Big data is used to improve many aspects of our cities and countries. For example, it allows cities to optimize traffic flows based on real time traffic information as well as social media and weather data. A number of cities are currently piloting big data analytics with the aim of turning themselves into Smart Cities, where the transport infrastructure and utility processes are all joined up. Where a bus would wait for a delayed train and where traffic signals predict traffic volumes and operate to minimize jams.
10. Financial Trading
My final category of big data application comes from financial trading. High-Frequency Trading (HFT) is an area where big data finds a lot of use today. Here, big data algorithms are used to make trading decisions. Today, the majority of equity trading now takes place via data algorithms that increasingly take into account signals from social media networks and news websites to make, buy and sell decisions in split seconds.
For me, the 10 categories I have outlined here represent the areas in which big data is applied the most. Of course there are so many other applications of big data and there will be many new categories as the tools become more widespread.

Big Data: The 5 Vs Everyone Must Know

Big Data is a big thing. It will change our world completely and is not a passing fad that will go away. To understand the phenomenon that is big data, it is often described using five Vs: Volume, Velocity, Variety, Veracity and Value
I thought it might be worth just reiterating what these five Vs are, in plain and simple language:
Volume refers to the vast amounts of data generated every second. Just think of all the emails, twitter messages, photos, video clips, sensor data etc. we produce and share every second. We are not talking Terabytes but Zettabytes or Brontobytes. On Facebook alone we send 10 billion messages per day, click the "like' button 4.5 billion times and upload 350 million new pictures each and every day. If we take all the data generated in the world between the beginning of time and 2008, the same amount of data will soon be generated every minute! This increasingly makes data sets too large to store and analyse using traditional database technology. With big data technology we can now store and use these data sets with the help of distributed systems, where parts of the data is stored in different locations and brought together by software.
Velocity refers to the speed at which new data is generated and the speed at which data moves around. Just think of social media messages going viral in seconds, the speed at which credit card transactions are checked for fraudulent activities, or the milliseconds it takes trading systems to analyse social media networks to pick up signals that trigger decisions to buy or sell shares. Big data technology allows us now to analyse the data while it is being generated, without ever putting it into databases.
Variety refers to the different types of data we can now use. In the past we focused on structured data that neatly fits into tables or relational databases, such as financial data (e.g. sales by product or region). In fact, 80% of the world’s data is now unstructured, and therefore can’t easily be put into tables (think of photos, video sequences or social media updates). With big data technology we can now harness differed types of data (structured and unstructured) including messages, social media conversations, photos, sensor data, video or voice recordings and bring them together with more traditional, structured data.
Veracity refers to the messiness or trustworthiness of the data. With many forms of big data, quality and accuracy are less controllable (just think of Twitter posts with hash tags, abbreviations, typos and colloquial speech as well as the reliability and accuracy of content) but big data and analytics technology now allows us to work with these type of data. The volumes often make up for the lack of quality or accuracy.
Value: Then there is another V to take into account when looking at Big Data: Value! It is all well and good having access to big data but unless we can turn it into value it is useless. So you can safely argue that 'value' is the most important V of Big Data. It is important that businesses make a business case for any attempt to collect and leverage big data. It is so easy to fall into the buzz trap and embark on big data initiatives without a clear understanding of costs and benefits.

-------------------