News on Hadoop - June 2017
Hadoop Servers Expose Over 5 Petabytes of Data. BleepingComputer.com, June 2, 2017.
According to John Matherly, the founder of Shodan, a search engine used for discovering IoT devices found that Hadoop installed improperly configured HDFS based servers exposed over 5 PB of information. He found approximately 4487 HDFS servers available without authentication through public IP addresses that in total exposed 5120 TB of data.The expert said that 47820 MongoDB servers exposed only 25TB of data while HDFS servers leak 200 times more data when compared to MongoDB servers which is 10 times more prevalent. According to the report from Binary Edge 2015, MongoDB, Memcached, ElasticSearch and Redis servers collectively exposed only 1.1 PB of data.
(Source : https://www.bleepingcomputer.com/news/security/hadoop-servers-expose-over-5-petabytes-of-data/ )
If you would like more information about Big Data and Hadoop Training, please click the orange "Request Info" button on top of this page.
Doug Cutting, ‘father’ of Hadoop, talks about big data tech evolution.ComputerWeekly.com, June 2, 2017.
The father of Hadoop - Doug Cutting spoke about the cyber security applications of the big data stack and hadoop’s evolution at the Strata Conference held in London.Doug spoke about a new cyber security application Apache Spot which follows a classic approach of having filters that scan for some specific sorts of behavior that somebody might have coded manually before prior attacks. It is difficult to catch new attacks but by building models that can identify usual behavior , it becomes easy to catch anomalies. You can more about Doug Cutting’s interview on big data tech evolution on ComputerWeekly.com.
(Source : http://www.computerweekly.com/news/450420002/Doug-Cutting-father-of-Hadoop-talks-about-big-data-tech-evolution )
For the complete list of big data companies and their salaries- CLICK HERE
NEC launches $10M centre for big data analytics in India. YourStory.com, June 7, 2017.
Japanese technology firm NEC Corporation announced the launch of Centre of Excellence for Analytics Platform and Solutions’ (COE-APS) in India to achieve a revenue of more than $100 million in a span of 3 years from its establishment. COE-APS will promote the services and solutions of NEC’s big data analytics platform named DPH i.e. Data Platform for Hadoop. It is an important step towards making the best use of NEC’s data platform for Hadoop to provide benefits of utilising big data analytics to government bodies and organizations in India
(Source : https://yourstory.com/2017/06/nec-launches-centre-big-data-analytics/ )
Intel teams up with DARPA to make HIVE Big Data platform a reality.Zdnet.com, June 7, 2017.
Intel and US Defense Advanced Research Projects Agency (DARPA) announced their collaboration to create a novel , powerful data handling computing platform that leverages Artificial Intelligence (AI) and Machine Learning (ML) technologies.DARPA’s program named Hierarchical Identify Verify & Exploit (HIVE) will surpass the current hardware used to handle big data by 1000 times in performance-per-watt. Intel’s technological expertise have been selected to push HIVE R&D further.
(Source : http://www.zdnet.com/article/intel-darpa-team-up-to-create-new-ai-data-platform/ )
IBM and Hortonworks go steady with OEM deal. Zdnet.com, June 13, 2017
IBM and the popular commercial hadoop distributor Hortonworks have signed multi-year OEM agreements for reselling particular hadoop and data science products. IBM will resell HDP as an official product and Hortonworks will resell BigSQL and IBM’s Data Science Experience (DSX that will be a part of its targeted versions of HDP.This is not a first such relationship for Hortonworks , earlier it has partnered with Microsoft for 5 years providing Hadoop pillar for Azure HDInsight. Now Hortonworks will be operating closely with 2 big players Microsoft and IBM.
(Source : http://www.zdnet.com/article/ibm-and-hortonworks-go-steady-with-oem-deal/ )
Big Data Gives China's Top 3 Internet Firms Big Leverage.voanews.com, June 14, 2017.
China’s three top internet giants - Baidu, Alibaba and Tencent are influencing a major section of the country’s business by collecting big data concerning consumer and social behavior of millions of people. Even utilities like retail chains and phone companies are also dipping themselves into the money pot from buyers who require information about buying preferences of potential customers so that they can design their strategies and products based on the data. Companies have got strong predictive power which comes from huge amounts of real-time and historical data that helps them predict the next thing a customer might want before the customer himself is aware of it.
(Source : https://www.voanews.com/a/big-data-gives-internet-big-leverage/3899830.html )
With Whole Foods Purchase, Amazon Just Bought a Playground for Big Data.Entrepreneur.com, June 21,2017.
Amazon acquired Whole Foods Markets for $42 per share in cash. The $13.7 billion acquisition is Amazon’s big play to enter the brick-and-mortar retail. Whole Foods acquisition will provide Amazon the opportunity to experiment with customer traffic management, in-store preference matching and retail analytics. The Whole Foods acquisition is not just about automating checkout but its all about bringing Amazon’s online big data analytics to the offline world.Amazon’s patent for “Physical Store Online Shopping Control” was awarded last week which will allow Amazon to limit searches on competitor websites while the customers are in a physical store. It will also allow Amazon to capture data about customers location, physical navigation, typical route through the store , tracking the time taken to pick specific items, and other information while inside a store.A customer will never will be alone , big data will always be right there with them.Retail presence analytics requires continuous data flow so that any big data application can identify meaningful trends and patterns. With Whole Foods acquisition, Amazon will now have offline brick-and-mortar retail network rich with analytics about what customers buy and how they shop.
(Source : https://www.entrepreneur.com/article/296075 )
Hadoop Engines Compete in Comcast Query ‘Smackdown. Datanami.com, June 22, 2017.
Comcast has made big investments in Hadoop technology to store and process petabytes of data that it collects from its operations. Most of the hadoop workload for $80 billion company involves execution of SQL queries to drive BI reporting. Comcast decided to put the hadoop engines against each other with no holds barred competition referring i as Comcast Query Smackdown. The contestants for the Comcast Snackdown were MapReduce2, Hive/LLAP 1.2, Tez 0.7, Spark 2.1, and Presto 0.175. The data for the competition was a collection of 1 TB of sequence, text, parquet and ORC files.Comcast ran 66 queries with each SQL engine having complete access to the cluster and the same tests were executed against each of them. Whenever a SQL engine failed a test, it was given a penalty of 10 minutes. The winner was Hive/LLAP SQL engine that won 44 of the TPC-DS queries taking a total execution time of 78.6 seconds which was 24% times faster than Presto and Tez.
(Source : https://www.datanami.com/2017/06/22/hadoop-engines-compete-comcast-query-smackdown/ )
Nasscom identifies key job roles in big data analytics.Indiatimes.com, June 24, 2017.
Nasscom as part of its reskilling initiative in the IT industry identified the key job roles in the big data analytics domain. In the days to comes data scientists, business analysts, data architects, data analysts and data integrators are expected to be the hottest and growing career options in the big data analytics domain. According to Nasscom’s Strategic Review 2017 , analytics market grew by 20% in 2017 which is likely to give rise novel trends in the analytics sector. The major factors driving growth in the big data analytics sector are -
- Making organizations and businesses more customer centric
- Creating innovative and novel business models
- Improving the operational performance.
(Source : http://telecom.economictimes.indiatimes.com/news/nasscom-identifies-key-job-roles-in-big-data-analytics/59295014 )
BMW, Volkswagen try to outrace Apple & Google as big data emerges as challenge for automakers.Indiatimes.com, June 26, 2017
BMW is building a giant new data centre ,Volkswagen is jumping into quantum computing and Bosch has unveiled its plan to construct a factory to build chips for self-driving cars- all these moves are being made to expand the efforts towards building compute capacity by collecting big data which will be required as automobiles digitise and become driverless. Cars will have to constantly communicate, absorb and analyse information from thousands of vehicles at once for making decisions on smooth traffic low, save fuel and avoid hazards. Companies like Volvo have been a pioneer in marrying automobiles and digital tech. Volvo unveiled its plan to install Android OS in new cars beginning in 2019. Volvo is also in talks with Uber to develop self driving cars. Big data is a challenge for automakers and in particular German companies as they target affluent customers who want latest technology.
(Source : http://economictimes.indiatimes.com/news/international/business/bmw-volkswagen-try-to-outrace-apple-google-as-big-data-emerges-as-challenge-for-automakers/articleshow/59316736.cms )
Big data investments to top $76B worldwide by 2020. HealthDataManagement.com, June 29, 2017.
Majority of the big companies have shifted their focus on big data over the last decade. Over $57Bn worth big data investments are likely to be made in 2017 alone, according to SNS Research. Big data investments are anticipated to grow at CAGR of 10% in the next 3 years reaching $76 billion by end of 2020. As the data is growing exponentially in terms of its size, complexity and versatility, so is the need to process the increasing amount of data growing at a rapid pace. Big Data has found its implementation in the most diverse market segments like healthcare, fraud detection, weather forecasting etc. Many countries have implemented big data analytics to solve most complex problems to rejuvenate their economies.
Big Data Tells Mortgage Traders an Amazing Amount About You.Bloomberg.com, June 29, 2017.
TheNumber, a recently launched New-York based startup pulls data from marketing firms, public loan filings, courthouses and dozens of other sources, and sells it to mortgage bond and loan traders. This means that if you have taken any home loans recently chances are that all the activities that you have made are available with these traders. This will help them identify the likelihood of paying the loans by borrowers. However, this is raising a concern among consumer-rights watchdogs that it is invading the privacy of borrowers. There may be a situation where you may have not been granted loan based on your gender or race. To some extent sharing of this information will benefit the mortgage traders, as they can take quick decisions which earlier use to take weeks or even months.