In an era of digital transformation of enterprises, there are several questions that have arisen-
- How can business intelligence provide real time insights?
- How can business intelligence scale and analyse the growing data heap?
- How can business intelligence meet changing business needs?
Business Intelligence (BI) combines human knowledge, technologies like distributed computing, and Artificial Intelligence, big data and analytics to augment business decisions for driving enterprise’s success. Business Intelligence programs transform big data from an enterprise data warehouse into actionable and usable business information with the use of various business intelligence tools.Read on to understand how big data is transforming business intelligence by transforming the productivity, ability to innovate and succeed in ways that where unimaginable.
Solocal Group (previously known as PagesJaunes Groupe) reaches 90% of French citizens via its collection of mobile, digital, print and various social products. Solocal was ranked among the top 10 global companies in 2013 based on it ROI, with close to $1.5 billion revenue (59% of the revenue) generated per annum merely from digital offerings.
“Solocal is a company that Yellow Media had always admired in terms of their ability to grow their online audiences.”-said Yellow chairman and interim CEO Robert MacLellan.
"Solocal remarkably made a transformation from traditional print publisher to one of the world’s best local search companies. Solocal has taken big data to the next stage of BI by designing a novel vision of BI with the open source distributed computing framework Hadoop. It replaced its traditional BI structure by integrating big data and Hadoop."-April 13, 2015, Silicon.fr –“Big Data: How to tame Hadoop Solocal”
So what is big at Solocal?
At Solocal there are 5,000,000 researches per day in the website. Just to show you how it will be big in terms of data and what can be analysed. Also the mobile app downloads is more than 17,000. There are more than 800,000 search results from mobile apps.
At DeZyre we had the pleasure to invite Abed Ajraou , the Director of the BI & Big Data in Solocal Group (Yellow Pages in France) to speak about the digital transformation from BI to Big Data. This industry interview series with Abed Ajraou focuses on the design and deployment of Solocal’s novel data architectural model using big data and Hadoop.
So what is BI?
BI is not a tool, a report or a database. BI is not like Oracle or SQL Server. BI is a whole framework.
The goal of BI is to create intelligence through Data. BI is exactly that -to give the right data to the right person with the right tool at the right time.
BI is a complete environment –we have Data Architecture, we have this staging area, data warehouse, data mart. But there is also Data Quality. Data Quality can be across all the companies.
There is Data Quality inside the BI as a lot of data is being processed inside the company. So there is a lot of data cleaning and data profiling that needs to be done. It also detects any problems that might be there in the data – so there is some data quality layer. There is also a lot of transformation – so people know that you are using a lot of ETL. There is Master Function Data and Meta Function Data. Then there is the Access layer to bring the reports to the end users. Also some people are missing out on the Business Alignments – we need to be aligning to the Business Strategy of the company. If the BI strategy is not aligned to business requirements – then BI is of no use. There is also a lot of Change management – you have to get the end users to the change operations and to give them a better understanding of what we are bringing to them at the Access layer. Then there is the ‘Foundations’. Foundations in BI is more about the methodology, and the technical watch and what we can do in BI which is in accordance with the Rules& Regulations of the company.
So let’s discuss what the problem is.
In the data transformation we saw lot of limitation with this kind of BI architecture.
So what are the pains of the BI?
- A data warehouse with more than 50 TB is very difficult to maintain. At Solocal there are more than 100 TBs of data – so when we are using relational databases from IBM, and we want to move, or change or add a new measure or new a new column or new dimensions to the fact table, what happens? To change one table, which contains 1 TB – takes a lot of time and is very complicated. We know that data warehouse is very big and a very complicated tool to maintain and to meet Big Data problems.
- In BI we just consider structured data. We never imagined that we can also analyze some videos or logs or similar semi structured data. We just focused on structured data on the databases. This is something that we completely miss out on – as the BI architecture did not support this.
- The next pain point is the batch treatment. There is still much to be done in terms of transforming the data, cleaning the data, to allocate the data. And this takes time, as there is no option of getting the data to the end user in real time. All the BI reports are pushed out at the end of the day.
The business intelligence framework is completely at risk. For example, say we get a project on analysing twitter data. In terms of Business Alignments, we have to stick to the data and after that we have to supply internal architecture. So the source system is completely at risk because we have external data, we don’t know how to analyse it, how to supply it in other layers. The data warehouse, change management and the access layer is completely at risk, because we do not know how to prorate this kind of data, we do not know how to do text analysis.
So these are the reasons why Solocal group had to do some major BI transformation.
Originally the BI architecture was built in the 1980s and was called Traditional BI. This is fine, this is what can be done and has been done in all the companies. But as we saw earlier – this kind of BI architecture is completely at risk. Around 2004-2005, there emerged Departmental BI, it is named as such because it works in various departments in the company. This kind of BI architecture detects all the data and stores it in the memory – this eliminated the data architecture. Instead it began to do data association which made data visualizations and data discovery very quick. But it is limiting as it only works in departments of small companies. It cannot work in the whole company or MNCs, as this kind of architecture takes a long process. This is implemented at Solocal.
But now Solocal is looking improve the maturity of Data Architecture in the company. To analyse all structured and unstructured data, they need to bring in a data architecture that will analyse all internal and external data. Here Solocal has to use the MPP transformation and NoSQL databases and some tools to analyse text documents, i.e. Text Agent Mining. This is something that was completely missing in the BI architecture. We have to propose this kind of vision in the new BI architecture.
The new BI architecture helped Solocal get down to some real time analysis – which was completely impossible in the traditional BI.
At Solocal, Cloudera Data Hub was used for Extended Data Warehouse and the entire data warehouse was moved to Cloudera.
Using Impala has increased our performance at the time of reporting for the end users.
In BI – there is a need to use ETL on top of Hadoop as there is not much scripting. Solocal uses SyncSort for its ETL purposes. This is the best choice with the old architecture and the new BI architecture with Hadoop.
For data visualization and data story telling – Zoomdata is used by Solocal. This gives a lot of possibilities to analyse data.
Solocal has completely introduced a new BI architecture that deals with the 3 Vs of Big data. They take all the structured data and the unstructured data and bring it into the open data group in Cloudera Hadoop Stack. After that they put all the Open data marts on top of the data group – which brings together the intelligence of the data by crossing all the data marts and creating intelligence for data visualization on top of that.
So once the new age BI architecture is in place, it is time to understand why it was done –
- For your CRM – this kind of BI architecture will allow you to analyze in depth on what you customer is saying about your product. For example, this BI architecture will allow you to analyze Facebook posts texts for sentiment analysis.
- For launching a new product, data from twitter can be analyzed to find out who will be the buyers of the new product, based on their tweet preferences.
With this kind of BI architecture in place – it is possible to ask Bigger Questions.
With thisBI architecture in place, Solocal created two BI applications.
For the complete list of big data companies and their salaries- CLICK HERE
1.Real Time Business
This application provides a fresh BI report every 5 minutes – on how much Solocal sells in the business.
2.Search Power Selector
Image Credit: slideshare
This combines internal data and external data from Google. This tool allows you to understand in which location you can introduce sale offers and in which locations it would not help to do so.