Skip to main content
06 June 2017

How to manage big data

By Joseph Moore

The concept of “Big Data” has been with us since the early 00’s.  Nevertheless, in 2017 the reality of its importance, volume and how it can take up our day to day work is now very apparent. We have all heard the phrase Big Data but what does it actually mean?  In 2012, Gartner updated its definition as follows: "Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimisation.

The three V’s concept is essential for understanding Big Data and its importance to aviation in 2017. Big Data is now intrinsic to each person’s job in the industry - flight, passenger journey, logistics to name only a few which all produce vast numbers of Big Data.  The three Vs are:


•    Volume: MB/GB/TB/PB

•    Variety: Database/ Photo/Audio/Web/ Mobile/ Video

•    Velocity: Batch/ Periodic/ Near real time/ real time

Since the 1970’s when data was first tracked in aviation there has been an explosion of data., If we looked at all the data from when it was first created until the year 2000, it would be less that we create in a minute now!  Currently, a single flight will create 1TB of data, broken down that is 62.5 I-phones (16GB) of information from each flight that could be processed and received.  We are struggling to cope with the three V’s as we don’t know how to process or analyse all the data.

ICAO request for more data and a greater sharing of data from all aviation organisations which, in turn, they hope promotes a better safety culture and then hopefully a reduction in incidents. In theory, this a great idea and a positive step to an overall improvement in culture.  However, the more documents, audits and reports that we all have to do in our day to day job can make us feel overwhelmed by data.  It’s not at the same TB/PB as big data or at the same rate but it will be affecting our efficiency to do our job well.

Prior to joining the Aviation team at Ideagen I was a consultant for 8  years for the company and spent my time trying to help hundreds of customers to translate their existing manual systems and legacy systems into something that would add value to their business and provide meaningful data to the decision makers within the organisation.

I have visited many customers over the years that have been truly overwhelmed with the volume of data that these systems create, even more so when they have inherited it from previous members of staff. What may have made sense to the previous person suddenly isn't telling us what we need to know. KPI's may have changed along with senior managers who are looking to drill down into this data in different ways. In some cases, I've seen customers archive years of data because the task is just too overwhelming.

I met a customer who was in this situation recently - they had inherited a host of open audit findings stretching back many years and a significant number of Occurrences that hadn't been investigated and were still awaiting action. To make matters worse, the issues hadn't been properly risk assessed or classified to prioritise what was in front of him. He had no trace of the actions his predecessor may have sent out by email and the spreadsheets in front of him were becoming more overwhelming by the day, with new audits and occurrences producing more data every day.

It was great to be able to present him with a solution, the expert guidance of our professional services team coupled with the implementation of a new system to help him begin the journey of getting things prioritised, actioned and properly analysed. All of his data will be validated before it's entered into the system to make sure we are not copying a legacy system that was already ineffective. Going forward we are making sure he is able to effectively track the QHSE KPI's and the system will be regularly health checked to make sure we stay strategically aligned.

Does this sound familiar? We would be glad to introduce you to some of our customers to share their experiences first hand with you about how Ideagen’s solutions have driven their business forward and taken a little bit of pressure away from Quality and Safety teams within their organisations.

For more information on our Aviation solutions please click here

Back To Top