Over 35 years of entertainment

Optimising big data queries for large volumes of complex data

22 July 2022 | Noor Khan

Optimising big data queries for large volumes of complex data in Amazon Athena

Key Challenges

Our client's existing system was unable to handle the large volumes of data they were dealing with on a daily basis and required a flexible, robust data reporting engine for time-efficient results.

Key Details

Service

Data Engineering

Technology

Cloud Watch, Cloud Trail, IAM, DynamoDB, SQL, Athena, S3 Data Lake, Angular, API Gateway, Lambda, AWS Batch

Industry

Media

Sector

Broadcasting

Key results

  • Ability to export a large amount of data to Excel
  • Multi-factor authentication to validate users for security purposes
  • Ability to save preferences and default settings for searches increasing efficiency when carrying out future searches
  • Speedy and accurate data results
  • Email alerts to inform users when data results are ready so they do not have to wait around
  • High level of flexibility when carrying out searches with capabilities of combination searches

Multinational broadcasting company

Entertaining and inspiring audiences since 1984

Our client are a well-established, multi-national broadcasting company reaching 300 million subscribers across 150 countries, through 50 channels and in 37 languages. They deal with large sets of complex data audience data ranging from streaming to TV audiences. They need to be able to efficiently carry out searches, for research and sales purposes. Their data sets are sourced from Nielsen AMRLD which is stored in their data warehouse.

Optimising big data queries for large volumes of complex data

Program Search and Market Break Ranker tool

Huge data requires a powerful system

Our client’s existing data search tool performed the search queries extremely slow, frequently timed out because of the large volumes of complex data and did not offer the flexibility required. They needed a tool that would help them gain insights from large amounts of data to help support the research and sales efforts of the business.

Optimising big data queries for large volumes of complex data

Flexible, user-friendly tool that handles data searches on large volumes of complex data

Making big data searches, speedy

The client was looking for an improved data search tool that allowed them to have a Program Search as well as a Market Break Ranker. These two different components were essentially search engines for the two types of data the company dealt with .The Ardent engineering team took on board the requirements and created a tool that optimised big data queries for large volumes of complex data with a user-friendly interface, using API Gateway and Angular technologies. We offered flexibility by adding filters and drop-down menus, allowing the users to select multiple categories for the data results see t.As the client wanted the ability to be able to carry out Boolean logic combination searches, we needed to build a system that could search vast data sets to pull out relevant data results. We were able to provide a solution on Lambda and then moved to AWS Batch to ensure a quick and efficient search of the data.

More on our technology partners.

Optimising big data queries for large volumes of complex data

Ardent's data engineering expertise

Drill down searches with boolean logic

Our expert engineers made recommendations on boolean logic to streamline data searches as we have a vast amount of experience working on a similar scale of data and clients.

Optimised big data queries

Searching huge data sets quickly and easily

Our experienced engineers delivered a fit for purpose data solution, allowing our client to optimise big data queries on large, complex big data with a number of features to make the search process easy and straightforward. This enabled our client to have a quick turnaround time on analysis of the data (household viewing figures), increasing efficiency in the data search process. Overall, they were able to save time, be efficient, and save costs allowing them to focus on other important business goals.

Explore our data engineering services or get in touch to find out how we can help you optimise your big data.

More Success Stories

recruitment software product

Success Story

Getting the job done

Technology | Recruitment, Software

Our client were looking to upgrade their existing software which was windows based and presented multiple limitations. They required expert skills to design and deliver software which would make the subscription-based software easier to deliver to end customer with improved features and functionality. [...]

Read More... from Over 35 years of entertainment

Keeping data consistent with operational monitoring and support

Success Story

Making data science efficient

Market Research | Entertainment, Broadcasting

Our client required high data availability with no delays therefore, they needed to ensure that their data pipeline consisting of various sources were continuously and consistently running without errors. [...]

Read More... from Over 35 years of entertainment

Robust, scalable data pipelines with AWS infrastructure

Success Story

Powerful insights driving growth for global brands

Market Research | Entertainment, Media

Our client were looking to collate data from various sources with high data accessibility and accuracy to provide a speedy turnaround time for their end customer. [...]

Read More... from Over 35 years of entertainment

Ardent Insights

Data Warehousing Architecture: Azure SQL vs Redshift

Data Warehousing Architecture: Azure SQL vs Redshift

There are quite a few different data warehouse technologies to choose from, and knowing where to start, what to look for, and what data warehouse service is going to be best for your business is key. Deciding on a data warehousing technology that is suitable to your business needs, goals and objective is essential as [...]

Read More... from Over 35 years of entertainment

Data Warehouse vs Database: The key differences

Data Warehouse vs Database: The key differences

Cloud technology and advances in hybrid-storage technologies have meant that data storage is no longer left entirely to internal equipment and a single company-owned server. Finding a database management solution that works with the size and scale of your needs is not a task to be overlooked. [...]

Read More... from Over 35 years of entertainment

Building data pipelines – a starting guide

Building data pipelines – a starting guide

Data visibility can be a huge driving factor for organisation growth. Poor data visibility can lead to a lack of compliance with data security, difficulty in understanding business performance and increased complexity in dealing with system performance issues. Developing secure, robust and scalable data pipelines can empower businesses to gain data visibility by connecting the [...]

Read More... from Over 35 years of entertainment