The Big Book of Data Engineering 2nd Edition

Organizations realize the value data plays as a strategic asset for various business-related initiatives, such as growing revenues, improving the customer experience, operating efficiently or improving a product or service. However, accessing and managing data for these initiatives has become increasingly complex. Most of the complexity has arisen with the explosion of data volumes and data types, with organizations amassing an estimated 80% of data in unstructured and semi-structured format. As the collection of data continues to increase, 73% of the data goes unused for analytics or decision-making. In order to try and decrease this percentage and make more data usable, data engineering teams are responsible for building data pipelines to efficiently and reliably deliver data. But the process of building these complex data pipelines comes with a number of difficulties:

• In order to get data into a data lake, data engineers are required to spend immense time hand-coding repetitive data ingestion tasks
• Since data platforms continuously change, data engineers spend time building and maintaining, and then rebuilding, complex scalable infrastructure
• As data pipelines become more complex, data engineers are required to find reliable tools to orchestrate these pipelines
• With the increasing importance of real-time data, low latency data pipelines are required, which are even more difficult to build and maintain
• Finally, with all pipelines written, data engineers need to constantly focus on performance, tuning pipelines and architectures to meet SLAs

White Paper from