We use Cookies to enhance your experience on our website. By continuing your navigation, you accept the placement and use of Cookies. To learn more about Cookies or opt-out from these services please see our privacy policy

X

EventPage

Not Another Big Data Conference

Spend a day with some of the most knowledgeable folks working on large scale distributed systems as they share their tools, models, successes and failures.

Chat with fellow engineers and experts from Google, Spotify, Mesosphere and Criteo. Breakfast and lunch included, as well post-event cocktails on our rooftop deck featuring a 360° view of Paris.

Registration: €50. All proceeds will be donated to Simplon.co, an organization that fights against the digital divide in deprived communities of Paris.

Register Now

If you do not wish to donate to charity, please register here.

 

Speakers

Brice Arnould

Criteo

Barclay Osborn

Google

Justin Coffey

Criteo

Martin Gorner

Google

Rafal Wojdyla

Spotify

Guillaume Bort

Criteo

Francois Jehl

Francois Jehl Company

Josh Baer

Spotify

Stuart Pook

Criteo

Neil Conway

Mesosphere

Maxime Brugidou

Criteo

Serge Danzanvilliers

Criteo

Brice Arnould

Criteo

Brice built an availability tracking service at Google and elements of a PaaS at Gandi, then became techlead of MySQL Dublin at Google. Now devlead at Criteo for the data stream team, he has yet to produce something that he does not feel should have been done differently in retrospect.

Agenda

8:30 – 9:30

Welcome Breakfast (Sponsored by Google)

Registration will be held between 8:30 – 9:00 am. Please arrive early to register.

9:30 – 9:45

NABD Conference: What Does It All Mean?

9:45 – 10:25

Life Sized Kafka: Lessons Learnt Building a Service from an OSS Software

Kafka is a distributed system providing sharded replicated append-only files and managing consumer offsets. As is often the case for OSS-distributed systems, deployments are often tiny, some features don't scale, and some decisions must be taken to formalize an SLA. We will present here our approach, some of the mistakes we made and the novel tools we built.

10:25 – 11:10

Bad SRE: A Hyperbole in 12 Parts

SRE is hard at constant speed. It’s even harder to scale the team and systems. In this talk we’ll review how to organize and grow your team while maintaining good culture and quality of work. We’ll review some guiding principles, tools and thresholds, architectural details, and then follow up with a Q&A session. This is really what I think of as some of the most important parts I've learned about building and running huge systems.

11:10 – 11:50

Productivity and Hadoop: A Life's Work

Building data in Hadoop, especially for consumption by non- poorly-connected tools, generally involves using a mix of Hadoop frameworks driven by a dependency-driven scheduler. In this talk, I’ll present how to get hyper productive using our NIH scheduler, Langoustine, to program a mix of Hive and Scalding jobs for data transformations, exporting to a Vertica database for reporting.

11:50 – 12:30

No One Uses MapReduce at Google Anymore

The MapReduce paper, published by Google 10 years ago (2004!), sparked the parallel processing revolution and gave birth to countless open source and research projects. We have been busy since then and the MapReduce model is now officially obsolete. The new data processing models we use are called Flume Java (for the processing pipeline definition) and MillWheel for the real-time dataflow orchestration. They are available as a public tool called Cloud Dataflow, which allows you to specify both batch and real-time data processing pipelines and have them deployed and maintained automatically - and yes, dataflow can deploy lots of machines to handle Google-scale problems. What is the magic behind the scenes? What is the post-MapReduce dataflow model? What are the flow optimization algorithms? Read the papers or come for a walk through the algorithms with me. 

12:30 – 13:30

Lunch (Sponsored by Criteo)

13:30 – 14:10

Scio – Scala API for Apache Beam

Apache Beam (based on Google’s Dataflow Model) provides a simple, unified programming model for both batch and streaming data processing. If only it wasn’t so low-level and unfamiliar. Learn how Scio leverages Scala to provide more developer-friendly and safer API.

13:30 – 14:10

Playing Nice With SQL

14:15 – 14:55

Old is Good: Data Warehousing for Sub-second Query Times

Data warehousing is all but an anachronism today, but building out structured data sets still has its place when working on many TB scale datasets that need sub-second query performance. In this talk, François will present how he built out a Vertica cluster and the associated schema for just this purpose.

14:15 – 14:55

Real-time Data at Spotify: Leveling Up Features, Users and Developers

Hadoop-powered insights gave Spotify an early advantage in differentiating the user experience, but what if we could get that information even quicker? What if those insights and features could respond in real-time? With help from Google’s powerful cloud infrastructure tools like Pub/Sub and Dataflow, Spotify has been attempting to answer that question. This talk will go over what we’ve built and learned so far.

15:00 – 15:40

Migrating 39PB of Data to a New Cluster in a New DC

Criteo’s primary Hadoop cluster has 26K cores, 39 PB of raw storage and 11 TB RAM and runs up to 90,000 jobs per day. Out of both scaling and resiliency concerns, Criteo needed to build out a new cluster in a different DC. In this talk, we will discuss the evaluation of server and network hardware, the build out and bootstrapping of CDH5x across 800 servers via chef along with the speed bumps encountered migrating data and jobs from the existing CDH4x cluster.

15:00 – 15:40

Service Backplanes for the Modern Data Center: What, Why and How

Modern data centers and clouds allow cluster resources to be mapped to applications dynamically, which can dramatically improve utilization and deployment agility. However, deploying off-the-shelf software in this environment is not easy: devops staff typically need to write "backplanes" to deploy, manage and upgrade each software package that is deployed at scale. Despite their ad-hoc nature, these backplanes quickly grow in complexity and importance: a bug in the application backplane often means a production outage.

What is needed is a consistent framework for designing "backplanes": controllers that mediate between cluster resource APIs and server software. Backplanes capture much of the operational complexity of running a software package in production, allowing off-the-shelf software to be turned into an elastic service. In this talk, we motivate the need for service backplanes, discuss the functionality that belongs in a backplane, and consider how backplanes should be built, using Apache Mesos as an example. 

15:45 – 16:00

Coffee Break

16:00 – 16:40

Scaling Automation Up to a Global Mesos-scale Infrastructure

Five years ago, Criteo was already a successful startup. However, most of its (bare metal) servers were set up by hand from cabling to app deployments via drag-and-drop of DLLs over remote desktop connections. Today, we support a global infrastructure with abstraction layers that help us scale both our operations and our business. How did we get there? What went wrong? You’ll find out.

16:00 – 16:40

A History of Streaming at Criteo

16:45 – 17:25

Optimizing Hadoop Jobs from the Outside

18:00

Cocktail Drinks (Sponsored by HPE Vertical)

EventPage

Find the Event

Criteo Paris
32 Rue Blanche
75009 Paris France

 

Contact Us

For more information about this event, please contact: 
R&D Events at r&devents@criteo.com

Be Social

Continue the conversation with Criteo 
#criteo