We talk with our flink-loving colleague Lars about stateful and queryable state stream processing.

$('#2017_11_05_ib18-statefull-stream-processing_player').podlovewebplayer({poster: 'http://qnib.org/insert-buzzword/img/logo-360x360.png',subtitle: 'We talk with our flink-loving colleague Lars about stateful and queryable state stream processing.',chapters: '00:00:21.832 Intro'+"\n"+'00:02:02.008 Who is Lars?'+"\n"+'00:03:45.022 Lars knows his BigData'+"\n"+'00:06:35.542 Hagen has a question'+"\n"+'00:08:15.341 The Basics: What is MapReduce?'+"\n"+'00:16:02.190 The Basics: What is Cascading?'+"\n"+'00:18:37.197 The Basics: Problems with MapReduce'+"\n"+'00:23:36.264 Better safe then sorry'+"\n"+'00:27:00.104 We want real-time and graph-processing!'+"\n"+'00:28:43.994 From batch to stream'+"\n"+'00:34:30.518 Bounded and Unbounded'+"\n"+'00:42:31.923 Time Windows'+"\n"+'00:46:31.471 ACID transactions in stream processors'+"\n"+'00:53:08.223 Flink'+"\n"+'00:57:37.325 Flink State Partitioning'+"\n"+'01:02:37.476 Flink: Source/Operators/Sink'+"\n"+'01:11:23.097 How to write a flink job'+"\n"+'01:14:56.870 How to update the stream jobs'+"\n"+'01:18:46.222 Hagens rant about the update procedure'+"\n"+'01:19:55.780 Outro: Why are we not rich?'+"\n"+'01:20:52.431 Capitalism Tips'+"\n"+'01:26:26.556 Good Bye',duration: '01:27:56',permalink: 'http://qnib.org/insert-buzzword/2017/11/05/ib18-statefull-stream-processing.html',title: 'IB18 Stateful Stream Processing',alwaysShowHours: true,startVolume: 0.8,width: 'auto',summaryVisible: false,timecontrolsVisible: false,chaptersVisible: true,sharebuttonsVisible: false});

Initial Agenda

As always, we derail a bit from the original plan… :)

Housekeeping

About Lars

• Why do you code (History)
• What do you like about coding
• What’s your interest in stream processing

Background Stream Processing

• Unbounded data sets
  • why can we not use MapReduce anymore
• Time windows (Event & Processing Time)
• How to do transaction’s here, limitations
• Event Sourcing vs Stream processing, two of a kind ?

Flink

• Background
  • Runs on Hadoop, or stand alone on Container infrastructure even ??
• What to know
  • Partitioning
  • Capacity planning
  • Consistency
  • What to write
• Getting started
  • Github
  • Community Docs

Kafka Streams API & Ksql

• How to run it, containers latest version
• How to implement a rest things that queries local state
• What to know
  • Partitioning
  • Capacity planning
  • Consistency
  • What to write
• Getting started
  • Github
  • Community Docs

Outro

• Why are we not rich?
• What conferences to attend in 2018 ? Calendar
• Food sports health Tips
• Capitalism Tips
• Good bye