Data Engineering Bootcamp 15

1. Week

• Linux Basic, Bash Script (Ön hazırlık)
• Git Basic (Ön hazırlık)
• Crontab (Ön hazırlık)

Bonus:

• Azure Data Engineering Intro
  

2. Week

• Postgresql & SQL Basics (Ön hazırlık)
• Docker & Kubernetes (Ön hazırlık)
• Python Programming (Ön hazırlık)

Bonus:

• Airflow MCP Server DAG Management
  

3. Week - Advanced Python for Data Engineers

• Decorators: wrapping functions for retry, logging, timing
• Generators & lazy evaluation for large datasets
• Context Managers for resource handling (DB connections, files)
• Concurrency: Asyncio vs. Multiprocessing vs. Multithreading
• When to use which: I/O-bound vs. CPU-bound tasks
• Type Hinting & Pydantic for data validation Logging patterns for observable pipelines

Hands-On:

• Build a REST API ingestor with retries
• Database operations with Python
• Incremental loading strategies
• Idempotent pipeline design Pydantic schema validation

Bonus: 

• Block, File, and Object Storage

4. Week - Advanced SQL & Database Internals

• Complex Joins: self-joins, cross joins, anti-joins
• Window Functions: ROW_NUMBER, RANK, LAG/LEAD, running aggregates
• CTEs & recursive queries for hierarchical data
• Query lifecycle: parse → plan → optimize → execute
• Indexing strategies: B-tree, hash, GIN, GiST — when to use which
• Reading & interpreting Execution Plans (EXPLAIN ANALYZE)
• Query optimization: predicate pushdown, join reordering, statistics

Hands-On:

• Window function exercises on real datasets
• CTE-based complex reporting queries
• Index creation & performance benchmarking
• EXPLAIN ANALYZE deep-dive
• Query tuning workshop

Bonus: 

• Database Operations with Python

5. Week - Data Ingestion — Batch Patterns

• Ingestion in the DE lifecycle: sources → storage
• Batch ingestion patterns: time-interval, size-interval
• ETL vs. ELT — architectural implications
• Change Data Capture (CDC) fundamentals
• Idempotency, ordering, late-arriving data
• Connectivity: JDBC/ODBC, REST APIs, SSH/SCP/SFTP
• Data migration strategies
• Event streaming infrastructure — why distributed messaging?

Hands-On:

• Python RDBMS ingestor (JDBC/SQLAlchemy/SQLModel)
• REST API batch ingestion pipeline
• Ingestion from Object Storage
• Streaming Data Ingestion: Apache Kafka
• Multinode Kafka setup on Docker
• Kafka topics, partitions, console producer/consumer
• Consumer groups & offset management

Bonus: 

• Python Tests (Unit, Integration)

6. Week - Streaming & Event-Driven Architecture

• IEvent-driven architecture principles
• Pub/sub model vs. point-to-point messaging
• Consumer groups & partition assignment strategies
• Ordering guarantees & exactly-once semantics
• Backpressure & flow control
• Schema evolution in streaming systems
• Producers, consumers, admin operations as patterns

Hands-On:

• Kafka Python Driver: Producer
• Kafka Python Driver: Consumer
• Kafka Python Driver: Admin
• Data generator: stream your dataset

Bonus: 

• Change Data Capture

7. Week - Distributed Computation & Transformation

• MapReduce paradigm — divide and conquer
• Lazy evaluation & DAGs in computation
• DataFrames as a universal abstraction
• SQL on big data — declarative vs. imperative
• Batch transformation patterns
• Query lifecycle: parse → optimize → execute
• Data wrangling: cleaning, deduplication, type casting

Hands-On:

• Apache Spark intro & docs
• Local Spark setup
• Spark Docker-compose
• Spark DataFrame + Spark SQL

Bonus: 

• Query big data with Trino

8. Week - Data Interoperability & External Sources

• Data serialization & deserialization
• Format trade-offs: row vs. columnar, text vs. binary
• Reading from heterogeneous sources: DB, object store, file
• Federation: querying across systems
• Data virtualization engines (Presto, Trino, Drill)
• External tables & cross-system connectivity

Hands-On:

• Apache Spark Built-in sources: Parquet, JSON, CSV, ORC
• Apache Spark External: Avro, PostgreSQL, S3

Bonus: 

• SQL window functions

9. Week - Scaling & Performance Engineering

• Spark cluster resource management concepts
• Spark job scheduling & resource negotiation
• Spark shuffle operations — the hidden cost
• Partitioning strategies for performance
• Caching & persistence levels
• Broadcast joins & data skew mitigation
• Cost-based optimization

Hands-On:

• Spark cluster managers: YARN, Standalone, K8s
• Submit Spark applications
• Spark optimization exercises

Bonus: 

• Databricks vs Snowflake

10. Week - Stream Processing Patterns

• Micro-batch vs. true streaming (Spark vs. Flink)
• Stateful vs. stateless stream processing
• Windowing: tumbling, sliding, session
• Watermarks & handling late data
• Stream enrichment patterns
• Sink patterns: ForEachBatch, file, Kafka
• Exactly-once delivery in stream processing

Hands-On:

• Spark Streaming: file source/sink
• Stateful operations
• ForEachBatch sink
• Kafka source & sink

Bonus: 

• Creating real-time dashboards with Apache Superset

11. Week - Data Modeling & Dimensional Design

• Data modeling lifecycle: conceptual → logical → physical
• Normalization: 1NF, 2NF, 3NF — don't repeat yourself
• Dimensional modeling: Star & Snowflake schemas
• Fact & dimension tables, cardinality
• Slowly Changing Dimensions (Type 1, 2, 3)
• Data Vault approach
• DWH vs. Data Lake vs. Lakehouse
• Open table formats: Delta, Iceberg, Hudi

Hands-On:

• DWH approach design
• Tools for data modeling
• Example showing all phases: conceptual → logical → physical

Bonus: 

• DBT

12. Week - Orchestration & Workflow Management

• DAGs & dependency management
• Scheduling strategies: cron, event-driven, sensor
• Pipelines as code — why it matters
• Idempotent task design
• Retry strategies & failure handling
• Monitoring & alerting for pipelines
• Orchestrator trade-offs: Airflow vs. Prefect vs. Dagster

Hands-On:

• Airflow: Dummy, Bash Operators
• Airflow: PythonOperator
• DB connection + SQLExecuteQueryOperator
• SSH connection

Bonus: 

• Clickhouse

13. Week - Advanced Orchestration & Pipeline Patterns

• Branching & conditional logic in pipelines
• Notification & alerting patterns
• Containerized task execution
• Orchestrating heterogeneous workloads (Python/Spark/Docker)
• API ingestion patterns
• Object storage read/write with Python Pandas
• Pipeline observability & SLAs

Hands-On:

• Airflow BranchOperator
• Airflow email notification
• Airflow DockerOperator
• Orchestrate Spark applications
• Pandas + object storage & DB

Bonus: 

• BigQuery

14. Week - DataOps & CI/CD for Data

• DataOps principles: rapid innovation + high quality
• CI/CD for data pipelines — not just for code
• Testing strategies: unit, integration, data quality
• Infrastructure as Code (Terraform, Ansible)
• Data governance & accountability
• Data observability: lineage, freshness, volume, schema
• Collaboration across people, tech, environments

Hands-On:

• CI/CD Pipeline with Airflow
• dirty_store_transactions data pipeline
• DataOps process implementation

Bonus: 

• Schema Registries in Streaming

15. Week - Data Management & NoSQL Paradigms

• Data management lifecycle: govern, catalog, discover
• Metadata types: business, technical, operational
• Data catalog & discoverability
• NoSQL paradigms: document, key-value, wide-column, graph
• Search & analytics engines — inverted index
• When to use NoSQL vs. RDBMS
• Security: auth, encryption, GDPR/KVKK, access control

Hands-On:

• Elasticsearch setup with Docker
• Indexing with Python and Spark
• Using Elasticsearch as vector database
• Vector embeddings
• Full-text, vector and hybrid search
• Creating visualizations on Kibana dashboard

Bonus: 

• Spark ML

16. Week - Cloud-Native Data Engineering

• Cloud data architecture: IaaS, PaaS, SaaS trade-offs
• Serverless computing for data workloads
• Managed services vs. self-hosted
• Cloud security & shared responsibility model
• FinOps: cost optimization for data pipelines
• Build vs. buy decision framework
• Multi-cloud & cloud-agnostic design

Hands-On:

• AWS core services overview
• AWS Athena: serverless query
• AWS Lambda: event-driven compute
• AWS Glue: managed ETL

Bonus: 

• Infrastructure as Code (IaC - Terraform)

17. Week - Capstone Project Preps

• Self-study

18. Week - Capstone Project Presentations

• Final Projects (self-study)
• Final Project presentations

Example Projects:

• Real-time Sensor Dashboard (Kafka → Spark → Elasticsearch → Kibana)
• Real-time ML Prediction (Kafka → Spark → SparkML -> Kafka)
• Change Data Capture Pipeline (Debezium → Kafka Connect → Kafka → Spark → S3)
• Lakehouse with Orchestration (Airflow → Spark → Delta Lake → S3)
• Batch RDBMS to Lakehouse (Python Ingestor → Airflow → Spark → S3)
• Analytics Engineering (dbt Core → BigQuery)
• Students can present their own projects as well