Blockchain Data Engineering Programme Curriculum

Week 1

Foundations & Engineering Environment

Orientation, tooling, and reproducible local environments.

Tools: Python · Docker · SQL

Learning objectives

• Understand the Blockchain Data Engineering workflow
• Set up a reproducible development environment
• Write foundational Python and SQL for pipeline work

Sessions

1. S1

   Programme orientation & systems thinking

   Map the full path from chain data to product and set programme expectations.

2. S2

   Python & SQL for data engineers

   Practical language foundations used throughout the cohort.

3. S3

   Docker fundamentals for data work

   Containerise a simple service and understand why reproducibility matters.

Practical exercises

• Run a baseline Docker environment
• Query sample tabular data with SQL
• Structure a project repository for engineering work

Week 2

Blockchain Data Extraction

Pull real onchain data using Web3.py and structure extraction jobs.

Tools: Python · Web3.py · Docker

Learning objectives

• Connect to blockchain RPC endpoints
• Extract transactions, logs and contract events
• Persist raw extraction outputs reliably

Sessions

1. S4

   Web3.py & RPC access patterns

   Connect, request and handle blockchain node responses.

2. S5

   Transactions, receipts and event logs

   Understand the data shapes you will extract and store.

3. S6

   Building your first extraction job

   Ship a working job that pulls and stores raw onchain data.

Practical exercises

• Fetch blocks and transactions with Web3.py
• Decode a basic event log set
• Containerise an extraction job

Project: Raw extraction service for a chosen contract or token set.

Week 3

Data Pipelines

Turn extraction into a pipeline with validation and multi-service compose setups.

Tools: Python · Docker Compose · PostgreSQL

Learning objectives

• Design extract → validate → load steps
• Run multi-service environments with Docker Compose
• Handle failures and retries in batch jobs

Sessions

1. S7

   Pipeline architecture for onchain data

   Break work into dependable stages instead of one-off scripts.

2. S8

   Docker Compose for data services

   Coordinate application and database containers locally.

3. S9

   Validation, idempotency and load

   Make ingestion safer to re-run and easier to debug.

Practical exercises

• Compose app + database services
• Add validation before load
• Log pipeline run metadata

Week 4

Data Warehousing with PostgreSQL

Model and store structured blockchain data for analytical workloads.

Tools: PostgreSQL · SQL · Docker

Learning objectives

• Apply warehouse concepts to onchain entities
• Design schemas for transfers and events
• Use indexing to support analytical queries

Sessions

1. S10

   Warehouse thinking for blockchain data

   Move from dumping JSON to designing durable tables.

2. S11

   Schema design for transfers & events

   Normalize addresses, tokens and transfer facts.

3. S12

   Indexing and query performance basics

   Keep analytical queries usable as volume grows.

Practical exercises

• Model ERC-20 transfer tables
• Add indexes for common query patterns
• Inspect schemas with database tooling

Project: Structured warehouse schema for extracted blockchain activity.

Week 5

Analytics Engineering with dbt

Transform raw tables into cleaned, tested analytics-ready models.

Tools: dbt · PostgreSQL · SQL

Learning objectives

• Structure a dbt project for onchain datasets
• Build staging and mart layers
• Add tests to transformation models

Sessions

1. S13

   dbt project structure

   Sources, models and the path from raw to marts.

2. S14

   Staging models for blockchain tables

   Clean and standardize fields before business logic.

3. S15

   Testing and documenting models

   Make transformations trustworthy and shareable.

Practical exercises

• Create staging models from raw tables
• Build transfer marts
• Add uniqueness and not-null tests

Week 6

Normalization & Protocol Datasets

Turn messy chain records into protocol-ready analytical datasets.

Tools: dbt · SQL · Python

Learning objectives

• Normalize addresses and token metadata
• Model protocol-relevant entities
• Produce datasets ready for analytics and research

Sessions

1. S16

   Normalization patterns for onchain data

   Reduce duplication and inconsistency across entities.

2. S17

   Protocol analytics datasets

   Shape tables that analysts and apps can actually use.

3. S18

   Review: transformation quality

   Critique models, tests and documentation as a cohort.

Practical exercises

• Build address dimension logic
• Create protocol activity aggregates
• Review model lineage

Project: Analytics-ready protocol dataset from your warehouse.

Week 7

Orchestration with Prefect & Airflow

Schedule and coordinate multi-step pipelines reliably.

Tools: Prefect · Apache Airflow · Docker

Learning objectives

• Express pipelines as orchestrated workflows
• Understand Airflow DAGs and dependencies
• Handle retries, schedules and observability basics

Sessions

1. S19

   Why orchestration matters

   From manual scripts to dependable scheduled systems.

2. S20

   Prefect for practical workflows

   Compose flows that match your existing pipeline stages.

3. S21

   Apache Airflow DAGs

   Model dependencies and schedules with industry-standard tooling.

Practical exercises

• Build a Prefect flow for extract → load → transform
• Author a simple Airflow DAG
• Simulate and recover from a failed task

Week 8

Streaming Concepts & Data APIs

Introduce Kafka-style streaming ideas and expose data via FastAPI.

Tools: Apache Kafka · FastAPI · PostgreSQL

Learning objectives

• Understand streaming vs batch for blockchain data
• Work with Kafka concepts (topics, producers, consumers)
• Serve warehouse data through a FastAPI service

Sessions

1. S22

   Batch vs streaming for onchain systems

   Choose the right pattern for latency and reliability.

2. S23

   Apache Kafka fundamentals

   Topics, producers, consumers and where they fit your stack.

3. S24

   Building blockchain data APIs with FastAPI

   Turn infrastructure into a service applications can call.

Practical exercises

• Sketch a streaming architecture for event ingestion
• Produce/consume a sample topic locally
• Ship a FastAPI endpoint over warehouse tables

Project: Data API over your analytics-ready tables.

Week 9

Cloud & Deployment

Deploy practical projects beyond your laptop.

Tools: AWS · Google Cloud · Railway · Render · Docker

Learning objectives

• Package services for cloud deployment
• Configure environments and secrets safely
• Deploy at least one pipeline or API service

Sessions

1. S25

   Cloud options for data products

   Compare AWS, GCP, Railway and Render for cohort projects.

2. S26

   Deploying containerised services

   Ship a working service with environment configuration.

3. S27

   Observability & operational basics

   Logs, health checks and what to watch after deploy.

Practical exercises

• Container deploy to Railway or Render
• Map equivalent AWS/GCP building blocks
• Document runbooks for your deployed service

Project: Deployed service (pipeline worker, API, or both).

Week 10

Capstone Project & Demo

Integrate the stack and present a production-minded system.

Tools: Full stack

Learning objectives

• Assemble extraction → warehouse → transform → orchestration → API/deploy
• Document architecture decisions
• Demo and package work for portfolio use

Sessions

1. S28

   Capstone architecture clinic

   Pressure-test designs before the final build push.

2. S29

   Build & integration lab

   Connect remaining pieces and resolve production issues.

3. S30

   Demo day & portfolio packaging

   Present what you built and how the system fits together.

Practical exercises

• Capstone build sprint
• Architecture write-up
• Live demo to the cohort

Project: End-to-end Blockchain Data Engineering capstone.