Blockchain Data Engineering Programme Curriculum

Week 1

Foundations & Engineering Environment

Orientation, tooling, and reproducible local environments.

Tools: Python · Docker · SQL

Learning objectives

• Understand the Blockchain Data Engineering workflow
• Set up a reproducible development environment
• Write foundational Python and SQL for pipeline work

Sessions

1. S1

   Programme orientation & systems thinking · Python & SQL for data engineers

   Map the full path from chain data to product and set programme expectations. Practical language foundations used throughout the cohort.

2. S2

   Docker fundamentals for data work

   Containerise a simple service and understand why reproducibility matters.

Practical exercises

• Run a baseline Docker environment
• Query sample tabular data with SQL
• Structure a project repository for engineering work

Week 2

Blockchain Data Extraction

Pull real onchain data using Web3.py and structure extraction jobs.

Tools: Python · Web3.py · Docker

Learning objectives

• Connect to blockchain RPC endpoints
• Extract transactions, logs and contract events
• Persist raw extraction outputs reliably

Sessions

1. S3

   Web3.py & RPC access patterns · Transactions, receipts and event logs

   Connect, request and handle blockchain node responses. Understand the data shapes you will extract and store.

2. S4

   Building your first extraction job

   Ship a working job that pulls and stores raw onchain data.

Practical exercises

• Fetch blocks and transactions with Web3.py
• Decode a basic event log set
• Containerise an extraction job

Project: Raw extraction service for a chosen contract or token set.

Week 3

Data Pipelines

Turn extraction into a pipeline with validation and multi-service compose setups.

Tools: Python · Docker Compose · PostgreSQL

Learning objectives

• Design extract → validate → load steps
• Run multi-service environments with Docker Compose
• Handle failures and retries in batch jobs

Sessions

1. S5

   Pipeline architecture for onchain data · Docker Compose for data services

   Break work into dependable stages instead of one-off scripts. Coordinate application and database containers locally.

2. S6

   Validation, idempotency and load

   Make ingestion safer to re-run and easier to debug.

Practical exercises

• Compose app + database services
• Add validation before load
• Log pipeline run metadata

Week 4

Data Warehousing with PostgreSQL

Model and store structured blockchain data for analytical workloads.

Tools: PostgreSQL · SQL · Docker

Learning objectives

• Apply warehouse concepts to onchain entities
• Design schemas for transfers and events
• Use indexing to support analytical queries

Sessions

1. S7

   Warehouse thinking for blockchain data · Schema design for transfers & events

   Move from dumping JSON to designing durable tables. Normalize addresses, tokens and transfer facts.

2. S8

   Indexing and query performance basics

   Keep analytical queries usable as volume grows.

Practical exercises

• Model ERC-20 transfer tables
• Add indexes for common query patterns
• Inspect schemas with database tooling

Project: Structured warehouse schema for extracted blockchain activity.

Week 5

Analytics Engineering with dbt

Transform raw tables into cleaned, tested analytics-ready models.

Tools: dbt · PostgreSQL · SQL

Learning objectives

• Structure a dbt project for onchain datasets
• Build staging and mart layers
• Add tests to transformation models

Sessions

1. S9

   dbt project structure · Staging models for blockchain tables

   Sources, models and the path from raw to marts. Clean and standardize fields before business logic.

2. S10

   Testing and documenting models

   Make transformations trustworthy and shareable.

Practical exercises

• Create staging models from raw tables
• Build transfer marts
• Add uniqueness and not-null tests

Week 6

Normalization & Protocol Datasets

Turn messy chain records into protocol-ready analytical datasets.

Tools: dbt · SQL · Python

Learning objectives

• Normalize addresses and token metadata
• Model protocol-relevant entities
• Produce datasets ready for analytics and research

Sessions

1. S11

   Normalization patterns for onchain data · Protocol analytics datasets

   Reduce duplication and inconsistency across entities. Shape tables that analysts and apps can actually use.

2. S12

   Review: transformation quality

   Critique models, tests and documentation as a cohort.

Practical exercises

• Build address dimension logic
• Create protocol activity aggregates
• Review model lineage

Project: Analytics-ready protocol dataset from your warehouse.

Week 7

Orchestration with Prefect & Airflow

Schedule and coordinate multi-step pipelines reliably.

Tools: Prefect · Apache Airflow · Docker

Learning objectives

• Express pipelines as orchestrated workflows
• Understand Airflow DAGs and dependencies
• Handle retries, schedules and observability basics

Sessions

1. S13

   Why orchestration matters · Prefect for practical workflows

   From manual scripts to dependable scheduled systems. Compose flows that match your existing pipeline stages.

2. S14

   Apache Airflow DAGs

   Model dependencies and schedules with industry-standard tooling.

Practical exercises

• Build a Prefect flow for extract → load → transform
• Author a simple Airflow DAG
• Simulate and recover from a failed task

Week 8

Streaming Concepts & Data APIs

Introduce Kafka-style streaming ideas and expose data via FastAPI.

Tools: Apache Kafka · FastAPI · PostgreSQL

Learning objectives

• Understand streaming vs batch for blockchain data
• Work with Kafka concepts (topics, producers, consumers)
• Serve warehouse data through a FastAPI service

Sessions

1. S15

   Batch vs streaming for onchain systems · Apache Kafka fundamentals

   Choose the right pattern for latency and reliability. Topics, producers, consumers and where they fit your stack.

2. S16

   Building blockchain data APIs with FastAPI

   Turn infrastructure into a service applications can call.

Practical exercises

• Sketch a streaming architecture for event ingestion
• Produce/consume a sample topic locally
• Ship a FastAPI endpoint over warehouse tables

Project: Data API over your analytics-ready tables.

Week 9

Cloud & Deployment

Deploy practical projects beyond your laptop.

Tools: AWS · Google Cloud · Railway · Render · Docker

Learning objectives

• Package services for cloud deployment
• Configure environments and secrets safely
• Deploy at least one pipeline or API service

Sessions

1. S17

   Cloud options for data products · Deploying containerised services

   Compare AWS, GCP, Railway and Render for cohort projects. Ship a working service with environment configuration.

2. S18

   Observability & operational basics

   Logs, health checks and what to watch after deploy.

Practical exercises

• Container deploy to Railway or Render
• Map equivalent AWS/GCP building blocks
• Document runbooks for your deployed service

Project: Deployed service (pipeline worker, API, or both).

Week 10

Capstone Project & Demo

Integrate the stack and present a production-minded system.

Tools: Full stack

Learning objectives

• Assemble extraction → warehouse → transform → orchestration → API/deploy
• Document architecture decisions
• Demo and package work for portfolio use

Sessions

1. S19

   Capstone architecture clinic · Build & integration lab

   Pressure-test designs before the final build push. Connect remaining pieces and resolve production issues.

2. S20

   Demo day & portfolio packaging

   Present what you built and how the system fits together.

Practical exercises

• Capstone build sprint
• Architecture write-up
• Live demo to the cohort

Project: End-to-end Blockchain Data Engineering capstone.