← All lessons
Warehouses and pipelines · Lesson 1 of 9

Before you start

This course follows SQL: Querying data. If you haven't done it, this lesson is the short version, plus the everyday words data engineers use that the later lessons take for granted.

A database keeps data in tables of rows and columns. Each table here has an id column with a different number for every row, its primary key; another table points at a row by storing that number in a column like customer_id, a foreign key. A missing value is NULL. SQL asks questions: SELECT the columns FROM a table, WHERE rows match a condition. JOIN … ON combines two tables on their shared key, and GROUP BY with COUNT or SUM gives one row per group.

SELECT c.name, COUNT(o.id) AS orders
FROM customers c
JOIN orders o ON o.customer_id = c.id
GROUP BY c.name
ORDER BY orders DESC, c.name
LIMIT 3;
Returns
name | orders
-----+-------
Ana  | 3
Ben  | 2
Cleo | 2
(3 rows)

Other statements change data: INSERT INTO adds rows, UPDATE changes them and DELETE removes them. CREATE TABLE makes a table, and CREATE VIEW name AS SELECT … saves a query under a name so it can be read like a table. A window function (… OVER (PARTITION BY …)) works out a value across related rows without collapsing them.

Words you'll meet. A schema means two things: a named folder of tables inside a database (shop.marts), and the shape of some data (its columns and types). A transaction groups several changes so they all happen or none do. A partition is a slice of a big table, often one day's rows, stored so it can be read or replaced on its own. JSON is plain-text data with named fields, like {"type": "click", "device": {"os": "ios"}}. Object storage is a cloud service that keeps files in buckets (named containers). A dashboard is a page of charts that reads from the warehouse. Version control (Git, for example) records every change to code so you can review it and go back.

Practice

List every product's name and price, most expensive first.

The tables
  • customers (id, name, city, joined)
  • products (id, name, category, price)
  • orders (id, customer_id, ordered_on, status)
  • order_items (order_id, product_id, qty)
  • signups (id, email, name, signed_up_at)
  • payments (id, order_id, amount, currency, paid_at)
  • events (id, received_at, payload (JSON text))
More info

Quick reference

Primary key, foreign keyA row's own id; a column pointing at another table's id.
SELECT … FROM … WHEREWhich columns, which table, which rows.
JOIN … ON, GROUP BYCombine tables on a key; one row per group.
INSERT, UPDATE, DELETEAdd, change, remove rows.
CREATE VIEWA saved query you can read like a table.
SchemaA folder of tables; or the shape of data.
TransactionChanges that all happen or none do.
PartitionA slice of a table, often one day.
Object storage, bucketCloud file storage; a named container in it.

The other AI and data engineering courses