1. The Memory Problem with Standard `json.load()`

When standard `json.load(file)` parses a JSON object, Python creates internal PyObject instances for every key, string, integer, array, and dict node. A 500 MB raw JSON file on disk can consume over 3 GB of active RAM when converted into a native Python dictionary hierarchy due to PyObject pointer overhead.

Memory Crash vs. Streaming Solution 
# ❌ NAIVE APPROACH (Fails on 2 GB File - MemoryError!)
import json

with open("massive_database_dump.json", "r") as f:
    data = json.load(f) # Tries to allocate gigabytes of RAM at once

# ✅ STREAMING APPROACH (Uses < 10 MB RAM!)
import ijson

with open("massive_database_dump.json", "rb") as f:
    # Iterates over objects in the 'records' array one item at a time
    records = ijson.items(f, "records.item")
    for record in records:
        process_record(record) # Constant low memory consumption
Explanation: Streaming with `ijson` reads file buffers incrementally without loading the entire JSON graph into memory.

2. Streaming JSON Arrays with `ijson`

`ijson` is an iterative JSON parser for Python built on top of YAJL (Yet Another JSON Library). It generates Python generator iterators that yield JSON items on-the-fly as they are read from disk.

Filtering Giant Log Arrays Line by Line 
import ijson

def extract_errors(file_path):
    with open(file_path, "rb") as f:
        # Parse objects inside root array
        for item in ijson.items(f, "item"):
            if item.get("status") == "error":
                yield item["message"]

# Efficiently print error logs from a 5 GB file
for error_msg in extract_errors("server_logs.json"):
    print(f"CRITICAL ERROR: {error_msg}")
Explanation: The generator function yields matching error messages sequentially without filling memory.

Pro Tips

  • Install ijson via `pip install ijson`.
  • Always open the file in binary read mode (`"rb"`) for optimal ijson performance.

3. Line-Delimited JSON (JSONL) Streaming

For ultra-large datasets (10 GB+), data engineering teams often structure data as JSON Lines (`.jsonl` or `.ndjson`). Each line contains an independent, valid single-line JSON string. Parsing JSONL files requires zero third-party libraries:

Native Line-by-Line JSONL Parsing 
import json

# Read 20 GB JSONL file line-by-line using Python standard library
with open("analytics_events.jsonl", "r", encoding="utf-8") as f:
    for line_number, line in enumerate(f, 1):
        if not line.strip():
            continue
        event = json.loads(line) # Parse single line
        if event.get("event_type") == "purchase":
            track_purchase(event)
Explanation: Line-by-line iteration consumes constant RAM equal to the size of a single JSON line.