DreamMachine — industrial-grade recommendation engine supporting 10 billion items and 10 billion users with ≤ 100 ms end-to-end latency.
Not a wrapper. A full research-grade production system built from scratch.
K = 3 independent hash tables eliminate ID explosion. Supports 10¹⁰-item catalogues with collision ε < 1% at B = 100 M buckets.
ε_eff = (1 − e^{−N/2B})^KIndependent item and user towers with L = 2 Transformer layers, H = 8 attention heads, d = 128. L2-normalised output for cosine similarity.
s(u,i) = u·v / τ (τ = 0.07)Multi-channel retrieval combining FAISS HNSW/IVF-PQ vector search, collaborative filtering, popularity signals, and business rules.
< 50 ms P99 end-to-end9 operating points from 10² to 10¹⁰ items with closed-form proofs. Predict the right config before writing a single training line.
9 breakpoints · 10² → 10¹⁰Listwise re-ranking with simultaneous CTR, CVR, and dwell-time prediction. Uncertainty-weighted InfoNCE multi-task loss.
Learnable σ_k² per taskFastAPI + ONNX + INT8 quantisation + A/B testing framework. Runs identically from RTX 3060 laptop to 250 × A100 cluster.
≤ 100 ms · ONNX · INT8Evaluated on 1,000 simulated e-commerce items (JD / Taobao schema). Numbers reflect the real checkpoint hf_model/model.safetensors.
End-to-end pipeline from raw JSON input to ranked recommendations.
Input JSON
│
┌─────────▼──────────┐
│ FeatureExtractor │
│ (JD / Taobao fmt) │
└──┬──────────────┬──┘
│ │
┌────────────▼────┐ ┌──────▼──────────────┐
│ Item Tower │ │ User Tower │
│ │ │ │
│ hash_ids × 3 │ │ hash_ids × 3 │
│ category_l1–l3 │ │ demographics │
│ brand_hash_ids │ │ city / region hashes │
│ numeric [29] │ │ numeric [57] │
│ │ │ │
│ L=2 Transformer │ │ L=2 Transformer │
│ H=8 d=128 │ │ H=8 d=128 │
│ L2-normalised │ │ L2-normalised │
└────────┬────────┘ └────────┬─────────────┘
│ v ∈ S^{d−1} │ u ∈ S^{d−1}
└──────────┬─────────┘
│
s(u,i) = u·v / τ (τ = 0.07)
│
┌──────────▼──────────┐
│ FAISS HNSW Index │
│ top-K ANN recall │
└──────────▼──────────┘
│
┌──────────▼──────────┐
│ Cross-Attention │
│ Listwise Re-ranker │
└──────────▼──────────┘
Top-N results
Seven original contributions spanning the full stack:
| # | Contribution | Key Mechanism |
|---|---|---|
| C1 | Multi-Hash Transformer Dual-Tower | ε_eff = (1 − e^{−N/2B})^K |
| C2 | Hierarchical Feature Architecture | Typed token: ID → sparse → dense |
| C3 | RAG-Enhanced Multi-Channel Recall | FAISS IVF-PQ + CF + popularity |
| C4 | Cross-Attention Listwise Re-ranker | Simultaneous CTR / CVR / dwell-time |
| C5 | Uncertainty-Weighted InfoNCE | Learnable σ_k² per task |
| C6 | Closed-Form Scaling Laws | 9 breakpoints 10² → 10¹⁰ items |
| C7 | Production Serving Stack | FastAPI + ONNX + INT8, ≤ 100 ms |
From zero to running recommendations in four steps.
# Install all dependencies
pip install torch safetensors transformers \
faiss-cpu huggingface_hub datasets \
uvicorn fastapi
from huggingface_hub import snapshot_download # Download the full model repository model_path = snapshot_download( repo_id="dream-machine-ai/Dream-Machine-08-09", repo_type="model", ) print("Model at:", model_path)
from transformers import AutoConfig, AutoModel config = AutoConfig.from_pretrained( "dream-machine-ai/Dream-Machine-08-09", trust_remote_code=True, ) model = AutoModel.from_pretrained( "dream-machine-ai/Dream-Machine-08-09", trust_remote_code=True, ).eval() # Forward pass — cosine similarity score # s(u, i) = u · v / τ, τ = 0.07
# Clone and launch FastAPI inference server git lfs install git clone https://huggingface.co/dream-machine-ai/Dream-Machine-08-09 cd Dream-Machine-08-09 python serve_hf.py --model_dir ./hf_model --port 8000 # Query the recommendation API curl -X POST http://localhost:8000/recommend \ -H "Content-Type: application/json" \ -d '{"user_id":"U001","top_k":10,"scene":"homepage"}'
Fully reproducible synthetic data in JD / Taobao schema — dream-machine-ai/Dream-Machine-08-09-Dataset
from datasets import load_dataset REPO = "dream-machine-ai/Dream-Machine-08-09-Dataset" products = load_dataset(REPO, data_dir="data/products", split="train") users = load_dataset(REPO, data_dir="data/users", split="train") behaviors = load_dataset(REPO, data_dir="data/behaviors", split="train") print(f"Products : {len(products)} records") print(f"Users : {len(users)} records") print(f"Behaviors: {len(behaviors)} records")
from huggingface_hub import snapshot_download dataset_path = snapshot_download( repo_id="dream-machine-ai/Dream-Machine-08-09-Dataset", repo_type="dataset", ) print("Dataset at:", dataset_path)
Full mathematical derivation, algorithm pseudocode, scaling proofs, and ablation studies.
Wen, Fangjun · DreamMachine Research Team · August 2026
Covers: Multi-Hash collision bounds · Closed-form scaling law proofs (Theorems 3.1–3.3) · Algorithm pseudocode for all 7 contributions (Alg. 1–9) · Ablation studies · Scaling charts
📄 Read on Zenodo