OpenAI has detailed how it moved its online storage platform Habitat from Python to Rust, a shift that the company says delivered a sixfold improvement in CPU efficiency and a 15-fold improvement in memory efficiency. The platform now processes more than 70 million requests per second and serves over a billion users weekly, according to a post from OpenAI cited by IThome.
Habitat began in 2024 as a Python client library that interacted with ChatGPT's main servers, with the underlying data stored in Azure Cosmos DB. Its purpose was to let product engineers store and retrieve data without directly managing databases, handling tasks such as data-type identification, routing, authorization, encryption, serialization, and connection-pool management.
As OpenAI's product lineup and data needs grew, the client architecture began to show limits by mid-2025. Every modification to the shared library required coordinating deployments across dozens of services, increasing release complexity and the risk of failures. That prompted OpenAI to rework Habitat into a standalone service with centralized deployment, monitoring, and platform functions.
Why Python hit a wall
OpenAI initially chose Python for the high-throughput service to prioritize platform stability and development speed. But under high concurrency, the Python implementation faced challenges with CPU overhead, memory overhead, and network latency, particularly in controlling request tail latency.
The team tried various optimizations, including limiting concurrent requests, increasing the number of Python worker processes, and improving asynchronous task scheduling. They also adjusted how feature-flag configurations were refreshed to reduce the impact of large-scale JSON parsing on request handling.
In the connection pool, OpenAI found that Python's aiohttp library used a LIFO (last-in, first-out) connection reuse strategy by default, which could cause traffic to concentrate on slower service processes. Switching to FIFO (first-in, first-out) connection reuse reduced load imbalance, according to the company.
Habitat uses a restricted NoSQL API to prevent clients from executing uncontrolled SQL queries, complex joins, or large-scale data scans. For scenarios requiring complex queries, OpenAI synchronizes data in real time or near-real time to Rockset via change data capture (CDC), enabling offline analytics and search.
Rust rewrite and results
In the second quarter of 2026, OpenAI assigned two engineers, along with its Codex and GPT-5.5 models, to rewrite the Habitat service entirely in Rust. The new Rust service now handles 95% of production requests, with average latency and tail latency also reduced, the company said.
OpenAI plans to fully retire the Python implementation of Habitat in the coming weeks. The company also said future posts will cover how Habitat scales the Azure Cosmos DB storage layer and optimizes multi-tenant reliability and read performance at the scale of over 500PB of data and 70 million requests per second.
Based on reporting by IT之家. Edited and published in English by geisou.
