AI On-Premises: A Look at OpenAI GPT-OSS-120B

OpenAI’s release of gpt-oss-120b marks a pivotal moment in the democratization of large language model deployment. As organizations increasingly seek to maintain data sovereignty, the availability of a 120-billion parameter model optimized for on-premises deployment addresses a critical gap in the enterprise AI landscape. This model represents not just a technical achievement, but a strategic shift toward enabling organizations to leverage powerful AI capabilities within their own infrastructure boundaries with affordable hardware solutions.

The importance of this model extends beyond mere availability. For industries handling sensitive data like healthcare, finance, or government, the ability to run sophisticated language models locally means maintaining complete control over proprietary information while still benefiting from state-of-the-art natural language processing capabilities. This model bridges the gap between the desire for advanced AI capabilities and the necessity of data governance and regulatory compliance.

Additionally, as a Mixture-of-Experts (MoE) model, it performs very well on single GPUs with 80GB or more memory, so it scales linearly as GPUs are added. This allows organizations to grow as they need, in affordable, incremental steps. The models offers reasoning with Chain-of-Thought (CoT) and reduces overthinking common in many other current open models.

Signal65 tested gpt-oss-120b on NVIDIA H200, AMD MI300X, and NVIDIA RTX Pro 6000 GPUs. The model runs well across all three accelerators, generating impressive token rates up to 64 or 128 (H200) simultaneous requests on a single GPU. Batch sizes were increased until per thread TPS dropped below 20.