How to Make Your AI Model ONNX-Compatible for Microsoft Foundry Local

I recently presented on Foundry Local at the Azure User Group Sweden meetup, showing how to run AI models entirely on-device: no cloud, no API keys, no per-token costs. The demo landed well, but the best moment came during Q&A, when someone asked the question every developer eventually hits:

"This is great for the models you ship. But how do I make my model ONNX-compatible so I can run it with Foundry Local?"

Great question! This blog post is the detailed answer.

🎥 Watch the recording

TL;DR

If you only remember five things:

Foundry Local runs ONNX Runtime (ORT) under the hood, so anything ORT can load, Foundry Local can serve.
For most models, one command with the ORT GenAI Model Builder is all you need.
Reach for Olive when you need GPTQ/AWQ quantization, NPU targeting, or fine-tuning in the same pipeline.
Start at INT4 with block_size=32. It is the sweet spot for on-device.
Conversion is not the finish line. You still need an inference_model.json to register the model with Foundry Local.

Who this is for: developers who have a fine-tuned or custom model and want it running locally, and anyone evaluating whether their architecture is supported before investing a weekend in it.

Understanding the Ecosystem

Conversion goes badly when you do not know which component owns which problem. So before any commands, here is the map and the vocabulary.

🔤 Jargon Buster: Key Terms

Term	Definition
ONNX	Open Neural Network Exchange, an open format for representing machine learning models. Think of it as a universal translator between ML frameworks.
ONNX Runtime (ORT)	Microsoft's high-performance inference engine that runs ONNX models across different hardware (CPU, GPU, NPU).
ONNX Runtime GenAI	A layer on top of ORT that adds what language models need: KV caching, sampling, beam search, and a token-generation loop.
Foundry Local	Microsoft's end-to-end solution for shipping on-device AI in your applications, with a lightweight runtime and an OpenAI-compatible endpoint.
Quantization	Reducing model precision (for example, 32-bit to 4-bit) to shrink size and speed up inference, with minimal quality loss.
Execution Provider (EP)	The hardware backend ORT uses: CPU, CUDA (NVIDIA GPU), DirectML (Windows GPU), QNN/OpenVINO/VitisAI (NPU).
Inference	Running a trained model to get predictions, as opposed to training the model.

How Foundry Local Works Under the Hood

Key insight: Foundry Local uses ONNX Runtime under the hood. That single fact simplifies everything that follows. You are not really targeting Foundry Local, you are targeting ORT. If ORT can load your model, Foundry Local can serve it.

Two Paths to ONNX Conversion

With that established, the next decision is which tool does the converting. There are two, and the choice is mostly about how much control you need.

Tool	Best For	Complexity
ORT GenAI Model Builder	Quick conversion of supported architectures	⭐ Simple
Microsoft Olive	Advanced optimization, fine-tuning, custom pipelines	⭐⭐⭐ Advanced

My advice: always try the Model Builder first. It takes one command to find out whether your architecture is supported, and if it works you are done. Olive is the right answer for the cases where it does not.

We will cover both.

Prerequisites

System Requirements

Python 3.10+ (3.11 recommended)
16GB+ RAM, and comfortably more than the unquantized model size, because conversion loads full-precision weights before quantizing
GPU (optional, speeds up conversion)
50GB+ free disk space (you will hold the source weights and the output at the same time)

Install Required Packages

# Create and activate virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1

# Install core packages
pip install onnxruntime-genai
pip install transformers torch
pip install huggingface_hub

# For Olive (optional, for advanced optimization)
pip install olive-ai

🔤 Jargon Buster: Virtual Environment

A virtual environment is an isolated Python installation. It keeps your project's dependencies separate from other projects, preventing version conflicts. Always use one for ML projects.

Method 1: ONNX Runtime GenAI Model Builder

The Model Builder is Microsoft's official tool for converting models into the ONNX GenAI format. It is not a general-purpose converter, and that is exactly why it works so well: it knows the shape of each supported architecture and generates a graph that ORT GenAI is already tuned to run fast.

Supported Model Architectures

That design has one consequence worth checking before you spend an hour downloading weights. Your architecture must be on the list.

Architecture	Example Models	Status
Llama	Llama 3, Llama 3.1, CodeLlama	✅ Supported
Phi	Phi-3, Phi-3.5, Phi-4	✅ Supported
Qwen	Qwen2, Qwen2.5	✅ Supported
Mistral	Mistral 7B, Mixtral	✅ Supported
DeepSeek	DeepSeek-R1, DeepSeek distills	✅ Supported
Gemma	Gemma, Gemma 2	✅ Supported
Granite / OLMo / ChatGLM	Various	✅ Supported
Whisper	whisper-tiny to whisper-large	✅ Supported
Custom or research architectures	Your own model class	❌ Needs Olive or a hand-written exporter

💡 Tip: This list moves fast. Check the onnxruntime-genai repository for the current set before concluding your model is unsupported. What matters is the model_type field in your config.json, not the name on the Hugging Face card. A model fine-tuned from Llama is still a Llama.

Step 1: Identify Your Source Model

The Model Builder accepts inputs from several places:

Step 2: Basic Conversion from Hugging Face

Start with a known-good model so you can confirm your environment works before pointing the tool at your own weights. Phi-3-mini is small and fast to convert:

python -m onnxruntime_genai.models.builder `
    -m microsoft/Phi-3-mini-4k-instruct `
    -o ./phi3-onnx `
    -p int4 `
    -e cpu

Parameter breakdown:

Parameter	Value	Meaning
`-m`	`microsoft/Phi-3-mini-4k-instruct`	Hugging Face model ID
`-o`	`./phi3-onnx`	Output directory
`-p`	`int4`	Precision (quantization level)
`-e`	`cpu`	Target execution provider

🔤 Jargon Buster: Precision Levels

Precision	Bits per Weight	Model Size	Quality	Speed
`fp32`	32 bits	Largest	Best	Slowest
`fp16`	16 bits	Large	Excellent	Fast
`int8`	8 bits	Medium	Very Good	Faster
`int4`	4 bits	Smallest	Good	Fastest

Rule of thumb: start with int4 for on-device deployment. It gives the best size-to-quality ratio for most use cases, and if the output disappoints you can step up to int8 without changing anything else in your pipeline.

Step 3: Conversion with Advanced Options

The basic command is deliberately conservative. When you are ready to trade conversion time for output quality, or you are targeting specific hardware, the extra options are where the real tuning happens:

python -m onnxruntime_genai.models.builder `
    -m microsoft/Phi-3-mini-4k-instruct `
    -o ./phi3-onnx-optimized `
    -p int4 `
    -e cpu `
    -c ./cache `
    --extra_options `
        accuracy_level=4 `
        block_size=32 `
        algo_config=k_quant

Extra options explained:

Option	Values	Purpose
`accuracy_level`	1-4	Higher = better quality, slower conversion
`block_size`	32, 64, 128	Quantization granularity (smaller = better quality)
`algo_config`	`default`, `rtn`, `k_quant`	Quantization algorithm
`-c`	path	Cache directory for downloaded files

🔤 Jargon Buster: Quantization Algorithms

RTN (Round-To-Nearest): simple and fast. Rounds each weight to the nearest quantized value. Good default.
K-Quant: more sophisticated, borrowed from the llama.cpp ecosystem. Allocates precision based on how much each weight matters. Better quality, slower conversion.
GPTQ: calibration-based. Runs sample data through the model to decide how to quantize. Best quality, but you need a calibration dataset and Olive to run it.

Step 4: Converting Local or Fine-tuned Models

Once the sample conversion succeeds, point the same tool at your own model. Swap -m (a Hugging Face ID) for -i (a local directory):

python -m onnxruntime_genai.models.builder `
    -i ./my-finetuned-phi3 `
    -o ./finetuned-onnx `
    -p int4 `
    -e cpu `
    -c ./cache

The -i flag points to your local model directory, which must contain config.json and the weight files. Everything else stays the same.

Step 5: Converting Models with LoRA Adapters

If you fine-tuned with LoRA, you have two artifacts instead of one. The Model Builder can merge them during conversion:

🔤 Jargon Buster: LoRA

LoRA (Low-Rank Adaptation) is a technique for fine-tuning large models efficiently. Instead of updating all weights, LoRA trains small "adapter" matrices that modify the model's behavior. This makes fine-tuning faster and produces smaller files.

python -m onnxruntime_genai.models.builder `
    -i ./base-model `
    -o ./merged-onnx `
    -p fp16 `
    -e cuda `
    --extra_options adapter_path=./my-lora-adapter

Step 6: Converting GGUF Models

If your model already lives in the llama.cpp world, you can bring it across without going back to the original PyTorch checkpoint:

python -m onnxruntime_genai.models.builder `
    -m model_name `
    -i ./model-f16.gguf `
    -o ./gguf-to-onnx `
    -p int4 `
    -e cpu `
    -c ./cache

⚠️ Important: only fp16 and fp32 GGUF files are supported. Pre-quantized GGUF files such as Q4_K_M cannot be converted directly, because the quantization has already been baked in using a scheme ONNX does not share.

Understanding the Output

Whichever path you took, the output folder should look like this:

phi3-onnx/
├── model.onnx              # The converted model (main file)
├── model.onnx.data         # Model weights (if large)
├── genai_config.json       # Runtime configuration
├── tokenizer.json          # Tokenizer vocabulary
├── tokenizer_config.json   # Tokenizer settings
├── special_tokens_map.json # Special token definitions
└── added_tokens.json       # Additional tokens

The genai_config.json file is the one to understand. It tells ONNX Runtime GenAI how to drive your model, and it is plain JSON you can edit by hand when something needs adjusting:

{
  "model": {
    "bos_token_id": 1,
    "eos_token_id": 2,
    "pad_token_id": 0,
    "type": "phi3",
    "vocab_size": 32064,
    "context_length": 4096
  },
  "search": {
    "do_sample": false,
    "max_length": 4096,
    "min_length": 0,
    "num_beams": 1,
    "temperature": 1.0,
    "top_p": 1.0,
    "top_k": 50
  }
}

Method 2: Microsoft Olive (Advanced Optimization)

If the Model Builder covered your case, you can skip to testing. If it did not, this is where you go next.

Olive (ONNX LIVE) is Microsoft's model optimization toolkit. Where the Model Builder is one opinionated command, Olive is a pipeline you compose. Use it when you need:

Advanced quantization techniques (GPTQ, AWQ)
Fine-tuning and optimization in a single pipeline
NPU-specific optimization for Qualcomm, Intel, or AMD silicon
Custom optimization passes

Why Olive?

Learn more: Why Olive?

Step 1: Install Olive

pip install olive-ai
pip install transformers onnxruntime-genai

Step 2: Quick Optimization with the Olive CLI

Despite the flexibility, the common case is still a single command. auto-opt picks a sensible pipeline for your target hardware:

olive auto-opt `
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct `
    --output_path ./qwen-olive `
    --device cpu `
    --provider CPUExecutionProvider `
    --use_model_builder `
    --precision int4

What Olive does for you here:

Downloads the model from Hugging Face
Captures and converts the graph to ONNX
Applies transformer-specific graph optimizations
Quantizes to INT4
Emits a Foundry Local-compatible output folder

💡 Tip: swap --device npu --provider QNNExecutionProvider (Qualcomm), OpenVINOExecutionProvider (Intel), or --device gpu --provider DmlExecutionProvider (Windows GPU) to retarget the same command. This retargeting is the main reason to prefer Olive over the Model Builder for edge deployments.

Step 3: Advanced Olive Configuration

When the CLI defaults are not enough, describe the pipeline explicitly in a config file:

# olive-config.yaml
input_model:
  type: HfModel
  model_path: microsoft/Phi-3-mini-4k-instruct

systems:
  local_system:
    type: LocalSystem
    accelerators:
      - device: cpu
        execution_providers:
          - CPUExecutionProvider

passes:
  conversion:
    type: OnnxConversion
    target_opset: 17
  
  optimize:
    type: OrtTransformersOptimization
    model_type: phi3
    opt_level: 2
    
  quantize:
    type: OnnxMatMul4Quantizer
    block_size: 32
    is_symmetric: true

output_dir: ./phi3-olive-output

Run with:

olive run --config olive-config.yaml

When to Use Olive vs Model Builder

To summarise the choice:

Scenario	Recommended Tool
Quick conversion of a supported HF model	Model Builder
Simple INT4 quantization	Model Builder
GPTQ or AWQ quantization	Olive
Fine-tuning and conversion in one pipeline	Olive
NPU targeting (QNN, OpenVINO, VitisAI)	Olive
Custom optimization passes	Olive

Choosing the Right Quantization Strategy

Whichever tool you use, you still have to pick a precision. This is the decision that most affects how your model feels to users, so it is worth a minute of thought rather than copying whatever the last tutorial used.

Decision Flowchart

Quantization Comparison

Configuration	Model Size	Quality	Inference Speed	Best For
fp16	~7GB (7B model)	★★★★★	★★★☆☆	Quality-critical apps
int8	~3.5GB	★★★★☆	★★★★☆	Balanced
int4 (block=128)	~2GB	★★★☆☆	★★★★★	Edge devices
int4 (block=32)	~2.2GB	★★★★☆	★★★★☆	Best balance

Execution Provider Selection

# For NVIDIA GPU
-e cuda

# For Windows GPU (AMD, Intel, NVIDIA)
-e dml

# For CPU (universal)
-e cpu

# For web deployment
-e webgpu

Testing Your Converted Model

A conversion that completes without errors is not the same as a conversion that works. Test in two stages: first prove the ONNX model generates sensible text on its own, then wire it into Foundry Local. Doing it in that order tells you immediately whether a problem is in the model or in the plumbing.

Quick Test with Python

import onnxruntime_genai as og

# Load your converted model
model = og.Model('./phi3-onnx')
tokenizer = og.Tokenizer(model)

# Configure generation
params = og.GeneratorParams(model)
params.set_search_options(max_length=200)

# Create prompt
prompt = "<|user|>\nWhat is the capital of Sweden?<|end|>\n<|assistant|>\n"
input_tokens = tokenizer.encode(prompt)

# Generate response
generator = og.Generator(model, params)
generator.append_tokens(input_tokens)

print("Response: ", end="", flush=True)
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode([new_token]), end="", flush=True)

print()  # New line at end

If that prints a coherent answer, the conversion itself is sound. Now to get it into Foundry Local.

Register the Model with Foundry Local

This is the step most guides skip, and it is where people get stuck. Foundry Local will not pick up a bare ONNX folder. It needs a manifest that tells it the model's name and how to format prompts.

Create an inference_model.json file inside your model folder, alongside model.onnx:

{
  "Name": "my-phi3-custom",
  "PromptTemplate": {
    "assistant": "{Content}",
    "prompt": "<|user|>\n{Content}<|end|>\n<|assistant|>\n"
  }
}

The prompt template must match the chat format your model was trained on. Get this wrong and the model will still respond, just badly, which makes it a uniquely annoying bug to diagnose. Copy the format from the original model card's chat template.

Then point Foundry Local's cache at the parent folder and run it:

# Point the cache at the directory CONTAINING your model folder
foundry cache cd C:\models

# Confirm Foundry Local can see it
foundry cache ls

# Run it
foundry model run my-phi3-custom

Call It from Your Application

Foundry Local exposes an OpenAI-compatible endpoint, so any OpenAI client works. The SDK's job is simply to start the service and tell you which port it landed on:

import openai
from foundry_local import FoundryLocalManager

# Starts the Foundry Local service and loads your model
manager = FoundryLocalManager("my-phi3-custom")

client = openai.OpenAI(
    base_url=manager.endpoint,
    api_key=manager.api_key  # not used locally, but required by the client
)

response = client.chat.completions.create(
    model=manager.get_model_info("my-phi3-custom").id,
    messages=[{"role": "user", "content": "Explain quantum computing simply."}]
)

print(response.choices[0].message.content)

Because the endpoint is OpenAI-compatible, porting an existing cloud-backed app is usually a two-line change: swap the base URL and the model name. That is the whole pitch for Foundry Local.

Troubleshooting Common Issues

Most conversion problems fall into a handful of buckets, and nearly all of them announce themselves either at conversion time or at the very first token. Here is how to triage.

Error Decision Tree

Common Fixes

Problem	Cause	Solution
`KeyError: 'model_type'`	Unsupported architecture	Use Olive or wait for support
`CUDA out of memory`	Model too large for GPU	Use `-e cpu` or reduce layers
`Tokenizer not found`	Missing files	Re-run conversion, check output
`Slow first inference`	KV cache warming	Normal behavior, subsequent calls faster
`Poor output quality`	Over-quantization	Use `int8` or `fp16` instead
`Model not loading`	Corrupted conversion	Delete output, re-convert

Hugging Face Authentication

Some models require authentication:

# Login to Hugging Face
huggingface-cli login

# Or pass token directly
python -m onnxruntime_genai.models.builder `
    -m meta-llama/Llama-3-8B-Instruct `
    -o ./llama3-onnx `
    -p int4 `
    -e cpu `
    --extra_options hf_token=your_token_here

Complete Conversion Workflow

Putting it all together, here is what actually happens between your command and your first generated token:

Best Practices Checklist

Everything above, compressed into the list I actually run through before shipping a converted model:

[ ] Verify architecture support before downloading anything large
[ ] Convert a known-good model first to validate your environment
[ ] Start with INT4 for fast iteration, then step up only if quality demands it
[ ] Test output quality with prompts representative of your real workload
[ ] Compare against the original model to catch quantization regressions
[ ] Verify the prompt template in inference_model.json matches the model's chat format
[ ] Document your conversion parameters so the build is reproducible
[ ] Keep the original model for debugging and A/B comparison
[ ] Test on target hardware, since CPU results do not predict GPU or NPU results
[ ] Set an appropriate max_length to keep KV cache memory bounded

Resources

This Talk

Recording — Azure User Group Sweden

Official Documentation

Hands-on Learning

Foundry Local Lab - Step-by-step exercises
Olive Recipes - Example configurations

Community

Conclusion

When that question came up at the meetup, my honest reaction was that the answer sounds harder than it is. Converting a model for Foundry Local is not a research project. It is a short checklist:

Check architecture support. Most popular families (Llama, Phi, Qwen, Mistral, Gemma) already work.
Try the Model Builder first. One command usually produces a working model.
Escalate to Olive when you need to. Fine-tuning, GPTQ, and NPU targeting live there.
Start at INT4. Best size-to-quality ratio for on-device, and easy to walk back.
Do not forget inference_model.json. Conversion gets you a model; the manifest gets you a served model.

The payoff is worth the afternoon. Once converted, your model runs entirely on the user's machine: no per-token bill, no network round trip, and no data leaving the device. For a lot of use cases, that last point is not an optimization. It is the whole reason the project exists.

So pick a model, run the conversion, and see how far the first command gets you. It is usually further than you expect.

Questions, or hit an error this post did not cover? Drop a comment below, or find me at the next speaking event.

Appendix: Quick Reference Commands

Convert Hugging Face Model (Basic)

python -m onnxruntime_genai.models.builder -m MODEL_ID -o OUTPUT_DIR -p int4 -e cpu

Convert with Quality Optimization

python -m onnxruntime_genai.models.builder -m MODEL_ID -o OUTPUT_DIR -p int4 -e cpu --extra_options accuracy_level=4 block_size=32 algo_config=k_quant

Convert Local Model

python -m onnxruntime_genai.models.builder -i LOCAL_PATH -o OUTPUT_DIR -p int4 -e cpu

Convert with LoRA

python -m onnxruntime_genai.models.builder -i BASE_MODEL -o OUTPUT_DIR -p fp16 -e cuda --extra_options adapter_path=LORA_PATH

Olive Quick Optimize

olive auto-opt --model_name_or_path MODEL_ID --output_path OUTPUT_DIR --device cpu --provider CPUExecutionProvider --use_model_builder --precision int4

Test Model

import onnxruntime_genai as og
model = og.Model('./output')
tokenizer = og.Tokenizer(model)
# ... generate text

Register and Run in Foundry Local

foundry cache cd PARENT_FOLDER
foundry cache ls
foundry model run MODEL_NAME

About the Author

Siddhesh Prabhugaonkar is a Generative AI & Agentic AI Enablement and Adoption Specialist with two decades as an Architect, Consultant, and Trainer across IT, Cloud, and Generative AI. He is a Microsoft Certified Trainer, a Pluralsight Instructor, and helps enterprises move from GenAI curiosity to production adoption at scale.

His consulting and training practice spans GenAI, Azure, Microsoft Foundry, Foundry Local, ONNX Runtime, Anthropic Claude, GitHub Copilot, Amazon Q, Google Gemini, OpenAI Codex, Cursor, Windsurf, and modern full‑stack engineering (.NET, MEAN, MERN). Notable engagements include GenAI enablement for ADP, IoT platform consulting for IIT Bombay's E‑Yantra program, and early work on Microsoft's Repository platform (which later became Entity Framework).

Empowering organizations and individuals to adopt, build, and scale with Generative AI, Cloud, and Modern Software Engineering.

Connect & explore:

💼 LinkedIn — linkedin.com/in/siddheshprabhugaonkar
📝 Blog — azureauthority.in
📬 Newsletter — cloud-authority.com
🎥 YouTube — youtube.com/c/SiddheshPrabhugaonkar
🤝 Book a 1:1 on Topmate — topmate.io/siddheshp
🎓 Research Papers (Google Scholar) — scholar.google.com

Command Palette