Hy3

Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters, developed by the Tencent Hy Team. Following the Hy3 Preview launch in late April, we gathered feedback from 50+ products and scaled up post-training with higher quality data. Today, we introduce Hy3, which outperforms similar-size models and rivals flagship open-source models with 2-5x parameters. It also shows significant gains in utility across various products and productivity tasks. What I read online says its supposed to offer nearly GLM5.2 performance at half the size.

I'm really impressed with HY3. If you haven't heard of this model, it's a new 295B-A21B MoE release from Tencent that competes directly on the frontier of open weights models, at a significantly smaller size (https://the-decoder.com/tencent-releases-hy3-open-source-model-that-allegedly-matches-models-up-to-five-times-its-active-size/), comparable to DeepSeek v4 Flash but with better benchmarks. I was intrigued by this article, and I'd just recently finished updating my Macbook M5 Max 128GB setup from antirez's DeepSeek V4 Flash quant running on dwarfstar to Unsloth's IQ3_XXS on mainline llama.cpp. I figured I had a good baseline for comparisons, if I could get it running, so I set about researching, and this is what I found. First off, I had to pick a quant. There are a few on HF, and after some comparison shopping I settled on this UD128 "unsloth dynamic"-style 107GB quant (https://huggingface.co/YanissAmz/Hy3-295B-A21B-GGUF). It was the only one that had published perplexity numbers at the time I searched, and while that's not KLD, it shows the creator was at least thinking about measuring quality degradation. PPL didn't see horrible for a dynamic 3-bit quant, and it felt like a similar checkpoint to the UD DS4 quant I was using. Next, I had to get llama.cpp working. As the quant's readme helpfully notes, there's PR #25395 (https://github.com/ggml-org/llama.cpp/pull/25395) which implements support for this model and its built-in speculative decoding module all at once! A quick build got this up and running: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp git fetch origin pull/25395/head:hy3 && git checkout hy3 cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON cmake --build build --config Release -j Don't forget to raise your Mac's GPU memory ceiling from the default 96GB! I put mine at 122GB to ensure 24k context would fit safely for testing (it resets on reboot): sudo sysctl iogpu.wired_limit_mb=124928 However, when I tried to run the model, it wasn't recognized by llama.cpp, so the server errored out. A quick review of the log and the quant's readme explained why: "these files carry general.architecture = hy-v3 (this port's original naming). PR #25395 registers the arch as hy_v3 (underscore)". I could have avoided this hiccup if I'd paid a bit more attention, but I put together a quick script to swap those characters (21 instances, all in the first of the 3 GGUF shards). I can share if anyone wants it (EDIT: see here (https://www.reddit.com/r/LocalLLaMA/comments/1usy9ie/comment/owywe5t/)), but honestly, just ask your agent to do the job -- it takes 2 minutes to fix the GGUF. Running my launch script again, after a nail-biting ~30 second load time to read 107GB from the SSD, the WebUI popped open and we were up and running! Benchmarks (M5 Max, llama-bench, Metal, q8_0 KV cache, MTP off) test tokens/sec prefill pp512 @ empty ctx 528 decode tg128 @ empty ctx 32.4 prefill pp512 @ 16K ctx 124 decode tg128 @ 16K ctx 16.3 In practical use, token generation speed is *DOUBLE* what I was getting from DeepSeek (edit: using mainline llama.cpp, not dwarfstar, for 1:1 comparison), with the same or better quality outputs! And I haven't even trialed MTP yet to see if there's any improvement... Here's my llama.cpp launch command: ~/llama.cpp-hy3/build/bin/llama-server \ -m ~/AI/models/Tencent-HY3-295B-A21B-YanissAmz/Hy3-UD128-00001-of-00003.gguf \ -a "Hy3 295B-A21B (UD128)" \ -ngl 99 \ -c 24576 \ -ctk q8_0 -ctv q8_0 \ -fa on \ -np 1 \ --jinja \ --temp 0.9 --top-p 1.0 \ --host 127.0.0.1 --port 8080 I've only played around with the model for a couple hours, but I am really impressed with performance on normal prompts and basic tool use. I've been using it to do ML research using the HF MCP and CLI tools, and it's definitely better than my DS4 setup so far -- on vibes and basic tool calls, at least. I haven't set it loose on longer-horizon agentic tasks or challenging coding yet, so I apologize for not having more in-depth reporting to offer at this time. Here's my launch command: I made this post because I'm really hyped to have a new large MoE to play with at this size checkpoint, and a very promising one at that. I hope it inspires some other folks to give it a try, and report back on their experience or compare other quants/MLX performance. EDIT: Got MTP working, n=2 is the best-performing variant on my hardware. Now peaking at 38 tok/sec, so +19% speedup! Similar speeds as my Qwen3.6-27B-MTP q8_0 daily driver now. Add this to your launch command for MTP: --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75 The p-min gate is crucial for acceptance rates to be high enough for MTP to help, not hurt. Be forewarned that MTP gains still turn negative after ~16k ctx, but honestly at that point, the GQA architecture starts slowing down significantly anyways. This is the biggest catch about this model vs. DeepSeek v4 Flash, and it's substantial.