Initial ET backend

[marty1885]

Overview

This PR is developed by AINekko and by members of AIFoundry (AINekko's OSS community) and adds the ET backend that supports the ET-SOC-1 processor. ET-SOC-1 was originally created by Esperanto Technologies which AINekko later open sourced under Apache 2.0 and would like to upstream the llama.cpp backend we developed for it as a way to integrate open source hardware into the open source inference ecosystem.

The the ET processor core documentation and RTL can be found at the following links

• CORE-ET RTL (compute, unfortunately NoC/memory/PCIe are not open)
• et-platform SDK and driver

As ET-SOC-1 is an older low power processor, the absolute performance is not impressive compared to even CPUs. But it still provides better performance per watt then my ARM R7 7700 development machine can do. Please refer to the following table for concrete performance number

The backend supports most smaller language models, passes llama.cpp unit testing (with minimal modification to loosen bounds a bit). And we consider Uberkernel (the in-kernel version of device side dispatch) experimental and good-to-have if the model supports it.

If CI is needed for the backend being accepted, please let me know and I can setup CI rigs or provide card access. AINekko is planning on making the ET-SOC-1 publicly available in card form. But manufacturing needs significant de-risking and we feel it's right to upstream as in principle it is viable to construct an FPGA or even tape out using the open source RTL.

Also if desirable, I am willing to be the maintainer of the ET backend and find others so bus factor > 1. Please let me know what is required and I will handle it from my side.

(As a side node. People might recall me working on llama.cpp integration for Tenstorrent, I no longer work there but the Tenstorrent community and me are still working on that integration gradually)

(Also side note, the PR was supposed to go through AIFoundry's repo, but due to timing and other reasons, the PR is created via my personal github repo).

Additional information

Some existing OSS discussions and material on ET-SOC-1

Posts:

• Which RISC-V instructions does the ET-SoC-1 give us?
• ET's Minions
• Investigating the ET-SoC-1 NoC (by me)
• Programming the ET-SoC-1 (by me)

FOSDEM:

• Zero to matmul with the ET-SoC-1
• The ET Minion RISC-V ISA
• A practical introduction to the ET platform.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Some kernel optimization strategies are explored by AI. Potentially also usage by community members. But I have personally ensure we are capable of hand writing and reason through. As well as some implementation tests.

[ggml-gh-bot]

Hi @marty1885, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[taronaeo]

Can you refactor this PR such that it only focuses on the ET backend? Leave things like tests/test-backend-ops.cpp to another PR so that its easier to review.

[marty1885]

@taronaeo Yes and done. Changes in test-backend-ops were needed for stability and finishing perf tests in a reasonable amount of time. And change in ggml-backend.cpp was needed to get the llama-tts working on ET backend. I will separate them into their own PR.

[taronaeo]

Haven't read through everything. In general, please clang-format your codebase so that the indents are consistent. Follow the contributing guidelines too.

[ubergarm]

@marty1885 great seeing you pull all the hard work by ainekko/aifoundry crew together here!

Since I still have some access to a rig with multiple ET-SoC-1 PCIe cards I just gave it a try. Got it running with the latest google/gemma-4-E2B-it-qat-q4_0-gguf and also tried older Qwen3-0.6B with and without UBERKERNEL successfully!

👈 Details

# build riscv-gnu-toolchain and et-platform using ET.md docs
# https://github.com/marty1885/llama.cpp/blob/et-backend-n-way-merge/docs/backend/ET.md#build
# https://github.com/aifoundry-org/et-platform#how-to-build-et-platform

# get PR code
$ git fetch marty1885
$ git checkout et-backend-n-way-merge
$ git rev-parse --short HEAD
35e190acd

# build for ET backend
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_ET=ON -DGGML_ET_SYSEMU=OFF
$ cmake --build build --config Release -j $(nproc)

# run the latest official Gemma-4 QAT Q4_0 on single ET-SoC-1 device
model=/models/google/gemma-4-E2B-it-qat-q4_0-gguf/gemma-4-E2B_q4_0-it.gguf
GGML_ET_UBERKERNEL=0 \
./build/bin/llama-cli \
    --model "$model" \
    --device ET0 \
    -fa on \
    --parallel 1 \
    --ctx-size 8192 \
    -ngl 999 \
    -t 1 \
    --single-turn \
    -p "Count from 1 to 3 in French."

...

[ Prompt: 32.0 t/s | Generation: 5.1 t/s ]

# Now run ggml-org/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q8_0.gguf
# with and without GGML_ET_UBERKERNEL

# UK=0
[ Prompt: 79.7 t/s | Generation: 10.0 t/s ]

# UK=1
[ Prompt: 86.6 t/s | Generation: 19.1 t/s ]

[giladgd]

I think it'll be great if you can have this script be in cpp so we won't introduce python to the build process, like we have in the Vulkan backend.
Avoiding python means the build process has fewer dependencies and users will have fewer issues with it

[marty1885]

I ported the code into CMake to simplify the build graph since the only build system now. Hope that's fine, I can make it C if desired.

[giladgd]

Having this script in CMake is even better, let's keep that

[taronaeo]

1. Same comments as all *.in files
2. Can the scripts sit in its own scripts directory?

[taronaeo]

Rebase with upstream/master and regenerate docs/ops.md. I think we should be good to go after that.

[taronaeo]

Suggested change

	#ifndef GGML_TENSOR_H
	#define GGML_TENSOR_H
	#ifndef GGML_ET_TENSOR_H
	#define GGML_ET_TENSOR_H

[marty1885]

Sorry for the lack of progress - my computer(s) broken due to an accident and wasn't able to produce patches. It is working again. Will supply update soon

[marty1885]

@taronaeo please sorry for the delay. Please review when you are avaliable

[taronaeo]

Sorry for the delay. I'll review this by the end of this week.

[taronaeo]

Please add yourself to CODEOWNERS

[marty1885]

Oh crap... Sorry for the trigger. I was merging to resolve the conflict in docs/ops.md and did not realize how GGML's notification system works. Rebasing was possible but a long fight that I wish to avoid.

Will push a new one soon. Sorry!

[taronaeo]

2 minor changes and we're good to go :)

[taronaeo]

@ggml-org/maintainers Another review please.

[pwilkin]

No way :P

[marty1885]

Praying this works :)

[pwilkin]