Every inference runtime we have toured in this series — TensorFlow, ONNX Runtime, llama.cpp — assumes a fat host: gigabytes of RAM, an allocator that can hand out memory at will, maybe a GPU on the side. The repo under the microscope this time lives in a different world entirely. tflite-micro is the TensorFlow Lite for Microcontrollers (TFLM) runtime, and it is designed to run a neural network on a Cortex-M0 with 256 KB of RAM, a DSP with no operating system, or an ESP32 burning a few milliamps. There is no heap, no malloc, sometimes not even a filesystem — and yet the same .tflite flatbuffer that a desktop TensorFlow deployment would consume runs, byte for byte, on a microcontroller.
The trick is that TFLM does not load a model in any dynamic sense. It never copies tensor data, never parses the flatbuffer into heap objects, and never allocates anything after a one-time setup call. Instead, you hand it a single statically allocated byte array — the tensor arena — and it carves everything it needs out of that arena with pure pointer arithmetic. If you have read our NCNN tour, which does aggressive mobile inference but still runs on a real OS with a heap, the contrast is instructive: TFLM removes the allocator from the equation entirely, because on bare metal there is nobody to ask for more memory.
This post walks the actual source: how the flatbuffer is mapped zero-copy, how the MicroAllocator and its memory planners overlay activation buffers inside the arena, how MicroInterpreter drives a two-phase execution flow, and how the kernel layer splits into a portable reference and per-MCU optimized backends like CMSIS-NN. Two diagrams generated from the repository structure anchor the tour.
TFLite Micro at a glance: a zero-copy flatbuffer model, one static arena split by the MicroAllocator and a memory planner, and an operator layer resolved through a user-sized op resolver down to portable or CMSIS-NN kernels.
Reading the overview from left to right:
- tflite::GetModel, L25818 — casts your raw
.tflitebytes onto generated flatbuffer accessors. No copy, no parsing, no allocation; validation happens lazily as fields are read. - MicroInterpreter, L45 — the user-facing facade. Its constructor takes a
Model*, an op resolver, and your raw arena pointer; the primary overload, L55 is the whole dependency list of the runtime. - AllocateTensors, L189 — the one-time setup phase: builds subgraph allocations, runs every kernel’s
InitandPrepare, then commits the static memory plan. - MicroAllocator, L128 — the class that turns one
uint8_tbuffer into persistent structs, activation tensors, and scratch space, with the factory, L136 selecting the planner. - SingleArenaBufferAllocator, L32 — the bump allocator underneath, growing a persistent tail up and a temporary head down inside the same arena.
- GreedyMemoryPlanner, L47 — the default planner that overlays activation buffers whose lifetimes never intersect.
- MicroMutableOpResolver, L49 — a compile-time-sized array holding only the operators your model actually uses.
- InvokeSubgraph, L245 — the execution hot loop: for each operator, call its registration’s
invoke, then reset the temp region. - MicroInterpreterContext, L34 — the
MicroContextfacade through which kernels request tensors, scratch buffers, and profiling.
Loading a Model Without malloc
The foundation of everything is the flatbuffer format itself, shared with the parent TensorFlow Lite runtime. Flatbuffers serialize structures so that the serialized bytes are the in-memory representation: fields are accessed through vtable-relative offsets, so GetModel(buf) is literally a pointer cast plus a version check. There is a subtlety worth appreciating: the model is typically placed in flash (on embedded targets, often baked into the firmware image as a C array), and TFLM reads it directly from flash without ever touching RAM. The weights you trained on a workstation become const uint8_t bytes in the .text section of your firmware.
Everything mutable lives elsewhere. The TfLiteEvalTensor, L308 struct is the stripped-down, RAM-friendly cousin of the desktop TfLiteTensor: just data pointer, shape, and type, sized for the array-of-structs layout the allocator builds. Quantized models matter enormously here — an int8 tensor occupies a quarter of the flash and RAM that a float32 one does, and the hello world example, L139 demonstrates the dequantization dance ((int8_value - zero_point) * scale) at the output boundary.
The model-side metadata that kernels need — stride, dilation, activation function — is parsed out of the flatbuffer exactly once, during setup. The parser functions live in flatbuffer_conversions.cc; for instance ParseConv2D, L484 decodes Conv2DOptions into a Conv2DParams struct allocated from the arena’s persistent tail. This is one of the few places TFLM does transform flatbuffer data into native structs, and even that transformation allocates no heap memory — the BuiltinDataAllocator it receives hands out arena memory.
MicroInterpreter: AllocateTensors Then Invoke
The public API has exactly two phases, and the hello world test is a faithful minimal client. Its main(), L151 calls InitializeTarget, L24 for board bring-up, builds a resolver, and constructs an interpreter over a 3 KB stack array. Then AllocateTensors() and Invoke() — everything else is tensor I/O.
AllocateTensors is where the interesting orchestration happens. Reading its implementation, L189 top to bottom: StartModelAllocation walks the flatbuffer and allocates the persistent bookkeeping (node/registration arrays, eval tensor arrays), then PrepareNodeAndRegistrationDataFromFlatbuffer, L96 resolves each operator through the resolver and stores its TFLMRegistration*. Next the context switches into kInit and then kPrepare states, and every kernel gets its Init and Prepare callbacks — this is when a conv kernel picks tiling parameters, quantization multipliers, and requests scratch buffers. Only after all kernels have spoken does FinishModelAllocation, L195 compute the final activation layout, because only now are all buffer sizes and lifetimes known.
After allocation, Invoke, L271 is almost boring, and that is the point: it delegates to InvokeSubgraph, L245, which is a for-loop over operators calling registration->invoke(context_, node), L276. One line right after deserves attention: ResetTempAllocations, L285. Kernels allocate their TfLiteTensor views from the arena’s temporary head section; between ops that section is rewound to zero, so a graph of a thousand operators costs no more temp memory than a graph of one. Kernels that need to read tensors call back through the context — GetEvalTensor, L98 returns the zero-copy view by index.
The Tensor Arena and Its Planners
The tensor arena is the entire memory contract: a uint8_t array whose size you must guess up front. Internally it is managed as two regions growing toward each other — the persistent tail holds structs and op data that live as long as the interpreter, the head holds temporaries and the activation plan. The split is implemented by SingleArenaBufferAllocator, L32, which delegates to a PersistentArenaBufferAllocator, L31 for the tail and a NonPersistentArenaBufferAllocator, L30 for the head. If the two pointers cross, allocation fails with an error — there is no garbage collection and no second chance.
The activation buffers themselves are placed by a memory planner, an idea borrowed from compiler register allocation. The interface at MicroMemoryPlanner, L46 is tiny: add buffers with their usage intervals, get back offsets. The default GreedyMemoryPlanner, L47 sorts buffers by size and, via AddBuffer, L109, drops each one into the smallest gap where it does not overlap in time with already-placed buffers — classic interval scheduling, and remarkably effective since CNN activations tend to die young. The lifetime data comes from AllocationInfoBuilder::CreateAllocationInfo, L104, which marks each tensor’s first and last use by scanning the subgraph’s operator order. The final layout is committed in CommitStaticMemoryPlan, L1116, which writes the chosen offsets into every eval tensor.
A LinearMemoryPlanner, L27 exists as the conservative alternative: buffers are laid out strictly in execution order, which wastes more memory but behaves predictably and supports the preserve_all_tensors debugging feature. Because the whole plan is computed offline relative to a fixed arena, the planner runs inside the arena too — the greedy planner even takes its own scratch buffer as a constructor argument, practicing what it preaches.
Op Resolution and the Kernel Contract
The kernel contract is a single C struct. TFLMRegistration, L325 carries four function pointers (init, prepare, invoke, free) plus a builtin code or custom name, and the entire kernel ecosystem plugs into the interpreter through it. Where the desktop TensorFlow runtime registers hundreds of ops at startup, TFLM inverts the model: you tell the resolver what to include. MicroMutableOpResolver, L49 is templated on a compile-time count tOpCount, and each Add* call — AddConv2D, L207, AddFullyConnected, custom ops through AddCustom, L93 — copies a registration plus its flatbuffer parser into that array. Resolution during prepare is a linear scan in FindOp, L55; for five to ten ops that is faster than any hash table.
The kernels themselves come in two flavors sharing one contract. The portable reference implementations are plain readable C++ — ConvEval, L30 type-dispatches down to the reference math in Conv, L28, which is the textbook loop nest. For real deployments on Arm cores, the build-time OPTIMIZED_KERNEL_DIR=cmsis_nn flag swaps the innards for CMSIS-NN: the registration in cmsis_nn/conv.cc, L487 exposes the same Register_CONV_2D, but EvalInt8, L346 calls Arm’s DSP-optimized kernels underneath. This same directory pattern covers Xtensa (HiFi cores), Ethos-U NPU dispatch, and others — and it is the microcontroller sibling of what XNNPACK does for server and mobile CPU inference (our XNNPACK tour covers the same dispatch idea at larger scale). A decent part of the kernel directory exists purely to be swapped: Register_DEPTHWISE_CONV_2D, L184, Register_FULLY_CONNECTED, L359, and Register_SOFTMAX, L86 all follow the identical prepare-eval-register shape.
The framework comparison is worth spelling out. Compared to graph-compiler stacks like TVM, which generate fused C code per model, TFLM ships a pre-compiled interpreter and asks the model to conform; compared to end-to-end device runtimes like ExecuTorch, it is radically smaller and dumber — deliberately. The .tflite format itself is the same one ONNX users reach for via converters in spirit, though the details differ; the common thread across all of these is that the runtime cost scales with ops you select, not ops that exist.
From Install to Your First Model
There is no pip wheel for your dev board; TFLM is a source-first project meant to be vendored or compiled into your firmware. Clone the repo and you can build the canonical hello world example — a two-layer fully connected network approximating a sine wave — directly on your host machine first. The README-verified Make flow builds every dependency and compiles the test with the host toolchain:
make -f tensorflow/lite/micro/tools/make/Makefile test_hello_world_test
The equivalent under Bazel (also from the checked-out docs) is:
bazel run tensorflow/lite/micro/examples/hello_world:hello_world_test
Passing prints ~~~ALL TESTS PASSED~~~. The test source doubles as the shortest possible client. Stripped to its skeleton, the exact pattern verified against the checked-out tree is:
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/micro/examples/hello_world/models/hello_world_float_model_data.h"
namespace {
using OpResolver = tflite::MicroMutableOpResolver<1>;
TfLiteStatus RegisterOps(OpResolver& resolver) {
return resolver.AddFullyConnected(); // the only op this model uses
}
} // namespace
TfLiteStatus RunInference() {
const tflite::Model* model =
::tflite::GetModel(g_hello_world_float_model_data);
OpResolver op_resolver;
TF_LITE_ENSURE_STATUS(RegisterOps(op_resolver));
constexpr int kTensorArenaSize = 3000;
alignas(16) uint8_t tensor_arena[kTensorArenaSize];
tflite::MicroInterpreter interpreter(model, op_resolver, tensor_arena,
kTensorArenaSize);
TF_LITE_ENSURE_STATUS(interpreter.AllocateTensors());
interpreter.input(0)->data.f[0] = 1.f;
TF_LITE_ENSURE_STATUS(interpreter.Invoke());
float y = interpreter.output(0)->data.f[0];
return kTfLiteOk;
}
Three practical notes before you port this to a board. First, arena sizing: the example’s 3000 bytes is a guess, and the right tool for a real guess is the recording variant — run under RecordingMicroInterpreter, L37 and its allocator prints per-section usage, including arena_used_bytes() after AllocateTensors. Second, run it under MicroProfiler, L29 to get cycle counts per operator as CSV before you optimize anything. Third, the same Makefile cross-compiles: pass TARGET=cortex_m_generic OPTIMIZED_KERNEL_DIR=cmsis_nn and the tree produces binaries for Arm boards — the toolchain layer is exactly where per-MCU quirks surface.
Honest Limits
The static-memory design is the whole point, and it imposes real constraints. The arena is sized for the worst case at setup time and never renegotiated; if your arena is too small, AllocateTensors fails outright, and if it is too large you have permanently wasted RAM. There is no dynamic allocation at run time at all — models with data-dependent shapes, dynamic control flow beyond the explicit IF/WHILE operator support, or anything requiring new tensors mid-inference does not fit the framework. Even MicroMutableOpResolver capacity is a template constant: exceed your declared tOpCount and registration fails at build time (by design).
Op coverage is a second honest gap. TFLM supports a curated subset of the TensorFlow Lite op set, and the supported subset is int8-quantization-first; float16, complex control flow, and exotic layers frequently require either model surgery, custom op registration, or simply do not work. Kernel performance is uneven across targets: the portable reference is fine for correctness work but can be an order of magnitude off optimized paths, and the optimized backends have per-target toolchain demands — CMSIS-NN needs Arm GCC/Clang with specific flags, Xtensa needs the vendor’s toolchain and NNLib, Ethos-U needs the driver stack. The Windows and x86 paths work for CI but nobody ships a product on them.
Finally, debugging is spartan. Error strings are stripped in production builds ( Enjoyed this post? Never miss out on future posts by following us TF_LITE_STRIP_ERROR_STRINGS) to save flash, so failures often surface as a bare status code; the layer-by-layer debugging tools and the recording allocator help, but expect to reason about arena layout by hand. None of this is a flaw so much as the honest price of running inference where every byte has a name and no allocator exists to ask for more. 
