Eight.ly Agent model
Train for the job the product actually does.
A model that can discuss storage is not automatically a model that can operate a NAS. We built a training and evaluation pipeline around Eight.ly's real tools, then published task-specific fine-tunes. The work covers the data, training, conversion and product integration that turn open model weights into something useful.
View the published model work- 87.5%Historical accuracy
87.5% in the retired FunctionGemma router’s second training round; a historical evaluation, not a score for the current router.
- 4,684Training examples
May 4, 2026 dataset snapshot across 41 tools, before later no-tool augmentation.
- 200Held-out examples
Separate evaluation dataset with different seed and phrasing; not customer or live-use evidence.
- About $25Initial GPU rental
May 8, 2026 account of roughly three compute hours for the first three fine-tunes; excludes engineering time.
- Define the jobActual tool names, arguments and boundaries become the training contract.
- Generate and hold outVaried examples teach calls, restraint and recovery; evaluation data stays separate.
- Train and convertAdapters, token handling and quantization preserve the required output format.
- IntegrateThe product runtime routes requests and executes valid supported tools.
The problem
Our initial assistant used general models that had never seen Eight.ly's API. They invented tool names, omitted parameters and answered requests for action as conversation. We needed examples of the actual job, including cases where the right answer was to use no tool at all. Training only successful calls would teach the model to treat every request as an operation.
A finished training run was also only one stage. The exported model had to retain its tool-call format through adapter merging, tokenization, quantization and the runtime's chat template. A model could load successfully while losing the exact special tokens needed for the parser to recognize any call.
Who it’s for
We built this for a product that has to run on an owner's hardware and call a defined set of system operations. The same engineering problem faces teams adding AI to an existing product: the assistant needs to know the exact interface, recognize when a request is outside it, and return valid arguments rather than a convincing approximation.
What we built
We built a generator with randomized names, addresses, paths and values, so examples taught the shape of each call. The May dataset contained 4,684 examples across 41 tools, with single calls, chained workflows, hard negatives and error recovery. A separate 200-example evaluation set used different phrasing and a different seed.
We trained fine-tunes of existing open models, rather than claiming to have created their base weights. The first published set included models for different memory budgets and a small dedicated router. Later work separated routing from conversation more deliberately. The current product documentation identifies a 1.5-billion-parameter router in front of the installed talker.
How it works
The pipeline turns tool definitions into complete conversations: a request, a structured call, the result and a response grounded in that result. Training adapts the base weights, the adapter is merged, and the model is converted and quantized for local use. Product integration then checks that the runtime preserves native tool calling, rather than assuming a successful conversion is sufficient.
The recorded 87.5% accuracy belongs to a historical training round of the earlier FunctionGemma router. We keep that result with its scope: it is evidence from the model programme, not a fresh evaluation of today's router or a guarantee that 87.5% of production requests succeed. The current documentation explicitly marks that earlier router as retired.
What was hard
One training collator supervised only the final prose turn and masked the earlier tool-call tokens. Training could appear to work while teaching the wrong thing. We changed it to supervise every assistant turn. Another repair required resizing token embeddings before loading the adapter, then preserving the model's special tokens during conversion.
We also removed a training dependency after repeated compatibility failures and restored a stock runtime template when a custom one silently disabled tool calls. The initial three-model run used roughly $25 of rented GPU compute, but most of the engineering day went into those boundaries. Our contribution is the working path from a product's own interface to evaluated, deployable behavior.