Anyone who wants to run a pretrained AI model on a small embedded device usually has to adapt it laboriously by hand. According to a publication on arXiv, the research team presents AgentDP for this purpose. The system uses large language models (LLMs) as agents that take over the optimization and adaptation of the model largely on their own.

Why the adaptation is so laborious

Embedded devices differ greatly in hardware and software. A model that runs without problems on a server often doesn't fit the runtime environment of such a device directly. That is why several steps are usually necessary: shrinking the model, rebuilding individual parts, and finding suitable settings. This takes time and expertise. AgentDP combines the LLMs with device-specific domain knowledge so that the agents can steer these steps better.

Three technical building blocks

The authors name three core problems and one solution for each:

  • Clarifying requirements: A module for standardizing user requirements is meant to help users formulate their wishes clearly.
  • Matching the environment: An operator-aware model partitioning is meant to efficiently run different parts of the model on different inference engines. The inference engine is the software that actually computes a model on the device.
  • Saving search effort: A module for reusing configurations heuristically retrieves earlier hyperparameter settings. This way, new tasks don't have to start the search from scratch every time.

What the experiments show

According to the authors, AgentDP shortens the total adaptation time by up to 80% compared with manual work. At the same time, the system achieves on average more than 13% higher accuracy than the baseline methods. A user study is also meant to show that the approach can be adapted to different requirements for heterogeneous devices.

Assessment

These are the authors' claims in a preprint. Anyone planning their own embedded projects should check whether the test setups match their own hardware. The approach is of particular interest to teams that need to deploy a model on many different devices.