MASON

Supervised by Prof. Kayvan Karim (Heriot-Watt University Dubai).

MASON stands for Multi-Agent System for Open No-code development. I built it to test one question: if you split a coding task across several LLM agents, each with one job, do you get better code than from a single LLM? And can small models running on a laptop do the job, or does it need the big paid APIs? The short answer: a small, consistent gain in stability for some models, at the cost of speed, and small local models could not do it at all.

The problem

No-code platforms let people build software without writing code. They work well for simple apps but struggle once the work spans many connected steps. LLMs can write code from plain language, but a single model tends to lose the thread on long, multi-step tasks and can hallucinate. Multi-agent systems split the work into roles, but most published ones assume large proprietary models behind an API. That rules them out where data has to stay on the device or the internet is unreliable.

The idea

Give each step of the job to its own agent, run them in a fixed order, and pass each one's output to the next. Then measure it honestly: the same four models, run alone and inside MASON, on the same tasks.

I compared three orchestration frameworks before picking one. AutoGen was powerful but heavy to configure. LangChain is strong at tool use but not built for multi-agent coordination. CrewAI won for its simple YAML setup, clear task order and stable repeated runs.

How it works

Five agents, defined in YAML, run in sequence:

Agents can re-check their own output in a short feedback loop when something looks wrong. Early versions merged roles, such as parsing and analysis, but that blurred responsibilities and made coordination worse. Five roles gave the most consistent results in pilot tests.

A Python harness drives the evaluation. It samples tasks with a fixed seed, runs the crew, imports the generated output.py, runs the benchmark's unit tests against it, and logs time and pass or fail for each task.

Evaluation

Results

Average Pass@1 (M is MASON, S is the single model):

Average execution time per task, in seconds:

What that means:

• MASON was slower for every model. DeepSeek-Chat and Gemini 1.5 Flash more than doubled their time.
• Claude 3.5 Sonnet was the most accurate and stable in both setups.
• DeepSeek-Chat gained most from MASON on MBPP.
• GPT-4o Mini gained a little.
• Gemini 1.5 Flash got worse inside MASON, especially on HumanEval, and failed early in runs.
• HumanEval, the harder set, caused more failures than MBPP for every setup. Task difficulty mattered as much as architecture.

Small local models. Before the cloud runs I tried Qwen2.5, Qwen2.5-Coder, Gemma3, LLaMA 3.1 and 3.2, and DeepSeek R1 through Ollama, from 0.5B to 8B parameters. Models up to 1.5B passed 0% of tasks, 3B to 4B models passed 6%, and 7B to 8B models passed 11%. They ran out of memory, stalled, or produced incomplete output. That answered the original question: as designed, MASON needs a strong model underneath it.

Publication

The work was my undergraduate thesis at Heriot-Watt University Dubai. It was published as "MASON - A Multi-Agent LLM Framework for No-Code Development" in the AAAI 2025 Summer Symposium Series.

Limitations

• Everything ran on a CPU-only laptop, one task at a time, which limited scale.
• The gains were modest. MASON beat single agents on stability more than on accuracy, and always cost time.
• The fixed agent order and shallow prompts likely held back real collaboration between agents.
• The repo holds a sample of the framework plus the raw logs and charts. Some original files were lost to corruption before upload.

What's next

• Design a lighter pipeline for small models, or fine-tune them for it.
• Try adaptive agent flows and better prompt chaining instead of a fixed order.
• Compare against other orchestrators such as LangGraph and AutoGen.
• Repeat the study with open-source models only.