How synthetic images train vision models when real photos are scarce
You can train a computer vision model on images rendered in a 3D engine instead of photographed, and the labels come free, because the engine built the scene and knows which pixel belongs to which object. In the three studies that compared mixes of synthetic and real data, the best results came from synthetic pre-training followed by fine-tuning on real images, sometimes only a few dozen.
Why labelling real photos holds projects up
A vision model learns from examples, and each example needs an answer attached: a box around every car, or a colour mask showing which pixels are leaf, stem or fruit. People draw those answers by hand. NVIDIA researchers note in their paper on domain randomisation that labelling 3D poses or pixel-level segmentation by hand can take a person a substantial amount of time for a single image. The figure that matters is your own: time a labeller on twenty of your images, then multiply by the images you would need each season. That is the cost a generator has to beat.
Agriculture has it harder. Researchers at Wageningen University & Research describe the need for large datasets labelled pixel by pixel as a current bottleneck for crop image segmentation. Each growth stage is only in the field for part of the season; miss it and you wait for next year.
A rendering engine sidesteps both problems. Unity Technologies’ own Perception paper says the engine “has perfect information about the scenes it renders”, which removes the time and cost of human annotation and review. In practice you tag each 3D asset once (“wheat”, “broadleaf weed”, “soil”) and the software works out the answer for every frame. That is what pixel-perfect labels means: masks and boxes calculated from the scene itself, with no tracing by hand and no two labellers disagreeing about where a leaf ends. The same paper lists the standard label types: 2D and 3D bounding boxes, semantic segmentation (every pixel given a class), instance segmentation (every pixel given a class and a specific object) and keypoints (named points on an object, used for tasks such as pose estimation).
Domain randomisation: making reality one more variation
Domain randomisation was introduced in a 2017 paper by Josh Tobin and colleagues on moving networks trained in a simulator into real-world use. They describe it as a way to teach a model on rendered pictures so that it still works on real photographs, achieved “by randomizing rendering in the simulator”. The reasoning fits in one sentence of theirs: “With enough variability in the simulator, the real world may appear to the model as just another variation.”
Using only simulator images with deliberately unrealistic random textures, they trained an object detector that was accurate to 1.5 cm in the real world. The authors say this was, to their knowledge, the first time a deep network trained only on simulated images transferred to the real world for robotic control. Keep the context in mind: the task was object localisation, finding simple shapes so a robot arm could grasp them in clutter. It proves the principle. It says nothing directly about crops in a field.
NVIDIA’s Jonathan Tremblay and colleagues took the idea to car detection on the KITTI driving dataset, in work presented at CVPR Workshops 2018. Their simulator randomised lighting, pose and textures “in non-realistic ways” to force the network onto the features that matter. The recipe in the full paper is simple: random textures on the cars and on floating distractor objects, a random number of lights of different types, a random camera viewpoint and a random background image. They built it as an in-house plug-in for Unreal Engine 4; the photorealistic dataset they compared against, Virtual KITTI, was made in Unity.
The most useful part of their paper is the ablation, where they switched off one randomised element at a time to see what each contributed. With everything randomised, the detector scored 73.7 AP (average precision, a standard detection score where higher is better). Fixing the lighting dropped it to 67.6, a fall of 6.1 points. Leaving the cars without random textures dropped it to 69.0, a fall of 4.7. In this test, lighting mattered more than surface texture.
They also found a limit. Their simple randomisation ignored context, such as the way parked cars line up along a street, and they suspect this meant some variations in the real test set were never generated. For crops the equivalent is layout. Row spacing, drill lines and tramlines (the wheel tracks a tractor leaves through a crop) follow rules that random placement will not reproduce, so they have to be modelled explicitly and only then varied.
How realistic the scene needs to be
The sources partly disagree here. Tremblay’s team argue that crude, non-photorealistic randomisation can compete with carefully built photorealistic worlds. Wageningen’s agricultural work went the other way: R. Barth and colleagues generated their plant models from 21 plant properties measured at 115 positions on 15 real stems, 3D-scanned the fruit and photographed the textures.
Our view is that both are right about different parts of the image. For crops, the thing the model must recognise is the plant’s form at a given growth stage, and that is worth modelling carefully. Everything around it (soil, residue, sun angle, cloud, camera height) is where wide, even unrealistic, randomisation pays.
How much real data you still need
In all three studies below, adding real images on top of synthetic pre-training gave the best result the authors recorded, and in the two that report a real-only score, it beat real data alone. In two of the studies, 76 and 30 real images were enough: to beat a real-only baseline in one, and to reach the best score in the other. The scores are not comparable across rows, because each uses a different task and measure, but within each row the direction is clear.
| Study | Task | Synthetic data | Real images used | Result |
|---|---|---|---|---|
| Tremblay et al., NVIDIA (2018) | Car detection, KITTI | Domain-randomised renders, Unreal Engine 4 | All 6,000 real KITTI images for fine-tuning | 98.5 AP, which the paper says is 2.1% better than real data alone |
| Borkman et al., Unity Technologies (2021), SynthDet | Detecting 63 grocery products | 400,000 images | None, 76, or 760 | mAP 0.381 synthetic only, 0.528 with 76 real, 0.684 with 760 real; 760 real alone scored 0.450 |
| Barth et al., Wageningen (2019) | Segmenting 7 sweet pepper plant parts | 10,500 Blender renders from 42 procedural plant models | 30 | Highest score on all 7 classes (mean IoU 0.40), reached with only 30 real images |
Mean IoU (intersection over union) measures how closely a predicted mask overlaps the true one, from 0 to 1. In SynthDet, mAP is mean average precision averaged across overlap (IoU) thresholds from 0.5 to 0.95; higher is better. In our view, 0.40 is low in absolute terms, and thin parts such as stems are likely to pull IoU down, because a mask a few pixels off along a narrow edge loses much of its overlap. The finding is relative: after synthetic pre-training, 30 real images were enough to match the best result the team got.
Tremblay’s team summarise it: “With additional fine-tuning on real data, the network yields better performance than using real data alone.” In SynthDet, 400,000 synthetic images on their own scored below a model trained on 760 real photos, yet synthetic plus just 76 real beat that real-only baseline. And in the Wageningen sweet pepper study, part of the EU’s SWEEPER harvesting robot project, the authors found “only 30 empirical images were required to obtain the highest performance on all 7 classes” once the network had been pre-trained on synthetic data. In that study, learning on the synthetic set levelled off at around 3,000 images; our reading is that beyond a few thousand renders, variety is likely to matter more than volume. Their Capsicum dataset of 10,500 synthetic and 50 real labelled images is public.
The SynthDet authors note their model was still “struggling in situations with complex lighting”, exactly the kind of weakness a small real test set exposes and a synthetic test set hides. In a field, that means low evening sun or broken cloud throwing hard shadows across the rows: a test set made only of renders tells you how the model copes with the lighting you thought to render, and nothing about the lighting you didn’t.
How a crop data pipeline fits together
AgriSynth is the clearest example from our own work. An AgriTech company needed labelled crop imagery to train its computer vision models, without waiting for the seasons or labelling every plant by hand. We built AgriSynth, a procedural generator in Unity that grows photorealistic crop scenes from a custom plant-growth model and outputs every frame with pixel-perfect labels, in a project funded by Innovate UK, the UK’s innovation agency.
What it changed is timing. A photographed dataset contains only the growth stages that were in the field when someone went out with a camera. A growth model can produce a given stage on any day of the year, so training data arrives on demand instead of once a season.
A generator of this kind has five parts, and each one answers a point the research raises:
- A plant-growth model that builds geometry from parameters instead of reusing a fixed library of meshes, so no two plants repeat.
- A scene randomiser for everything that is not the crop: soil, straw and residue, weed density, light and camera position, with field layout such as row spacing modelled first and then varied.
- Labelling from the scene itself. Each plant type and surface is tagged once and the engine writes the mask for every frame.
- Seeded, repeatable randomness. The Perception paper used deterministic random sampling so that data generated in the cloud could be reproduced in the editor for debugging. When one frame in a million looks wrong, you need to rebuild it exactly.
- Checks before and after training. Dataset statistics, which the Perception paper says help decide whether you need more data or different randomisation, then scoring against hand-labelled real photos.
If you were planning to use Unity Perception
Unity Technologies has since discontinued the Perception package. Its GitHub page now says it “is no longer supported by Unity” and sends users to GitHub issues for community support. Tutorials built around it, including the examples in the 2021 paper, describe a tool that will not receive fixes, so a new project has to choose another route.
Forking Perception is possible because the code is Apache 2.0 licensed. It makes sense if you already have a Perception project that works and engineers willing to own the fork, since nobody upstream will fix it.
Rendering in Blender from procedurally generated models is the route Wageningen took for its sweet pepper images. It suits offline rendering, where the quality of each frame matters more than fast iteration on the scene. An in-house engine plug-in is how NVIDIA built its car scenes in Unreal Engine 4, a reasonable choice for a team already working in Unreal.
The Perception paper compares tools such as NVIDIA Isaac Sim, NVISII and BlenderProc, which generate labels for common vision tasks, though their support for domain randomisation varies. For crops, Wageningen had to build procedural plant models from field measurements, and the growth model is usually yours to write.
AgriSynth is a procedural generator in Unity built around a custom plant-growth model, outputting every frame with pixel-perfect labels. Building those five parts for a specific task is the core of our AI training data generation work.
Questions to settle before you generate a single image
From how we scope this kind of project, these are the questions that most affect whether synthetic data helps:
- What exactly will the model see in use: camera, lens, mounting height and angle, time of day, weather. The virtual camera should match it before anything else is randomised.
- Which label type the model needs (boxes, semantic masks, instance masks or keypoints), because it changes how assets are tagged.
- Which situations are rare, seasonal or unsafe to capture for real, and therefore worth over-representing in the synthetic set.
- Whether you can collect a small, carefully labelled real set. The studies above used as few as 30 or 76 real images for fine-tuning; you also want a separate real test set that never touches training and covers every class and every condition you care about, such as growth stage, light and soil.
- How you will measure the gap: score the synthetic-only model and the fine-tuned model on the same real test set, and compare.
For the test set, coverage matters more than a headline count. If a growth stage or lighting condition is missing from it, you cannot tell whether the model handles it. For a sense of scale, Wageningen published 50 hand-labelled real images alongside its 10,500 renders, and the SynthDet team collected 1,267 real photos of its 63 products.
On cost, a general ratio between synthetic and hand-labelled data would not hold across crops and label types. The comparison to make is the one from the start of this article: your measured labelling time per image, multiplied by a season’s worth of images, set against a generator that, once built, spends compute time instead of people’s time.
If your answers to the last two questions are “no” and “we haven’t decided”, settle those first: collect and label a real test set that covers your growth stages and light, and agree how you will score both models against it.
If you want a second opinion on those questions for your project, get in touch.