LRM: Large Reconstruction Model for Single Image to 3D Guide

LRM: Large Reconstruction Model for Single Image to 3D is the 2023 research paper that showed a single large transformer could look at one picture of an object and predict a full 3D version of it in seconds. If you build games with AI, this paper sits behind a lot of the image-to-3D tools you have tried, so it is worth understanding what it does, what its open-source version can and cannot give you, and how to turn a single image into a 3D asset you can actually ship.

This guide covers the architecture in plain language, the follow-up models, how to run OpenLRM on your own machine, the licensing trap that catches game developers, and a practical pipeline for game-ready characters.

What is LRM: Large Reconstruction Model for Single Image to 3D?

The paper LRM: Large Reconstruction Model for Single Image to 3D was submitted to arXiv on 8 November 2023 by Yicong Hong, Kai Zhang and colleagues, and it was accepted at ICLR 2024. Its claim is simple: give the model one image of an object and it predicts a 3D model of that object within about 5 seconds.

Two things made it notable:

• Scale. Earlier single-image methods were often trained on small datasets such as ShapeNet, one category at a time (only chairs, only cars). LRM uses a transformer with about 500 million learnable parameters. A transformer is the same family of neural network used in large language models; it processes data as a sequence of tokens and learns how they relate.
• Data. The model was trained end to end on multi-view data covering around 1 million objects: synthetic renders from Objaverse (a large public library of 3D models) and real captures from MVImgNet (videos of real objects filmed from many angles).

The result generalises to images it never saw, including phone photos and pictures made by image generators. That last point matters for game developers: it is why “generate a concept image, then lift it to 3D” became a workable pipeline.

Before models like this, getting 3D from images usually meant photogrammetry: shoot dozens of photos, match features between them, and align the resulting point clouds through point-set registration, the process of finding the rotation, translation and scale that line up two sets of 3D points. LRM skips all of that. It has learned, from a million objects, what the hidden side of a thing probably looks like, so one view is enough to make a confident guess.

How does LRM work? Encoder, triplane and NeRF

The LRM design has three stages. You do not need to understand the maths to use it, but knowing the stages explains the strengths and the artefacts you will see.

1. The image encoder

The input image is cut into small patches and passed through a pretrained vision encoder, which turns each patch into a feature vector, a list of numbers describing what is in that patch. The open-source version uses a DINOv2 encoder for this step.

2. The image-to-triplane decoder

A transformer decoder converts those image features into a triplane: three flat 2D feature grids aligned with the X, Y and Z axes, like the three walls in the corner of a room. To find out what exists at any point in 3D space, you project that point onto each of the three planes, read the features there and combine them. A triplane is far cheaper to store than a full 3D grid while still describing a whole volume.

3. The NeRF renderer

The triplane features feed a small network that predicts colour and density at any 3D point. That is a NeRF (neural radiance field): a scene stored as a function you can query, rather than as triangles. To render an image, you shoot rays from a virtual camera through the volume and add up colour and density along each ray. During training, the model renders its prediction from other viewpoints and compares those renders with real views of the same object, which is how it learns to guess the unseen sides.

To get a mesh (the triangles a game engine needs), the NeRF’s density field is sampled on a grid and converted into a surface. That conversion step is where game-specific problems start, which we will get to.

What came after LRM: OpenLRM, GS-LRM, Instant3D and others

The paper started a family of “large reconstruction models”. Here is how the main names relate, so the related papers make sense.

Project	Input	Output	What changed
LRM	One image	Triplane NeRF	The original large transformer approach
OpenLRM	One image	Triplane NeRF, video or mesh	Open-source reimplementation with public weights
Instant3D	Text prompt	3D object	Generates a few consistent views first, then reconstructs from them
GS-LRM	A few posed images	3D Gaussian splats	Swaps NeRF for Gaussian primitives
CRM	One image	Textured mesh	Convolutional reconstruction aimed at mesh output
Long-LRM, Real3D	Varies	Varies	Longer input sequences; training on real single-view photos

A “posed” image is one where the camera position is known. Gaussian splats represent a scene as many small coloured blobs, which render quickly and look sharp, but they are still not triangle meshes. For game work, the column to watch is “Output”: anything that is not a clean mesh needs conversion before it behaves like a normal asset in a game engine.

How to run OpenLRM yourself (with code)

The original LRM weights were not published as a download, so most people who want to try the LRM: Large Reconstruction Model for Single Image to 3D approach hands-on use OpenLRM, an open-source implementation released in December 2023. Version 1.1, from March 2024, added weights trained on both Objaverse and MVImgNet.

Pick a model size

OpenLRM publishes six checkpoints: small, base and large, each trained either on Objaverse only (“obj”) or Objaverse plus MVImgNet (“mix”).

• Small: 12 layers, 224-pixel input. Lightest on memory.
• Base: 12 layers, 336-pixel input. The default for the Hugging Face demo.
• Large: 16 layers, 448-pixel input. Most detail, most memory.

The “mix” models have seen real photos, so they tend to be the better pick for camera images; for clean AI concept art either family can work.

Install and run

You need Python, a CUDA-capable GPU and PyTorch 2.1 or newer (the project was tested on 2.1.2). Prepare an RGBA image, or an RGB image on a plain white background; strip backgrounds first. Then:

git clone https://github.com/3DTopia/OpenLRM.git
cd OpenLRM
pip install -r requirements.txt

# Weights download automatically on first run
EXPORT_VIDEO=true
EXPORT_MESH=true
INFER_CONFIG="./configs/infer-b.yaml"
MODEL_NAME="zxhezexin/openlrm-mix-base-1.1"
IMAGE_INPUT="./assets/sample_input/owl.png"

python -m openlrm.launch infer.lrm --infer $INFER_CONFIG \
  model_name=$MODEL_NAME image_input=$IMAGE_INPUT \
  export_video=$EXPORT_VIDEO export_mesh=$EXPORT_MESH

Use infer-b.yaml for base models and infer-s.yaml for small ones. The README also asks you to install xFormers for memory-efficient attention in the encoder. If you run out of GPU memory, lower frame_size in the inference config. Training your own model is a different story: the sample config assumes eight GPUs.

Can you use LRM output in a game?

For a prototype on your own machine, yes. For a game you sell, three problems stand in the way.

The licence

OpenLRM’s code is Apache 2.0, with some components under an NVIDIA proprietary licence. The model weights are licensed CC BY-NC 4.0 and the README says they are for research only and cannot be used commercially. If your game is on a storefront, has ads or takes donations through a paid model, treat assets from these weights as off limits. This is the single most common mistake people make with research models.

The mesh

A surface extracted from a NeRF is dense and irregular. Topology, the way triangles and edges flow across a model, matters for deformation: a character’s elbow bends cleanly only if the geometry around it is sensible. Colour is usually baked into vertices or a single texture with lighting partly painted in, so it will not react properly to your game’s lights. The back of the object, which the model guessed, is often blurrier than the front.

The missing rig

A rig is the internal skeleton of bones that lets a character move, and weight painting decides how strongly each bone pulls each part of the mesh. Reconstruction models produce a static statue. To walk, attack or idle, a character still needs rigging, skinning and animation, then export in a format your engine reads.

So the honest summary of the LRM: Large Reconstruction Model for Single Image to 3D work is this: it solved the hardest research step, guessing a whole object from one view, but a game asset needs several more steps after it.

Single image to a game-ready 3D character with 3D Studio

If what you really want is the outcome the paper points toward, “I have an idea or a picture, give me something I can drop in my game”, the cleanest route is a pipeline that handles every stage after reconstruction too. That is what 3D Studio in Sorceress does, in the browser.

3D Studio chains the full character pipeline into four steps:

1. Generate. AI image generation turns your text prompt into a concept character. This is your single input image, made for the job instead of hunted down.
2. Lift to 3D. One-click image-to-3D produces a fully textured model. This is the stage the LRM research line made possible.
3. Auto-rig and weight paint. A humanoid skeleton is attached automatically and the weights are refined, so limbs deform instead of tearing.
4. Text-to-animation and export. Describe the motion in words, then export to FBX, GLB or GLTF for any engine.

The overview at the 3D Studio page shows the pipeline end to end. If you want more control over the concept art first, generate and compare images in AI Image Gen, which drives several leading image models from one panel, then bring the best one into 3D Studio.

Worked example: a lantern-bearing scout

Say you are making a small night-time exploration game. In 3D Studio you type “hooded scout holding a lantern, full body, T-pose, plain background, stylised low-poly look”. A T-pose (arms straight out) gives the auto-rigger a clear view of the limbs. You lift the chosen image to a textured model, let the humanoid rig attach, then prompt an animation such as “slow cautious walk, lantern held forward”. Export as GLB and load it in a browser game:

import * as THREE from 'three';
import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(50, innerWidth / innerHeight, 0.1, 100);
camera.position.set(0, 1.5, 3);

const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(innerWidth, innerHeight);
document.body.appendChild(renderer.domElement);
scene.add(new THREE.HemisphereLight(0xffffff, 0x444444, 2));

const clock = new THREE.Clock();
let mixer = null;

new GLTFLoader().load('scout.glb', (gltf) => {
  scene.add(gltf.scene);
  if (gltf.animations.length > 0) {
    mixer = new THREE.AnimationMixer(gltf.scene);
    mixer.clipAction(gltf.animations[0]).play();
  }
});

renderer.setAnimationLoop(() => {
  const dt = clock.getDelta();
  if (mixer) mixer.update(dt);
  renderer.render(scene, camera);
});

Want the scout as a 2D sprite for a side-scroller instead? 3D to 2D renders a 3D character or animation into a sprite sheet from any camera angle. For a blocky look, Voxel Studio turns images, prompts or 3D models into rigged, animated voxel creations. You can also hand the GLB to WizardGenie, the Sorceress AI game engine, and describe the game around it.

OpenLRM vs 3D Studio at a glance

	OpenLRM	3D Studio
Setup	Python, CUDA GPU, PyTorch	Runs in the browser
Input	An image you prepare	A prompt (image generated for you)
Output	Static mesh or turntable video	Textured, rigged, animated character
Formats	Mesh export	FBX, GLB, GLTF
Best for	Learning and research	Shipping game characters

Common mistakes with single-image 3D

• Busy backgrounds. Reconstruction models expect one object on a clean background. Clutter becomes lumps of geometry. OpenLRM explicitly asks for RGBA or white-background images.
• Extreme angles. A three-quarter or front view gives the model the most to work with. A top-down shot hides the sides it has to invent.
• Hiding the limbs. Crossed arms or a cape over the legs confuse both reconstruction and rigging. Ask for a T-pose or A-pose for characters.
• Ignoring licences. Research weights, including OpenLRM’s, are non-commercial. Check before anything goes in a release build.
• Expecting the back to match the front. The unseen side is a learned guess. Inspect it in a viewer before committing, and regenerate rather than patching if it is badly wrong.
• Treating materials as finished. Baked colour does not respond to light like a proper material. For props and environments, Material Forge turns an image into a PBR material (physically based rendering: separate albedo, roughness, metallic, normal and AO maps) that reacts correctly to scene lighting.

For a broader comparison of methods, see how to make a 3D model from a 2D image, and if you want free hosted demos of research models, image-to-3D Spaces on Hugging Face covers which ones hold up for game assets.

The verdict

The LRM: Large Reconstruction Model for Single Image to 3D paper proved that one large transformer, trained on about a million objects, could turn a single image into a plausible 3D object in seconds. Its encoder, triplane and NeRF design shaped a whole family of follow-ups, and OpenLRM lets you run that idea locally if you have a GPU and want to learn how it works.

For a game, reconstruction is only one step. You also need commercially usable output, a skeleton, animation and a standard export format. Use OpenLRM to understand and experiment; use 3D Studio when you want a prompt to become a rigged, animated character in FBX, GLB or GLTF. To see where it fits alongside sprites, tilesets, audio and the engine, browse every Sorceress tool in one guide.