WorldGen: Generate Any 3D Scene in Seconds

Author 👨‍💻: Ziyang Xie Contact Email 📧: ziyangxie01@gmail.com
Feel free to contact me for any questions or collaborations!

🌟 Introduction

🌏 WorldGen can generate 3D scenes in seconds from text prompts and images. It is a powerful tool for creating 3D environments and scenes for games, simulations, robotics, and virtual reality applications.

Instant 3D Generation ⚡️ : Create full 3D scenes from input data in seconds
360° Free Exploration 🧭 : WorldGen supports free 360° consistent exploration of the generated 3D scene with loop closure.
Diverse Scenes Support 🌈 : WorldGen supports both indoor and outdoor scenes, both realistic and unrealistic scenes in any style.
Flexible Rendering 📸 : WorldGen supports rendering at any resolution with any camera setting and trajectory in real-time.

Two lines of code to generate a 3D scene in seconds

# Use our API to generate a 3D scene
worldgen = WorldGen()
worldgen.generate_world("<TEXT PROMPT to describe the scene>")

Text-to-Scene Generation

Image-to-Scene Generation

News and TODOs

📦 Installation

Getting started with WorldGen is simple!

# Clone the repository 
git clone --recursive https://github.com/ZiYang-xie/WorldGen.git 
cd WorldGen

# Create a new conda environment
conda create -n worldgen python=3.11
conda activate worldgen

# Install torch and torchvision (with GPU support)
pip3 install torch torchvision

# Install worldgen
pip install .

# Install DA-2 (360 depth estimation) -- use --no-deps to avoid version conflicts
pip install git+https://github.com/EnVision-Research/DA-2.git#subdirectory=src --no-deps

# Install pytorch3d dependencies
pip install git+https://github.com/facebookresearch/pytorch3d.git --no-build-isolation

# 🔥 [New feature]: If you want to use the ml-sharp experimental feature, you need to install the ml-sharp dependencies
pip install -e submodules/ml-sharp

# You should also accept the license of the gated model (FLUX.1-dev).
# https://huggingface.co/black-forest-labs/FLUX.1-dev
# Login to Hugging Face and accept the license.
# huggingface-cli login

🕹️ Quick Start / Demos

We provide a demo script to help you quickly get started and visualize the 3D scene in a web browser. The script is powered by Viser.

# Generate a 3D scene from a text prompt
python demo.py -p "A beautiful landscape with a river and mountains"
# Indoor scene example
python demo.py -p "A well-designed cozy bedroom"

# 🔥 New feature: Generate a 3D scene using the ml-sharp experimental feature (It may produce better results than the default mode)
python demo.py -p "<TEXT PROMPT to describe the scene>" --use_sharp

# Generate a 3D scene from an image
python demo.py -i "path/to/your/image.jpg" -p "<Optional: TEXT PROMPT to describe the scene>" --use_sharp

# Generate a 3D scene in mesh mode
# Make sure you installed my customized viser to correctly visualize the mesh without backface culling
# pip install git+https://github.com/ZiYang-xie/viser.git

python demo.py -p "A beautiful landscape with a river and mountains" --return_mesh

After running the demo script, A local viser server will be launched at http://localhost:8080, where you can explore the generated 3D scene in real-time.

🎮 Advanced Usage

WorldGen API

Quick start with WorldGen (mode in t2s or i2s) and generate your first 3D scene in seconds:

📝 Text to Scene: Generate a 3D scene from a text prompt

# Example using the Python API
from worldgen import WorldGen
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

worldgen = WorldGen(mode="t2s", device=device, low_vram=False) # Set low_vram to True if your GPU VRAM is less than 24GB.
splat = worldgen.generate_world("<TEXT PROMPT to describe the scene>")
splat.save("path/to/your/output.ply") # Save splat file as a .ply file, which can be loaded and visualized using a standard gaussian splatting viewer

🖼️ Image to Scene: Generate a 3D scene from an image

worldgen = WorldGen(mode="i2s", device=device, low_vram=False) # Set low_vram to True if your GPU VRAM is less than 24GB.
image = Image.open("path/to/your/image.jpg")
splat = worldgen.generate_world(
    image=image,
    prompt="<Optional: TEXT PROMPT to describe the image and the scene>",
)

🏡 Generate a 3D scene in mesh mode

mesh = worldgen.generate_world("<TEXT PROMPT to describe the scene>", return_mesh=True)
o3d.io.write_triangle_mesh("path/to/your/output.ply", mesh) # Save mesh as a .ply file

Tip

We also support background inpainting for better scene generation, but it's currently an experimental feature, which may not work for all scenes.
It can be enabled by setting WorldGen(inpaint_bg=True).

# If want to use background inpainting feature, install iopaint
pip install iopaint --no-dependencies

Free-viewpoint Exploration in 3D Scene

Note

WorldGen internally support generating a 3D scene from a 360° panorama image 📸, which related to how WorldGen works: You can try it out if you happen to have a 360° panorama (equirectangular) image. Aspect ratio of the panorama image should be 2:1.

 pano_image = Image.open("path/to/your/pano_image.jpg")
 splat = worldgen._generate_world(pano_image=pano_image)

⭐️ Star History

Give a star to WorldGen if you like it!

📚 Citation

If you find this project useful, please consider citing it as follows:

@misc{worldgen2025ziyangxie,
  author = {Ziyang Xie},
  title = {WorldGen: Generate Any 3D Scene in Seconds},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/ZiYang-xie/WorldGen}},
}

🤝 Acknowledgements

This project is built on top of the follows, please consider citing them if you find them useful:

Some of the core methods and ideas in this project are inspired by the following projects, special thanks to them:

WonderWorld [RGBD to GS conversion]
WorldSheet [Mesh Generation]

WorldGen