Meta 3D Gen

Such a silly argument. Fixing topology is a nearly solved problem in geometry processing. (Or just start with a good topology and 'paste' a texture onto it like they develop techniques for here.)

That doesn’t matter for things like 3D printing and CNC machining. Additionally, there are other mesh fixer AI tools. This is going to be gold for me.

Afaik, there's no topology - it outputs signed distance fields, not meshes.

Credit where it's due, unlike most of these papers they do at least show some of their models sans textures on page 11, so you can see how undefined the actual geometry is (e.g. none of the characters have eyes until they are painted on).

They seem to admit as much in Table 1 which indicates this model is not capable of "clean topology". Somewhat annoyingly, they do not discuss topology anywhere else in the paper (at least, I could not find the word "topology" via Ctrl+F).

VR is especially unforgiving of "fake" detailing, you need as much detail as possible in the actual geometry to really sell it. That's the opposite how these models currently work, they output goopy low-res geometry and approximate most of the detailing with textures, which would be immediately register as fake with stereoscopic depth perception.

Regardless of cost, the meta verse is a dead end concept - at least until we get Ghost in the Shell-style computer-brain interfaces. Second Life probably was peak popularity for this kind of meta verse.

There are a few services that do this already, but they are all somewhat lacking, hopefully Meta's paper / solution brings some significant improvements in this space.

The existing ones:

- Meshy https://www.meshy.ai/ one of the first movers in this space, though it's quality isn't that great

- Rodin https://hyperhuman.deemos.com/rodin newer but folks are saying this is better

- Luma Labs has a 3D generator https://lumalabs.ai/genie but doesn't seem that popular

I'm excited to see the main deterrents to Indy gave dev: art and sound, get AI'd away. A single developer could use an off-the-shelf game engine and some AI generated assets (perhaps combining with whatever they can buy cheap) to develop some fun games.

Still, getting from still models to something that animates is necessary:(

I have to be a soggy blanket person, but there are some pretty strong reasons why 3D generative AI is going to have a much shallower adoption cycle than 2D. (I founded a company that was likely the first generative AI company for 3D assets)

1. 3D is actually a broad collection of formats and not a single thing or representation. This is because of the deep relation between surface topology and render performance. 2. 3D is much more challenging to use in any workflow. Its much more physical, and the ergonomics of 3DOF makes it naturally hard to place in as many places as 2D 3. 3D is much more expensive to produce per unit value, in many ways. This is why, for example, almost every indie web comic artist draws in 2D instead of 3D. In an ai first world it might be less “work” but will still be leaps and bounds more expensive.

In my opinion, the media that have the most appeal for genAI are basically (in order)

- images - videos - music - general audio - 2D animation - tightly scoped 3D experiences such as avatars - games - general 3D models.

My conclusion from being in this space was that there’s likely a world where 3D-style videos generated from pixels are more poised to take off than 3D as a data type.

I've been bullish[1] on this as a major aspect of generative AI for a while now, so it's great to see this paper published.

Me too. My first thought when seeing 2D AI generated images was that 3D would be a logical next step. Compared to pixels, there's so much additional data to work with when training these models I just assumed that 3D would be an easier problem to solve than 2D image generation. You have 3D point data, edges, edge loops, bone systems etc and a lot of the existing data is pretty well labeled too.

Autodesk has been building practical 3d models for years with generative design. I have to imagine it's only getting better with these recent advances, but I'm not immersed in the space.

The gap from demos/papers to reality is huge. ML has a bad replication crisis.

SOTA text-to-image 5 years ago was complete garbage. Most people would think the same. Look how good it got now.

You have to look at this as stepping stone research.

The field evolves quickly. The Meta 3D Gen paper states that Rodin Gen-1 [1,2,3] has a clean topology. As a non professional, the wireframes from some examples indeed look nice.

[1]: https://github.com/CLAY-3D/OpenCLAY

[2]: https://hyperhuman.deemos.com/

[3]: https://huggingface.co/spaces/DEEMOSTECH/Rodin

Haven’t tried all, but yeah, pretty bad so far

We tried them too. My wife is a 3D artist, but we needed a lot of assets that frankly weren't that important. The plan was to use the output as a starting point and improve as needed manually.

The problem is that the output you get is just baked meshes. If the object connects together or has a few pieces you'll have to essentially undo some of that work. Similar problems with textures as the AI doesn't work normally like other artists do.

All of this is also on top of the output being basically garbage. Input photos ultimately fail in ways that would require so much work to fix it invalidates the concept. By the time you start to get something approaching decent output you've put in more work or money than just having someone make it to begin with while essentially also losing all control over the art pipeline.

I have too, and you’re quite right. Also the various 2D-to-3D face generators are mostly awful. I’ve done a deep dive on that and nearly all of them seem to only create slight perturbations on some base model, regardless of the input.

for an artist to provide some input

Sure, the results are excellent.

Or, just throw a PS1 filter on top and make some retro games

There's so many creative ways to use these workflows. Consider how much people achieved with NES graphics. The biggest obstacles are tools and marketplaces.

Unless the topology is good it may not be worth it.

Sure.

The paper doesn't show topology, UVs or the output texture, so we're left to assume the models look something like what you'd find when using photogrammetry: triangulated blobs with highly segmented UV islands and very large textures. Fine for background elements in a 3D render, but unsuitable for use in a game engine or real-time pipeline.

In my job I've sometimes been given 3D scans and asked to include them in a game. They require extensive cleanup to become usable, if you care about visual quality and performance at all.

Generally these things are useless for 3d artists because the wireframe is useless for them.

That would be great. I've learnt some Unity, building my own little VR game, and I dread having to learn Blender or any other tool to make more detailed shapes/models. I've tried a few GenAI tool to create 3D models and the quality is not useable.

The paper suggests Rodin Gen-1 [0] is capable of image-to-shape generation.

[0] https://hyperhuman.deemos.com/rodin

I think what they did here was go text prompt -> generate multiple 2d views -> reconstruction network to go multiple 2d images to 3d representation -> mesh extraction from 3d representation.

That's a long way of saying, no, I don't think that this introduces a component that specifically goes 2d -> 3d from a single 2d image.

Meshroom, if you have enough images ;)

This a pipeline for text to 3D.

But it's using for 3D gen, a model that is more flexible:

https://assetgen.github.io/

It can be conditioned on text or image.

Meta has spent like $50bn on their Metaverse line item since 2021 and hasn't stopped

that probably means a bunch of H100's now for this Meta 3D Gen thing, and other yet unnannounced things still incubating in a womb of datasets

zuck has said before that ML will help make the "metaverse" more viable.

he still needs a moat with its own ecosystem like the iphone

The obvious major caveat with MeshAnything is that it only scales up to outputs with about 800 polygons, so even if their claims about the quality of their topology hold up it's not actually good for much as it stands. For reference a modern AAA game character model can easily exceed 100,000 polygons, and models made to be rendered offline can be an order of magnitude bigger still.

The ultimate end goal is a VR game with infinite detail. Sword Art Online, however, remains fiction. Perhaps for the best.

It seems to be very well compared to the alternatives, however there's a long way to go forward indeed

Interesting, but what are the practical uses of 3d assets beyond gaming

Probably many areas that we already use 3D assets/texturing for. Maybe objects to fill out an architectural render, CG in movies/TV shows, 3D printing, or just as an inspiration/mock-up to build off of. I'd imagine this generator is less useful for product design/manufacturing at the moment due to lack precise constraints - but maybe once we get the equivalent of ControlNets.

If weights are released, it may also serve as a nice foundation model, or synthetic data generator, for other 3D tasks (including non-generative tasks like defect detection), in the same way Stable Diffusion and Segment Anything have for 2D tasks.

I cannot see VR replacing the interactions we have. It requires cumbersome, expensive hardware

Currently sure, but it's been a reasonably safe bet that hardware will get smaller and cheaper. Something like the Bigscreen Beyond already has a fairly small form factor.

But, I feel you're basing judgement of a 3D generator on one currently-niche potential use of 3D assets, that being VR/AR user interfaces (and in particular ones intended to replace a phone rather than, for instance, the interactive interfaces within VR games/experiences).

The potential availability of very expensively (in terms of computing power) generated assets doesn't change that

Even just comparing computing power and not the human labour required, this is probably going to be an extremely cheap way to generate assets. The paper reports 30 seconds for AssetGen, then a further 20 seconds for TextureGen - both being feed-forward generators. They don't mention which GPU, but previous similar models have ran in a couple of minutes on consumer GPUs.

Ya would be cool if we had something open that competed with rodin, but just like elevenlabs for voice, seems closed is gonna be ahead for a while