Transforming Design: Using AI for People in Architectural Renderings
I personally really enjoy using AI. With AI, gone are the days of painstakingly adding 2D people to renders, cutting them out and then awkwardly adding shadows and highlights. It would take me at least a day to source and integrate 2D people for populated images, which slowed down production time significantly. Of course, there is the option of using 3D people, but they still have not reached a high enough level of realism and still look slightly uncanny.
Clients love to populate images; it helps bring the image to life, shows the function of the space and the scale of architectural elements, so using AI is a great way to solve the problem of unnaturally integrated people in your images.
Through trial and error I have found the best pipeline that works for me when creating AI people in my renders, and I will highlight my overall process using Stable Diffusion (SD) as my generator.
Set up Stable Diffusion
Setting up SD is not very intuitive and you'll most likely need to follow a tutorial. I recommend watching Matt Wolfes video on installing Stable Diffusion locally on your computer.
Download a Stable Diffusion Checkpoint
A checkpoint contains all the learned parameters and can be used to continue training, fine-tune the model on specific data, or generate images directly. Basically you need one to create high quality images. SD comes with a checkpoint already downloaded but I recommend downloading the checkpoint Realistic Vision. As mentioned in the title, it creates photo real results which is what we need for our 2D People. These checkpoints take up quite a lot of storage so I recommend only downloading one or two.
Add 3D people to your image and render
It is important to use 3D people as their lighting and shadows will seamlessly integrate with the rest of the environment. The more information you can give Stable Diffusion going in, the more controlled your results will be.
Original image with 3D people
4. Crop your 3D person
Crop your image to just contain your 3D person and some of the environment. Make sure that the size of your image is no larger than 1024 x 1024, if it is larger then downsize the image and upscale it later using Ultimate upscale.
5. Settings
Below are screenshots showing my general settings for SD. Firstly, I bring my image into Inpaint; Inpaint allows you to mask the areas that you wish to generate resulting in more precision and quicker generations. I make sure I have my realistic vision checkpoint selected and then scroll down to further adjust my settings. Generally for improving 3D people, I use quite high sampling steps from the beginning as the generation time for in paint is pretty quick.
For scale I usually select ‘Resize by’ and make sure it's at 1, this means the scale or size of the image won't change from the original. However the size of the original image in this example is 278 x 557 pixels. That's pretty small, so when I press generate it gives me a totally distorted face. I can solve this problem by using the ultimate SD upscaler and scaling it by maybe 1.5x giving me a larger image to work with which will help with all the details. I can then scale down the image to 278 x 557 pixels in photoshop later.
As you can see in the screenshot below, I have used a Control Net to improve the accuracy of the generation. A Control Net makes sure that your generation doesn't steer too far from the original image you are generating from. There are many different types of Control Nets you can use, I mostly use Canny Edge Detection.
Stable Diffusion Settings
Stable Diffusion settings
6. My generation process
First, I'll mask out an area such as the face and generate 3 options by selecting batch size 3. I will then choose the best option from those 3 and drag that into Inpaint and regenerate again. I can do this up to 6 times before I am finally happy with the results. I will then move on to the shirt or to the hair for example. It's a very iterative process that takes patience and experimentation to produce desired results
7. Prompts
Below is a screenshot of my basic prompts, I have these saved in a pre-set so I can quickly add them to any generation I'm working on. You can put the more important words in brackets, and even double brackets. In this case I put any specific prompts in double brackets, for example: ((smiling brunette woman walking)).
Top tip: to stop your AI generated people from looking like literal Greek gods, I suggest you use prompts like “Middle aged” or “Messy hair” or “no muscles”. AI loves to generate unnaturally attractive people with big umm…. assets. These prompts can really help bring some human-ness back to your AI person.
8. Add your AI person into your original image
If you need to, resize your image and mask in your AI person over your 3D rendered person in Photoshop
And you’re finished!
Final image with AI people