Local Compute & Custom LoRA Training for Enterprise Design

In the creative industry, Generative AI offers incredible speed, but it often comes with a hidden cost: data privacy. To use popular tools like Midjourney or ChatGPT, studios must upload their proprietary designs and intellectual property to external servers. For a brand with a unique visual identity, “training the competition” with your own data is a major risk.

My objective was to build a private, offline AI workflow that brings the power of generative tools inside the studio’s own walls. The goal was to prove that a designer could create professional, on-brand product photography on their own hardware, without a single byte of data ever leaving the company’s control.

To test this, I simulated a workflow for a made-up high-end jewelry company. This company has a strict visual style: every ring is photographed from the exact same three angles, with identical lighting and a grey background.

I would then utilise an open-source generative AI model, run in a local environment, and train it to become the company’s own “virtual photographer” through the use of fine-tuned low-rank adaptations. Could I take a raw photo of any new ring design and have the AI instantly generate a set of polished, studio-quality marketing images that look exactly like they were shot by the company’s lead photographer — all while staying completely offline?

To understand why I built a custom workflow rather than training a model from scratch (hint: it’s because I’m not a billionaire), it helps to look at the difference between a Base Model and a LoRA through the lens of education.

The Base Model — the “Ivy League Student”: imagine a brilliant student who has read every book in the Library of Congress and memorised every art style in history. But because this student knows too much, they are unstable. Ask them to “draw a ring” and they might draw a plastic toy ring, a 70s mood ring, or a cartoon. They have too much to draw from, so their output is inconsistent.

Building a custom model from scratch requires you to fund the equivalent of 18 years of basic education (general visual understanding) before you can even get to the specific topic of rings. Training a foundation model like Stable Diffusion or Flux costs hundreds of millions, if not billions of dollars — financially impossible for a design studio.

The LoRA — the “PhD Course”: I can’t afford to raise a new student, but I can take this brilliant student and send them through a PhD course on my specific topic. Using my dataset (the company’s jewelry photos), I didn’t need to teach the student what a ring is — I just needed to teach it exactly how our rings look. This turns a generalist into a specialist.

For a style-transfer LoRA, consistency is far more valuable than volume. I selected 14 unique rings from the archive, but included 3–4 distinct angles of every item, resulting in a training set of a little over 50 high-fidelity images. Feeding the model multiple angles forces the neural network to “triangulate” the concept — it stops seeing the ring as a flat cluster of pixels and starts understanding it as a 3D object that exists in space.

To make this PhD effective, I manipulated the model’s weights using Trigger Words (a unique identifier like c0mpany_style that activates the specialist knowledge), Class Tokens (anchoring everything as a sub-category of Ring), and Description Tags (tagging variable elements like “gold” or “diamond” so the model memorises the lighting and angle rather than the material).

I used Captioning via Text-Encoder Training, orchestrated with the industry-standard script Kohya_ss. Token Shuffling and Dropout occasionally “hides” tags during training, forcing the model to learn that the studio style persists even when the material changes.

Training a high-fidelity LoRA for SDXL is computationally expensive. A dedicated local workstation capable of training these models efficiently would require an investment in the range of $10,000 to $15,000 for enterprise-grade hardware. As a solo designer building a proof of concept, this wasn’t feasible, so I used Amazon Web Services (AWS) to overcome the barrier.

The cost of renting an AWS GPU-accelerated EC2 instance was roughly $3.00 AUD per hour on-demand, and with the total project taking around 8 hours, the total cost came to right around $25.00 AUD. My use of AWS was purely a financial decision, not a technical requirement — for an SME concerned with IP privacy, a $15,000 hardware investment is a standard operational expense, and the workflow is entirely valid with training brought in-house.

If training is the “education” phase, then Inference is “doing the job.” For this step I used ComfyUI, a node-based interface that functions like a circuit board for precise control over the AI pipeline. I started with an optimised open-source SDXL workflow and inserted my custom LoRA via additional loader nodes.

To ensure professional accuracy, I wired in ControlNet nodes — a kind of “digital tracing paper” forcing the AI to strictly adhere to the geometry of my input image while applying the style from my LoRA. Once established, the entire process ran locally; I could disconnect the internet and watch the GPU generate a near-perfect, brand-compliant image in minutes.

The most impressive achievement was the model’s ability to generalise the lighting environment. Whether generating a simple silver band or a complex three-stone engagement ring, the LoRA successfully applied the same consistent “soft grey” studio background and clinical lighting setup.

The most significant finding was that the AI output is rarely “client-ready” straight out of the pipeline. While lighting and angles were often correct, subtle hallucinations — asymmetrical settings or warped geometries — persisted in about 30–40% of generations. The final 10% of polish still required a human designer’s intervention via post-processing and Photoshop compositing.

Restricting the training set to ~14 ring styles also caused occasional “overfitting,” where the model reverted to memorised shapes. A production-grade LoRA would likely require 100–150 images, rigorously balanced between metal types and cuts. Training on SDXL also meant battling the base model’s bias toward “dramatic, moody lighting,” whereas product photography demands a clean, clinical aesthetic — I spent significant cycles “un-teaching” these tendencies. I’m currently investigating newer architectures like Alibaba’s Z-Image Turbo and Qwen-Image.

Ultimately, this project proves that Local AI is a viable path for enterprise IP protection. The true value lies not in replacing the designer, but in empowering them with a custom, privacy-focused workflow that lives securely within their own studios.