Most LoRA tutorials end the moment the training loss goes down. That is roughly the halfway point. A fine-tuned model nobody can talk to is a file on a disk, and the distance between that file and a chat window someone can actually use is where most of the real work lives.
I have built that path end to end a few times now: training an adapter on a base model like facebook/opt-1.3b, merging it, publishing it to Hugging Face, and putting it behind both a Gradio interface and a plain PHP chat UI. This is the whole route, including the parts the tutorials skip.
What LoRA actually changes
A full fine-tune updates every weight in the model. For a 1.3 billion parameter model that means gradients and optimiser state for all of them, which is a lot of GPU memory for what is often a small change in behaviour.
LoRA (Low-Rank Adaptation) freezes the original weights and learns a small correction next to them. Instead of updating a large weight matrix W, it trains two thin matrices A and B whose product is added on top. For a 2048 by 2048 projection with rank 16, that is 65,536 trainable numbers instead of about 4.2 million, roughly 1.6 percent.
The base model does not move. You are training a small, removable opinion about it.
That has practical consequences beyond memory. The adapter is a few megabytes, so you can keep several for one base model, compare them, and throw away the bad ones without touching the original.
Start with the data, not the code
The training script is perhaps forty lines. The dataset decides whether any of it was worth running. A few rules I now follow every time:
- Pick one prompt format and never deviate. If training examples look like
### Question: ... ### Answer: ..., inference prompts must look exactly the same. A model fine-tuned on one template and queried with another behaves like it was never trained. - A few hundred clean examples beat thousands of noisy ones. LoRA learns fast, including your mistakes.
- Hold some examples back. If you only ever test on what it trained on, you are measuring memory, not learning.
- Read a random sample by eye before training. Ten minutes of reading saves an evening of retraining.
Training the adapter with PEFT
Hugging Face's PEFT library makes the setup short. You load the base model, describe the adapter, and wrap the model with it:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
base = "facebook/opt-1.3b"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
config = LoraConfig(
r=16, # rank of the correction
lora_alpha=32, # scaling for the correction
lora_dropout=0.05,
# attention projections in OPT
target_modules=["q_proj", "v_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
# ...tokenise your dataset, train with Trainer...
model.save_pretrained("./opt-lora-adapter")print_trainable_parameters() is worth the line. If it reports anything other than a small fraction of a percent, the adapter is not attached where you think it is.
On rank: start at 8 or 16. Higher ranks give the adapter more room, which helps with genuinely new behaviour and hurts when the dataset is small, because there is more room to memorise.
Merge before you ship
You can serve a base model with an adapter loaded on top, and during experiments that is the right call. For deployment I merge. Merging folds A times B back into the original weights and produces an ordinary model that any tool can load, with no PEFT dependency and no extra step at inference time.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b")
adapted = PeftModel.from_pretrained(base_model, "./opt-lora-adapter")
merged = adapted.merge_and_unload()
merged.save_pretrained("./opt-merged")
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
tokenizer.save_pretrained("./opt-merged")
# Publish it; private=True keeps it off the public listing
merged.push_to_hub("your-username/opt-1.3b-custom", private=True)Two things catch people out here. Save the tokenizer alongside the merged weights, or every consumer has to guess which one goes with it. And write a short model card: base model, dataset description, prompt format. The prompt format in particular is the one thing a future user cannot work out on their own.
Three ways to talk to it
Once the merged model exists, there are three serving setups I keep coming back to, and each one answers a different question.
- Gradio on a local GPU. The fastest way to find out whether the fine-tune worked. A chat box, the model loaded once, answers in real time. This is a testing bench, not a product.
- Hugging Face hosted inference. Your application sends HTTP requests and someone else owns the GPU. Good for sharing a model or putting it in front of a client quickly.
- A PHP chat UI with a Python backend. Fully self-hosted, no external API, the data never leaves your server. The most work, and the most control.
The self-hosted route has one trap worth naming. If PHP spawns a fresh Python process for every question, every question pays the full cost of loading the model, which for a 1.3B model is seconds before a single token appears. Load the model once in a long-running Python process and have PHP talk to it over a local HTTP port or socket. The difference in perceived speed is enormous.
What I would tell myself before the first run
- Write the evaluation prompts before training. Twenty questions you care about, answered by the base model first. Then you can see what the fine-tune changed instead of guessing.
- Fine-tuning teaches style and format far better than facts. If the goal is "know our documents", retrieval is usually the better tool, and LoRA is how you teach it to answer in your voice.
- Small base models are honest about their limits. A 1.3B model fine-tuned well is fast and cheap and excellent at narrow jobs. It will not become a general assistant, and it should not be sold as one.
- Version the adapter, the data and the prompt format together. Any one of them without the others is impossible to reproduce.
The short version
LoRA made fine-tuning cheap enough that the training run is no longer the hard part. The hard parts are the same ones as any other software: clean inputs, a contract everyone follows (here, the prompt format), and a deployment that does not do expensive work on every request. Get those right and a small custom model becomes something people can actually use. You can see the projects that came out of this on my projects page.
