Fashion retail / Influencer marketing2026

Outfit Recommender and Influencer Portfolio Optimizer

A fashion recommendation and influencer marketing system built around two deliberately separate embedding spaces: raw Fashion-CLIP for general visual and text similarity, and a 256-d slot-conditioned compatibility space learned by an OutfitTransformer-style encoder trained on 27,000 real Polyvore outfits. On top of it sit outfit completion, text-to-outfit composition, per-garment influencer analysis with Grounding DINO detection, and budget-constrained influencer portfolio selection solved both greedily and exactly as a MILP. A dedicated cross-attention adapter aligns studio product photography with candid social photography, so a brand's catalogue and an influencer's aggregate style are directly comparable.

0.888
Compatibility AUC (held-out): 27,000
Outfits used for training: 63.8%
Cross-domain recall@1: 1,030
Influencer profiles ingested

Approach

Trained an OutfitTransformer-style compatibility encoder: a partial outfit plus a target-slot query token passes through a self-attention encoder to produce the embedding of the ideal item for that slot, with candidates embedded as leaves so ranking is a plain dot product. Trained with in-batch InfoNCE plus an explicit same-slot hard-negative term on 27,000 real Polyvore outfits.
Built the ingestion layer around Fashion-CLIP image and text embeddings in Chroma, plus a structured attribute schema (category, color, pattern, fit, style tags, formality, garment gender) extracted per garment by a vision-language model into SQLite, giving explainable filtering alongside embedding similarity.
Analysed influencer posts per garment rather than per photo: Grounding DINO open-vocabulary detection crops individual items, then each crop runs the same attribute plus embedding pipeline as the catalogue, so a brand's products and an influencer's aggregate style land in comparable coordinates.
Compared four alignment strategies on 350 detected crops over 5 seeds: the production compatibility space, raw Fashion-CLIP, background-removal segmentation, unsupervised DANN, and a slot-supervised cross-attention adapter. A linear probe quantified how much category information transfers across the two photography domains (73.9% within domain, 20.3% across).
Refined the winning adapter through measured iterations: a structure-preservation term to keep within-slot spread, per-slot loss averaging so small slots weigh as much as large ones, a rank ensemble with raw cosine so an excellent raw match can still surface, and a fixed similarity temperature once the training target proved near-uniform.
Added zero-shot Fashion-CLIP slot and gender labelling as a hard eligibility mask at inference time, since real influencer items carry no metadata, and used real extracted attributes on the query side where ground truth exists.
Solved influencer portfolio selection two ways, side by side: a fast ratio-greedy submodular approximation and an exact 0/1 MILP, both scored against an LP-relaxation upper bound so quality is measured independently of which algorithm produced it, plus intra-list similarity, category entropy and coverage diagnostics.

Portfolio selection as a MILP

Decision variables

$x_{i} \in {0, 1}$ select candidate influencer i
$y_{ij} \in [0, 1]$ linearizes x_i AND x_j for each pair with a nonzero overlap penalty. Continuous, but forced to {0,1} at the optimum by the constraints below, since its cost coefficient is strictly negative.

Objective (maximize)

max i \sum x_{i} \cdot fit_{i}^{β} \cdot R_{i}^{2 - β} - (i, j) \in P \sum w_{ij} y_{ij} (R_{i}^{2 - β} + R_{j}^{2 - β})

$fit_{i}$ style-fit score for candidate i
$R_{i} = effective_reach (reach_{i}) = lo g (1 + reach_{i})$ log-compressed, not raw reach, since raw reach's multi-order-of-magnitude range was measured to swamp real fit differences
$β$ fit_bias. Default 1.0 gives plain fit_i · R_i; towards 2.0 it fades to pure fit, towards 0.0 to pure reach.
$w_{ij}$ assumed audience-overlap fraction between i and j (overlap_penalty), defined only for pairs with a real recorded value

Constraints

i \sum x_{i} \cdot cost_{i} \leq budget

i \sum x_{i} \leq max_count (if given)

i \in category c \sum x_{i} \leq max_per_category \forall c (if given)

y_{ij} \leq x_{i}, y_{ij} \leq x_{j}, y_{ij} \geq x_{i} + x_{j} - 1 \forall (i, j) \in P

Outcome

The compatibility model reaches 0.888 held-out AUC and 0.909 four-way FITB against random distractors, roughly at CSA-Net's published level, with a frozen visual backbone and about four times less training data. The larger result is catalogue-to-influencer matching, which moved from 6.7% recall@1, below the 9.9% random baseline, to 63.8% with the cross-attention adapter, with tight variance across seeds. Adding the hard gender mask lifts per-type recall@10 consistently across category, color and style, and makes gender-appropriate retrieval exact by construction.

Results in detail

Figures are from the delivered engagement, normalised where data is confidential.

Walkthrough: compatible recommendations, influencer matching, portfolio optimization

Compatible items are proposed for a product from the compatibility space learnt with a contrastive objective: from a partial outfit, items actually worn together are pulled in, while random items (easy negatives, phase 1) and items from the same required slot (hard negatives) are pushed away. The clip then shows selecting catalogue products, retrieving matching influencers, and optimizing an influencer portfolio under customizable diversity criteria and fit weights.

Animated 2D scatter plot of the compatibility embedding space, first separating catalogue products from influencer items, then colouring points by garment category — Embedding space: products vs. influencers, then by garment type
Animated 2D projection of the compatibility space. First products and influencer items are shown as two populations, then the view colours points by garment category (tops, shoes and the rest), making the slot structure visible.

Cross-domain retrieval: four alignment strategies vs. production

Recall@1 matching catalogue products to real influencer crops, 350 crops, 5 seeds. Ground truth is the detected slot. The cross-attention adapter wins clearly.

Per-type recall@10 with and without the hard gender mask

For each item, share of its 10 nearest catalogue neighbours that share its exact attribute value. Solid bars are without the mask, faint dashed bars with it; each attribute has its own colour. Category rises to about 0.59 , 0.60 for every adapter variant, color improves everywhere (mlp-learned 0.443 , 0.477), style improves for every learned variant, and gender reaches 1.000 by construction.

Compatibility AUC against published Polyvore results

AUC is the pool-size-independent metric, so these are directly comparable. Roughly at CSA-Net's level, behind OutfitTransformer SOTA, with a frozen backbone and far less data.

Portfolio selection: greedy vs. exact MILP

182 real candidates, budget 2000, at most 5 influencers, scored against an LP-relaxation upper bound. Greedy leaves more than half the achievable value unused and spends only 39% of the budget.

Related projects

Fintech / Credit risk