Hi. Hmm… there seem to be quite a few related attempts, including work on similar tasks, but it doesn’t look straightforward. For now, it seems more sensible to separate the visual naturalness of object replacement from the other calculations it requires:
Short answers to your three questions
1. Do you need 3D, or can this be done in 2D? For the question “Can a particular SKU’s physical footprint fit in this part of the room?”, 2D geometry in measured floor coordinates can be enough. A complete 3D reconstruction is not mandatory. The crucial prerequisite is a defensible mapping to real units: a measured floor plan, adequate known dimensions, a calibrated measurement procedure, or other suitable metric evidence. A single ordinary RGB photo does not automatically provide that evidence. For overhangs, obstacles at different heights, or realistic perspective, add a small 3D representation where needed; that still need not mean reconstructing the whole room.
2. Can the fit be checked before the image replacement? Yes, provided the required geometry is known. Use the SKU’s actual width, depth, outer footprint, rotation, and relevant clearance, and compare it with the usable floor boundary and obstacles. This is a containment/collision problem, not an image-generation problem. Keep known free, known occupied, and unverified/occluded space distinct; an uncertain fit should not be reported as a proven fit.
3. What stack would I start with? If you already have measured room information: SKU dimensions + a metric floor polygon + Python/Shapely for the fit test. Add OpenCV’s planar geometry if you need to place that verified layout into the particular photo. A small 3D box/mesh and known camera geometry can provide perspective, silhouettes and occlusion guides. Keep FLUX/Qwen as the visual editing stage you already have working, rather than asking it to decide what 150 cm means.
I would keep two outputs separate:
- Physical-fit decision: the measured or assumed scene, the exact SKU envelope, the proposed position/rotation, clearance, and a result such as fits / does not fit / unresolved.
- Visual preview: a plausible image of the same approved pose, with reference design, lighting and shadows. It should be checked for geometry drift, but it is not the evidence that the object physically fits.
A low-cost default route
- Reuse any existing measurements. A dimensioned plan may answer floor fit without image reconstruction. With an unmeasured photo alone, label the result an approximate preview.
- Describe the physical target area. Separate measured boundaries, obstacles that remain, objects actually being removed, and unknown/hidden floor. Inpainting does not measure hidden space.
- Place the actual SKU envelope in meters. Width alone is insufficient; use depth, orientation and any required margin. Check it before image editing.
- Store the accepted physical pose outside the prompt. Keep
(X,Y,yaw,width,depth[,height])and measurement provenance fixed. - Register the photo separately. A floor homography maps floor points. A simple projected 3D proxy guides the raised sofa geometry.
- Let FLUX/Qwen render and then check. Compare silhouette, product identity and room preservation against the fixed guide.
For a toy test, if the hypothetical 150 cm sofa also had an assumed depth of 80 cm, a 1.50 m x 0.80 m rectangle could be rotated on a measured floor plan and checked immediately. The 80 cm and any sample margin are illustrative, not measurements from your actual SKU and not universal furniture-clearance recommendations.
Which branch applies?
Measured usable floor geometry available?
YES: check the real SKU envelope in 2D first.
Need a photo overlay? Calibrate/register the floor to the photo.
NO: Can a known dimension, plan, marker, or scan be added?
YES: establish scale and geometry, then check fit.
NO: offer an estimated preview; leave physical fit unresolved.
Does anything important happen above floor level?
YES: add height-aware obstacle slices or a simple 3D proxy.
NO: keep the physical placement test in 2D.
For either branch: keep the real dimensions and accepted pose fixed,
and verify what the image-editing step does to them.
The optional sections below explain why I would split those stages, what each approach assumes, and where existing tools/projects may help. They are alternatives to choose from, not a list of things you must install.
1. Defining an actual floor-fit check
Choose what “fits” means
A segmentation mask of visible floor pixels is not the same as a map of physically available floor. The old sofa may hide part of the floor; a shadow can look like an obstacle; a rug may cover rather than obstruct floor; a wall or radiator can project beyond the visible floor boundary. If the old object is genuinely going to be removed, its former footprint can be conditionally free, but the area hidden behind it may still require verification. A model-created empty-floor image cannot settle that question.
For an initial geometric representation I would use:
room_boundary: measured physical floor boundary, including relevant cut-outs/columns;remaining_obstacles: objects or fixed features that will still be there;unknown: hidden or unmeasured patches, rather than silently treating them as empty;sku_envelope: the item’s maximum relevant horizontal outline, not merely the feet that touch the floor;clearance: a stated policy for the particular product/use case, kept distinct from measurement uncertainty.
A product’s contact footprint, horizontal outer envelope, operational envelope (reclining/opening/walkway), and delivery/assembly envelope are different. For a preliminary static sofa fit, a manufacturer’s width and depth, or a conservative rectangle around the full outline, may be enough. A near-threshold, irregular shape needs better product geometry. Height and overhangs belong in a later check if they can interfere. Width alone, even when known exactly to be 150 cm, does not determine the footprint area.
Keep metric (X,Y) floor coordinates separate from pixel (u,v) image coordinates. Record conversion units, axis directions, yaw convention and each dimension’s source. A generic sofa mesh is not evidence of the real SKU’s width; verify world-space bounds after any import or transform.
The minimum geometric operation
Let F be a verified room-floor boundary, O known obstacles, U unknown space and P(X,Y,theta) the transformed SKU envelope. Test whether P with its required margin is covered by F and avoids O and U. Shapely’s covers permits boundary contact; buffer can enforce a stated margin. intersects counts touching as intersection, so the boundary-contact policy matters.
Here is a deliberately small geometry-only illustration. Its room boundary and obstacles are invented, not inferred from your image, and its 0.10 m clearance is a made-up scenario choice. The measured-room flag represents an independent input-quality decision; Shapely itself cannot establish that a floor survey was correct.
from shapely.geometry import box
from shapely import affinity
# Example dimensions in meters, not dimensions of a real SKU.
width, depth = 1.50, 0.80
x, y, yaw_deg = 2.0, 1.4, 15.0
clearance = 0.10
# Replace these example shapes with measured room data.
room = box(0.0, 0.0, 4.0, 3.0)
remaining_obstacles = box(3.25, 0.4, 3.65, 1.1)
unknown = box(0.1, 2.5, 0.9, 2.9)
room_boundary_measured = True
local = box(-width / 2, -depth / 2, width / 2, depth / 2)
turned = affinity.rotate(local, yaw_deg, origin=(0, 0))
footprint = affinity.translate(turned, xoff=x, yoff=y)
required = footprint.buffer(clearance)
if not room_boundary_measured:
verdict = "UNRESOLVED: boundary not measured"
elif not room.covers(required):
verdict = "NO FIT: exceeds verified room boundary"
elif required.intersects(remaining_obstacles):
verdict = "NO FIT: known obstacle/contact conflict"
elif required.intersects(unknown):
verdict = "UNRESOLVED: overlaps unverified region"
else:
verdict = "FITS under these dimensions and assumptions"
print(verdict)
Only assign definite NO FIT to a verified wall/obstacle conflict. A missing floor-segmentation label is not equivalent to a measured wall. Unobserved space should make the verdict uncertain rather than automatically negative; likewise, visually empty floor is not automatically verified free space.
Clearance, uncertainty, and candidate search
Separate physical-use clearance, measurement uncertainty and numerical precision. A buffer is a geometric margin, not a calibrated error distribution. Shapely’s minimum_clearance measures robustness of a geometry, not free distance from the sofa to the wall; use appropriate boundary distances instead.
If a borderline verdict changes when annotated corners or SKU dimensions move within plausible error bounds, report it as sensitive/uncertain, not as a measured fit probability. One better local measurement may resolve it.
To find a pose rather than check one, try a coarse grid of (X,Y,theta) and refine promising candidates. Restrict the search to the intended target region and allowed orientations; a mathematically valid pose somewhere else is not the requested replacement. Validate polygons and holes.
A helpful analogy is robotics: Nav2 costmaps distinguish obstacles, inflated clearance and unknown space. Furniture is not robot path planning, but the representation principle transfers well.
2. Establishing real scale and aligning a photograph
Treat measurement and image registration as different tasks
“Metric spatial mapping” can hide at least three questions:
- How large is the available physical floor region in meters?
- Which image pixels correspond to which positions on that floor?
- Where do the raised parts of the SKU project after it is placed there?
A dimensioned floor plan can answer (1) before a camera is involved. A good image-to-image alignment can answer something about (2) while leaving absolute scale unknown. A floor homography alone does not answer (3). This separation is useful because it localizes errors: if the top-down fit is right but the photographic overlay is wrong, there is no reason to redo the fit solver.
I would tag the source of the metric information, for example measured_plan, measured_floor_reference, sensor_scan, learned_estimate, or unknown. These do not have interchangeable error guarantees. The data can be used at different confidence levels without promoting an estimate into a surveyed fact.
If a measured floor plan exists
A measured plan is the easiest branch. Build the room and obstacle polygons in its coordinate frame; the fit decision can exist without a photo. A raster plan can be digitized if its scale is reliable. Check whether the drawing has been stretched, whether renovations changed wall geometry, and which present obstacles it may omit.
Only when you need the result to appear in the user’s photo must you solve the additional alignment problem. Keep the two outputs distinct: a geometry result with a measured provenance, and a preview with an image-registration quality.
If the metric floor and photograph must be aligned
On an approximately flat floor, planar homography relates metric (X,Y) positions to image pixels (u,v). The OpenCV homography tutorial and calibration tools cover this. At least four sound, non-collinear coplanar correspondences are needed in the basic case. Real metric world points, not merely image corners, provide the scale.
Known floor point: (X_m, Y_m, Z=0)
Corresponding pixel: (u_px, v_px)
Plane mapping: [u, v, 1] ~ H [X, Y, 1]
Important: this maps the chosen PLANE, not the entire 3D sofa.
Prefer additional correspondences and validate on held-out floor points/lengths. Four points fitted perfectly do not establish accuracy elsewhere. Reprojection error in pixels is not automatically a centimeter error, especially across a steeply foreshortened floor.
Lens distortion, crop, padding, undistortion and resize can invalidate an otherwise correct transform. Track coordinate changes between the original photo, projected guide and model input; otherwise the result may look like “AI scale drift” when it is really a pixel-frame mismatch.
If only one ordinary photo is available
Without an independent scale anchor and adequate camera/scene constraints, absolute floor dimensions are ambiguous. A plausible predicted room can be the wrong physical size. Focal-length calibration alone does not supply absolute scene scale. Multi-view SfM can also have a free global scale; see COLMAP issue #1653.
One known length can be valuable, but is not generally sufficient for a full floor-to-photo projective mapping without additional constraints. It matters whether the camera pose, floor geometry, marker orientation, or room layout is already known. There is no single perspective-wide “centimeters per pixel” conversion.
If an additional measurement is feasible, choose one that resolves the disputed fit: e.g., target alcove width, measured floor points, or a known-size board. OpenCV ArUco markers may help with suitable calibration and visible known dimensions. Do not impose extra scans when an existing plan already answers the question.
If a scan, depth or multiple views are available
Apple RoomPlan can provide structured room dimensions using supported camera/LiDAR devices; scan quality and occluded regions still matter. ARCore Depth can assist surface estimation. ARCore Instant Placement documents a transition from approximate to refined poses that can visibly change object size, so an initial preview is not a settled measurement.
Depth Pro estimates metric depth and can use supplied or inferred focal length. Depth Anything V2 and MapAnything are other geometry branches; MapAnything explicitly handles camera intrinsics, depth, and metric-scale input flags. However, “metric” model output is not independent measurement, and relevant scale concerns appear in Depth Anything V2 #152 and MapAnything #168. They are examples of failure classes, not proof of a problem with your pipeline.
Predicted depth and floor segmentation remain uncertain behind occlusions. For marginal fits, a small physical check can be more valuable than more inference on the same ambiguous photo.
Quick test: project a known-size floor grid into the original photo. If it drifts, debug scale/registration first; if it aligns but the sofa is wrong, debug projection or image conditioning.
3. When 2D, 2.5D, or a small 3D proxy is appropriate
There is no need to choose full room reconstruction up front
A 2D footprint can certify only a conditional floor-area result for its supplied geometry. It cannot detect a high shelf striking the backrest, a reclining mechanism’s moving envelope, or a doorway/access-route problem. Add the smallest height constraint that can distinguish those cases; a textured whole-room reconstruction is not the default prerequisite.
For common obstructions, use a layered “2.5D” proxy: a polygon and [z_min,z_max] interval for each relevant SKU/obstacle part. Potential interference occurs when horizontal shapes and height ranges conflict. This approximation is useful if its dimensions are known; near-contact diagonal/irregular shapes may need a detailed mesh.
Example: one sofa part occupies (polygon P, height 0.4..0.9 m).
A window sill occupies (polygon S, height 0.7..0.85 m).
If P and S intersect in the horizontal plane and those
height intervals overlap, the proxy indicates interference.
This conclusion holds only for the supplied proxy dimensions.
The maximum top-down envelope is conservative but can reject placements where an overhang clears a lower obstacle. Checking just the legs can falsely accept a collision. Choose the representation for the intended risk level.
Use a 3D proxy for perspective even if fit remains 2D
This is a separate reason to have some 3D. A homography projects floor points, not points on the sofa’s raised arms and back. A box or simplified manufacturer mesh at the accepted (X,Y,Z,yaw) and known dimensions, projected through an estimated/known camera, produces a reasonable target silhouette, projected corners and contact location. This proxy can guide the visual editor even if the entire room was never reconstructed. It also provides a reference to measure whether the image model enlarged or shifted the sofa.
Potential guide layers include the projected outline, a depth/occlusion estimate, a coarse semantic image, contact area and perhaps normal/wireframe cues where a particular editor supports them. A binary mask gives a region, not complete perspective or hidden SKU geometry. A reference photo gives design details, not a complete calibrated mesh. The strongest workflow may combine a few complementary controls rather than expecting one model to infer every missing constraint.
SpatialGen converts structured layout/camera data into visual guidance. 3D-Copy-Paste also places before composing, but its published code samples size changes and approximate collisions; neither is a drop-in certified real-SKU checker for an unmeasured photo.
If there is a manufacturer 3D model
A manufacturer mesh helps only if the exact SKU, units, bounds, pivot, transforms and floor contact are checked. The glTF specification uses meters, but importing a generic or incorrectly scaled mesh into glTF does not validate its real dimensions.
If the actual use case includes getting the sofa through a doorway or up a staircase, that is another problem: a sequence of poses and a swept volume, not the static destination fit. Likewise, installation clearances may differ from aesthetic spacing. Those are reasonable future extensions, but need not block a first demonstration of floor-footprint verification.
Useful stopping points: (a) trusted 2D fit alone; (b) 2D fit plus projected proxy and visual preview; (c) detailed 3D collision checking only where an elevated obstruction warrants it.
4. Keeping the FLUX/Qwen visual swap while constraining geometry
I would preserve the existing visual replacement workflow, since you report that FLUX and Qwen are working well for that part. Its responsibility can be identity, texture, lighting, local occlusion and shadows, while a geometry layer owns metric product size and accepted pose. The interface need not be complicated: store a compact placement record, and treat every render as an attempt to visualize that record.
{
"room_scale_source": "measured_plan",
"sku_id": "example_only",
"sku_dimensions_m": {"width": 1.50, "depth": 0.80, "height": null},
"dimensions_note": "depth is hypothetical; verify actual SKU",
"pose": {"x_m": 2.0, "y_m": 1.4, "yaw_deg": 15.0},
"clearance_m": 0.10,
"fit_result": "conditional_fit",
"projection_status": "requires_camera_registration"
}
That is just an illustrative contract, not a proposed universal schema. In a real system add provenance, uncertainty, which objects will remain, and the exact image coordinate transforms. The critical property is that the physical dimensions do not change merely because the generated silhouette looks better.
For a first comparison, generate (A) a deterministic overlay of a projected box or SKU mesh, without diffusion, and (B) the FLUX/Qwen edit conditioned on that same approved location. If A is too large or has an impossible perspective, investigate the metric placement/camera/proxy. If A is right but B is oversized, investigate how the editor uses masks, reference images, conditioning strength and crops. If both align in the photo but the room measurement contradicts fit, return to the room/SKU input instead of tuning a prompt. A third version using a direct textured 3D composite plus only localized AI cleanup can be a useful lower-drift baseline when exact proportions matter.
Object removal and object insertion need different masks. Removing the old sofa may expose background that has to be inpainted; inserting the new sofa requires a new silhouette and appropriate occlusion. Reusing the old object’s mask as the new object’s size can reproduce the wrong geometry. Inpainted floor texture is not a measurement of previously occluded space. If a specific SKU is important, verify its arm shapes, proportions, feet and other distinctive design elements independently of the overall image quality. A general reference-photo editor can produce a convincing similar product that is not exactly the SKU.
One useful quality split is:
- Placement: Does the final silhouette remain consistent with the projected accepted pose?
- Identity: Does the image still depict the intended SKU rather than a redesigned generic sofa?
- Scene preservation: Apart from necessary removal/occlusion/shadows, are walls, doors and nearby furniture unchanged?
Mask coordinates need particular attention if the editor resizes or crops images internally. Keep one traceable transform between original photo pixels, model input pixels and projected physical coordinates. If a result looks wrong only after outpainting or final compositing, compare it against the pre-postprocessing version before blaming camera calibration.
AnyDoor and this HF Forum reference-object discussion concern preserving reference appearance, a separate issue from metric fit. The FIT fit-aware virtual try-on work illustrates the same conceptual split, although deformable clothing methods do not directly solve rigid furniture geometry.
5. A small test that separates the possible failures
A large new model benchmark would not be my first step. Start with a known, synthetic metric room and one fixed SKU envelope. Verify easy FIT, NO FIT and UNRESOLVED cases, plus a rotated object, a wall contact and a margin-sensitive pose. Then use one small real example with independently checked measurements, if available. This establishes whether the geometric core is correct without involving image synthesis.
The next inexpensive controls can be staged:
- Top-down only: measured/synthetic floor polygon, obstacle and SKU envelope. Confirm acceptance, rejection, and unknown handling with the same coordinates.
- Image registration: project several known metric floor points/grid lines into the original photo. Check additional observed points not used to estimate the transform; record both pixel error and the physical dimensions used.
- Simple 3D proxy: project the same approved SKU box to check height, camera pose and visible contour. This is especially useful when the floor contact is correct but the sofa back looks wrong.
- No-AI composite: overlay the proxy on the room before passing it to an image editor. This exposes mask or depth-order mistakes separately from generative changes.
- Visual edit: use the approved pose and reference SKU with FLUX/Qwen. Compare its silhouette and distinctive product features with the fixed guide, ideally across a few seeds.
- Negative controls: change the SKU from 150 cm to a larger width without changing the room, or add a known obstacle while holding the SKU fixed. The fit decision should change where the geometry dictates; a purely aesthetic answer might not.
For a borderline room, perturb uncertain annotations or dimensions over plausible ranges and observe whether the verdict flips. That is a sensitivity analysis, not a calibrated probability of correct installation. An independently measured held-out length, or a small real test with a tape measure, is more informative than repeatedly grading an image against the same mask used to generate it. If the exact threshold matters commercially, distinguish false acceptance (says the item fits but it does not) from false rejection and an honest unresolved result.
A compact debugging record can include: room scale source; SKU dimensions/source; which objects are removed; wall/obstacle/unknown polygons; candidate world pose and margin; an XY plot; any floor-image correspondences and residuals; projected proxy; final image; library/model versions; image resizing and crop policy. A synthetic or privacy-safe example is sufficient for discussing algorithms on a public forum.
I would call a first prototype successful if it can make the same fixed geometric decision without generating an image, then render that pose without uncontrolled rescaling. That would already be meaningful progress beyond “please draw this sofa at 150 cm”. It would not yet certify all real rooms or guarantee installation; that requires real measurements, tests near fit thresholds, and potentially inspection of hidden obstacles and access routes.
6. Sources and implementations worth borrowing from
These are examples of different parts of the workflow, not evidence that one existing open model already solves photo-only, exact-SKU metric fit end to end.
| Need | Resources | What to borrow / important limit |
|---|---|---|
| Measured 2D fit | Shapely manual, covers, buffer |
Polygon geometry and margins; input measurements are still your responsibility. |
| Floor-to-image mapping | OpenCV homography, calibration tools | Coplanar registration and reprojection tests; not above-floor projection or absolute scale from unanchored pixels. |
| Device-assisted measurements | Apple RoomPlan, ARCore Depth, ARCore Instant Placement | Room/camera sensing on compatible devices; scans and temporary tracking estimates still need quality checks. |
| Learned geometry estimates | Depth Pro, MapAnything, COLMAP | Useful geometric estimates and multi-view pipelines; metric labels and good-looking reconstructions are not substitutes for measured scale. |
| Candidate furniture layout | ATISS | Learned room-layout proposals, not SKU-specific collision certification. |
| Geometry-guided image synthesis | SpatialGen, 3D-Copy-Paste, Build-A-Scene | Pose/layout before appearance. Some implementations rescale or approximate objects, so audit assumptions before reusing code. |
| Reference-specific appearance | AnyDoor, HF reference replacement thread | Preserving reference identity is a separate task from physical fit. |
IKEA Kreativ is a relevant commercial staging example, but it does not establish a reproducible, photo-only centimeter-accuracy guarantee for this workflow.
The most relevant general lesson from these projects is an architectural boundary: geometry can decide and preserve a candidate placement; a visual model can make that candidate look natural. Selecting only the pieces supported by the actual input data is more practical than installing every system above.
Bottom line
I would first test a measured floor polygon plus the actual SKU envelope, with a stated clearance and explicit unknown regions. If that passes, map the pose into the picture and let the FLUX/Qwen workflow render it, using a simple 3D proxy for the sofa’s above-floor perspective as needed. If there is no metric reference at all, a visual approximation can still be worthwhile, but the real-size fit should remain unverified. If you already have a floor plan, some measured reference lengths, camera information or a scan, any of those can determine the shortest branch without requiring a full 3D reconstruction from the start.