Rearrange Objects in Scene without Changing Appearance

[Syzygianinfern0]

Is there some way i can use Controlnet to move an object from one location to another? Let's say a tabletop arrangement but I would like to rearrange the objects arrangements without changing the actual objects themselves.

One idea I had is that lets say I extract the segmentation map and then I use that information to rearrange some of the object locations in that map. And now I have a new segmentation map.

Can i feed it back into Controlnet and ask it to generate an image following the new segmentation map but with objects having the same appearance from the original image?

[blacklig]

how would you re arrange objects from segmentation map?

But would like to know generally how to provide already generated output of control net to it again.. like it generates canny map, I update it somehow, feed it back. Possible somehow?