Lightnews — Scholar-powered news

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵10/10 Lastly, huge thanks to my co-advisors Niloy and Duygu!
For more details check out our paper below-

🌐 Project Website: monetgpt.github.io
📄 Arxiv: arxiv.org/abs/2505.06176

MonetGPT

monetgpt.github.io

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵9/10 We quantitaively evaluate on the Adobe5k dataset as well as conduct user studies by expert and novice users. Our evaluations show that MonetGPT outperforms open-source alternatives and performs comparably to Google Photos AutoEnhance (closed-source).

1 2

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵8/10 Photo editing is subjective 🎨. Our framework adapts to user preference by guidance from natural language tags like ‘vibrant’ or ‘retro vibe’ to produce personalized and stylistically distinct retouching plans from the same input image.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵7/10 Our puzzle-based training with a 'reasoning as a pathway' approach allows MonetGPT to generate detailed justifications for each edit, delivering truly explainable image retouching

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵6/10 🧩 Puzzle C builds planning capabilities. The model learns to generate a complete, multi-step retouching plan to enhance a photo, structuring its reasoning as a sequence of discrete issues and solutions for clarity and control.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵5/10 🧩 Puzzle B imparts aesthetic judgement. By ranking professionally edited photos against altered versions, the MLLM learns to recognize the visual characteristics of an optimally adjusted image for any given operation, building an internal aesthetic model.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵4/10 🧩 Puzzle A builds an understanding of individual operations. The MLLM learns to map visual changes in before/after images to a specific tool and its precise parameter value, effectively learning the semantics of our procedural library.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵3/10 Our key recipe: MLLMs struggle to predict edit values directly. We solve this by generating rich textual reasoning for each puzzle ✍️. We then fine-tune MonetGPT on this data, creating a 'reasoning pathway' that enables it to regress final adjustment values.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵2/10 MLLMs lack the visual understanding to plan edits. 🧠 So, we use expert photos as our ground truth and work backward, procedurally creating puzzles by assuming any change to an expert edit makes it less optimal

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵1/10 Excited to share our #SIGGRAPH paper "MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skills" 🌟
We explore how to make MLLMs operation-aware by solving visual puzzles and propose a procedural framework for image retouching
#MLLM

1 2 4

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵10/10 Lastly, huge thanks to my co-advisors Niloy and Duygu!
For more details check out our paper below-

🌐 Project Website: monetgpt.github.io
📄 Arxiv: arxiv.org/abs/2505.06176

MonetGPT

monetgpt.github.io

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵9/10 We quantitaively evaluate on the Adobe5k dataset as well as conduct user studies by expert and novice users. Our evaluations show that MonetGPT outperforms open-source alternatives and performs comparably to Google Photos AutoEnhance (closed-source).

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵8/10 Photo editing is subjective 🎨. Our framework adapts to user preference by guidance from natural language tags like ‘vibrant’ or ‘retro vibe’ to produce personalized and stylistically distinct retouching plans from the same input image.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵7/10 Our puzzle-based training with a 'reasoning as a pathway' approach allows MonetGPT to generate detailed justifications for each edit, delivering truly explainable image retouching

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵6/10 🧩 Puzzle C builds planning capabilities. The model learns to generate a complete, multi-step retouching plan to enhance a photo, structuring its reasoning as a sequence of discrete issues and solutions for clarity and control.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵5/10 🧩 Puzzle B imparts aesthetic judgement. By ranking professionally edited photos against altered versions, the MLLM learns to recognize the visual characteristics of an optimally adjusted image for any given operation, building an internal aesthetic model.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵4/10 🧩 Puzzle A builds an understanding of individual operations. The MLLM learns to map visual changes in before/after images to a specific tool and its precise parameter value, effectively learning the semantics of our procedural library.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵3/10 Our key recipe: MLLMs struggle to predict edit values directly. We solve this by generating rich textual reasoning for each puzzle ✍️. We then fine-tune MonetGPT on this data, creating a 'reasoning pathway' that enables it to regress final adjustment values.

1

Niladri Shekhar Dutt @niladridutt.bsky.social · May 27

🧵2/10 MLLMs lack the visual understanding to plan edits. 🧠 So, we use expert photos as our ground truth and work backward, procedurally creating puzzles by assuming any change to an expert edit makes it less optimal

1

Niladri Shekhar Dutt @niladridutt.bsky.social · Apr 10

Hi Londoners

Join us on April 15 for an evening on Gen AI for 3D at UCL! We have an amazing list of keynote speakers and lightning talks. Register at londongenai.github.io

Very excited to co-organize this with Michael and Preddy!

3

Niladri Shekhar Dutt @niladridutt.bsky.social · Feb 27

Amazon came pretty late to India and we already had some homegrown companies like Flipkart which now competes with Amazon and is valued at $40B.
I think big tech's early access killed homegrown companies. China and to an extent South Korea (Naver) has some great tech companies because of barriers

1 2

Niladri Shekhar Dutt @niladridutt.bsky.social · Feb 7

Who will tell the silicon valley tech bros that it wasn't them alone

4

Niladri Shekhar Dutt @niladridutt.bsky.social · Jan 24

Haha exactly what I did today!

1

Reposted by Niladri Shekhar Dutt

Ruchira 🏳️‍🌈 @rayruchira.bsky.social · Jan 17

🤔What tasks do we want robots to handle? Are these preferences based on saved time or feelings we associate with the tasks?

Introducing Why Automate This?—a study exploring automation preferences across social groups, using feelings & time-spent as key factors. 👇 (1/5)

The image illustrates the evolution of cleaning tasks, balancing time (ATUS) and well-being (ATUS-WB). A central figure considers two options: Manual Labor (vacuum cleaner) and Automated Labor (robotic vacuum), connected by an orange arrow labeled B1-K.

2 4 15

Niladri Shekhar Dutt @niladridutt.bsky.social · Dec 2

Have a great time in Seattle!

1