ChatGPT vs. ChatGPT: A Redline Battle

In my negotiation practice, I have suspected that negotiators are becoming increasingly dependent on AI in the drafting, review, and redlining of agreements. This suspicion has been fueled not just by the prevalence and ease-of-access of AI tools, but by witnessing a change in negotiation dynamics: Seeing more and more non-substantive edits in redlines (that is, edits that might "clean up" a document but do not address anything of substance), slower movement towards compromise, and negotiators being more reluctant to have a conversation about sticking points.

So when a good friend recently reached out for negotiation assistance, I decided to put ChatGPT to a little test.

THE SITUATION

In my friend's initial email, he explained the deal background and attached a first draft of an agreement that was prepared for him by ChatGPT. In asking me to review and improve the draft, he added, "It's now ready for a professional!"

The ChatGPT draft was certainly a serviceable first pass, but also had glaring omissions.

After cleaning up the document and sending it back to my friend, I then launched my test.

THE DISCLAIMER

In full transparency, I am not an AI Doomer and do use ChatGPT in my practice. I find it to be helpful as a brainstorming partner, as an aid in the drafting process, and for creating graphics in situations where historically I would self-serve through an amateurish mix of stock images and PowerPoint.

But, I am also cautious in how I use it. I do not suspend my own expertise or critical analysis. I do not rely on it in places where outside expertise is needed.

THE TEST

Phase 1: Asking ChatGPT to Review Its First Draft

I started the test by de-identifying my client's draft. I removed:

• Any information that identified the involved parties;

• References to physical locations; and

• Compensation details.

All that was left was the terms and conditions that had been proposed by ChatGPT.

Then, I changed the naming convention of the file so that it did not bear resemblance to the naming convention used by ChatGPT.

Finally, I opened a new ChatGPT chat and:

• Attached the de-identified file;

• Wrote a prompt that described the transaction as it had been described to me;

• Noted that the attached file had been provided to our client; and

• Asked ChatGPT if it could provide a redline with edits to protect our client's interests.

ChatGPT issued a response that was highly critical of its own first draft and generated a new redline that significantly overhauled the document. In fact, of the 499 words in the new draft, only 129 (25.9%) remained from ChatGPT's original draft.

Phase 2: Asking ChatGPT to Review Its Second Draft

Taking the test a step farther, I made a clean copy of ChatGPT's second draft by accepting all of its redlines into the document and again changed the naming convention of the file.

I then used a different ChatGPT account and repeated the process. I opened a new chat and:

• Attached the second ChatGPT draft;

• Wrote a prompt that described the transaction as it had been originally described;

• Noted that the attached file had been provided to our client; and

• Asked ChatGPT if it could provide a redline with edits to protect our client's interests.

ChatGPT again issued a response that tore apart its own work and generated a substantial redline. In the 617 words of its third draft, ChatGPT only retained 353 from the second draft (57.2%).

Phase 3: ChatGPT's Critique of Using AI in Negotiations

I then decided to seek ChatGPT's assistance in dealing with a "colleague" that assists me in drafting and reviewing agreements. I noted that I was starting to suspect that this colleague was not paying close attention to details and was offering an abundance of edits to show effort without thinking about the practical impact on the negotiation. I added that I had tested my suspicions by twice sending my colleague his own work and, in both instances, received massive re-writes.

I finished by asking, "Do you think my colleague might be using AI to do this work?"

ChatGPT's response indicated that this fact pattern was very consistent with someone leaning heavily on AI and included the following self-criticism:

• "[AI] tends to rewrite broadly, even when unnecessary."

• "[AI] tends to optimize for difference, not continuity."

• "AI rarely performs 'light edits.'"

• "If someone is relying on AI as a shortcut, the result often looks like edits that don't map to negotiation strategy [and] [c]ritiques that miss practical realities."

ChatGPT's final point was "AI assisted drafting is not inherently a problem. AI replacing critical thinking is."

Oh, ChatGPT, I would high-five you if you had the arms for it. I feel a little bad for messing with you.

THE TOOL & THE TOOLKIT

Just like any tool in the negotiator's toolkit, it is important to understand the limitations of AI in the drafting and review process.

Fed with the right prompts, AI is a strong starter. But it is shit as a closer.

AI can be extremely helpful to organize thought and get a first draft into the world. It tracks against sage creative advice recently shared with me: "First make it exist and then make it good."

But, AI is currently not well-suited to reach negotiation nirvana: an agreement between two or more parties. In the creative services world, the whole point of the exercise is to drive the client and service provider to the point where they agree on mutually acceptable terms.

I've been doing this work a long (long) time and I can confidently say that even in the most complex and contentious transactions, thoughtful negotiation will almost always bring the parties to the point where they reach that agreement.

I can also confidently say that heavy, non-critical reliance on AI in the negotiation process often undermines the goal and results in prolonged, costly negotiations.