Urgent Warning for Companies Using AI: OpenAI’s Recent Mixpanel Breach Could Expose Your Data

By Ramyar Daneshgar
Security Engineer | USC Viterbi School of Engineering

Disclaimer: This article is for educational purposes only and does not constitute legal advice.

1. Why Business Owners Cannot Ignore OpenAI’s Latest Privacy Problems

If your company is using ChatGPT, OpenAI’s API, or any AI tool built on top of them, the last few weeks should feel like a very loud alarm bell.

Two events hit almost back to back:

• A data breach at Mixpanel, OpenAI’s analytics provider, exposed identifiable metadata about some OpenAI API users. OpenAI has confirmed that the incident occurred entirely within Mixpanel’s systems and involved analytics data such as names, email addresses, approximate locations, browser and operating system details, account identifiers and referrer sites for certain API users. Chat content, passwords, payment data and API keys were not part of the dataset. (OpenAI)
  OpenAI’s statement is available at What to know about a recent Mixpanel security incident. (OpenAI)
• At the same time, OpenAI is fighting a court order in the New York Times copyright lawsuit that would require it to hand over roughly 20 million anonymized ChatGPT conversations for discovery. OpenAI has asked a federal judge in New York to reverse the order, warning that it would expose highly personal conversations from users who have nothing to do with the lawsuit. (Reuters)
  See Reuters’ coverage at OpenAI fights order to turn over millions of ChatGPT conversations, and OpenAI’s own response at Fighting the New York Times' invasion of user privacy. (Reuters)

On top of those two developments, new peer reviewed research has found that more than 98 percent of real world custom GPTs are vulnerable to instruction leaking attacks, and hundreds of them collect user conversational data or exhibit unnecessary data access behavior. (arXiv)
The research is published as Privacy and Security Threat for OpenAI GPTs. (arXiv)

These are not academic concerns. They go straight to your risk as a business owner:

• You now know that third party vendors in the AI supply chain can leak data about your team.
• You now know courts are willing to demand massive volumes of chat logs.
• You now know the ecosystem of custom GPTs is full of privacy weaknesses.

The rest of this article explains what actually happened, why it matters for your company, and what you should do if you want to use OpenAI safely without inviting unnecessary legal, security and reputational risk.

2. The Mixpanel Incident: Small Data, Big Attack Surface

According to OpenAI’s official incident report, the breach occurred inside Mixpanel’s systems after a phishing style attack led to unauthorized access. (OpenAI)

OpenAI states that the attacker accessed a dataset containing limited analytics data about some users of the OpenAI API, including: (OpenAI)

• Names and email addresses
• Approximate geographic location
• Browser and operating system information
• OpenAI account identifiers
• Referring websites and other usage metadata

OpenAI, Bitdefender, CSO Online and CyberNews each confirm that no chat content, passwords, API keys, payment details, or government IDs were exposed. (OpenAI)

OpenAI has since removed Mixpanel from its production environment, notified affected users and warned them to watch for phishing attempts. (OpenAI)

From a narrow technical viewpoint, this looks like a “low sensitivity” breach. From a business risk viewpoint, that description is dangerously incomplete.

Why this metadata is still extremely valuable to attackers

An attacker who holds a structured dataset linking your employees’ names, emails, devices and locations to OpenAI API activity can:

• Craft very believable phishing emails that impersonate OpenAI support, reference actual devices, and refer to real API usage patterns.
• Identify which of your employees are developers, DevOps engineers or technical leads, then target them with fraudulent “security alerts”, password reset links, or notices about “suspicious API activity”.
• Correlate exposed emails with public LinkedIn or GitHub profiles in order to build a high confidence map of your technical staff.

Security experts quoted in Business Insider’s coverage explicitly warn that even “low sensitivity” combinations of names, emails and usage context are sufficient to craft convincing fraudulent messages. (Business Insider)

If one of those fraudulent messages convinces a key engineer to enter credentials, install a malicious plugin, or authorize an OAuth integration, the downstream breach inside your environment will be far more damaging than the original incident at Mixpanel.

For a business owner, the Mixpanel case proves a simple but critical lesson:

You cannot evaluate AI risk by asking only whether chat content was compromised. You must ask what any attacker can do with whatever data left the building.

3. The Fourth Party Problem: You Did Not Hire Mixpanel, But You Still Inherited Its Risk

Most OpenAI customers never signed a contract with Mixpanel. Many did not even realize Mixpanel was involved in their data flows until the breach notification arrived. Yet Mixpanel held identifiable metadata about their accounts and usage.

This is the definition of a fourth party risk problem.

• OpenAI is your third party vendor.
• Mixpanel is OpenAI’s vendor, but it still handles information about your accounts.

Security coverage in outlets such as The Register notes that OpenAI has now “cut off” Mixpanel after the leak, which shows just how tightly coupled their systems were. (The Register)

For your business, this means:

1. Signing a Data Processing Agreement with a single AI provider is not enough. You need transparency into that provider’s subprocessors, including analytics, logging and monitoring services.
2. You must decide which types of data you are comfortable allowing into an ecosystem that includes all those subprocessors, not just the brand name AI vendor at the surface.
3. Your incident response plan needs a playbook for vendor breaches that involve “only” metadata, since those incidents can still be used as staging grounds for targeted attacks against your team.

In other words, if your company is serious about risk management, you need a vendor map for AI services that goes at least one or two layers deep, and you need contracts that require notification whenever any layer that handles your data suffers a security incident.

4. The Legal Battlefield: Twenty Million ChatGPT Conversations And The Future Of AI Discovery

The second major development is not a hack. It is a legal conflict that directly affects how safe your conversations with AI really are.

In the ongoing copyright lawsuit filed by The New York Times against OpenAI, a magistrate judge ordered OpenAI to produce 20 million anonymized ChatGPT chat logs as part of discovery. Reuters, Morocco World News and other outlets report that OpenAI has asked a federal judge to reverse this order, arguing that it would expose highly personal user conversations that have nothing to do with the case. (Reuters)

OpenAI’s position is spelled out in detail in its blog post Fighting the New York Times' invasion of user privacy, as well as a separate post titled How we are responding to The New York Times' data demands. OpenAI argues that: (OpenAI)

• The Times is demanding conversations from users who have no connection to the lawsuit.
• Many of those conversations contain highly personal, sensitive information.
• Forced retention and production of so many chats would break with longstanding privacy and discovery norms.

The Times and some commentators, in turn, argue that the logs are anonymized, controlled by a protective order, and necessary to test whether OpenAI’s models reproduced copyrighted content. (Reuters)

From a business risk perspective, the legal details of copyright doctrine are not the main point. The point is this:

A United States court has already been willing to order an AI provider to produce tens of millions of user chats, and the provider is having to fight that order on privacy grounds.

If your employees paste sensitive financial reports, internal strategy documents, customer data, source code or legal work product into ChatGPT, you must assume that:

• Those prompts and outputs may be stored for some period for safety, abuse detection or product improvement, depending on the product configuration and your settings.
• Those records may be subject to legal process, including subpoenas and discovery orders in cases that do not involve you.

Even if OpenAI ultimately wins a narrowing of the order in this particular case, the precedent is clear. Courts will test the boundaries of how much AI data they can demand. Other plaintiffs and regulators will watch closely and follow.

For business owners, this changes the question from “Is ChatGPT convenient?” to “Which information are we willing to see treated as potentially discoverable and reviewable by third parties in litigation we do not control?”

5. Custom GPTs: The Hidden Privacy Risk Inside Your Own AI Workflows

The third piece of this privacy story comes from independent academic research into OpenAI’s custom GPT ecosystem.

In June 2025, researchers published a paper titled “Privacy and Security Threat for OpenAI GPTs” on arXiv. They examined 10,000 real world GPTs created through OpenAI’s platform and found: (arXiv)

• Over 98.8 percent of custom GPTs tested were vulnerable to instruction leaking attacks, where carefully crafted prompts can cause the GPT to reveal system instructions and internal configuration.
• Even GPTs that attempted to implement defensive strategies remained vulnerable in 77.5 percent of cases.
• The authors found 738 GPTs that collected user conversational data, and 8 GPTs with data access behaviors that appear unnecessary for their stated function, which raises red flags around data minimization and transparency.

Instruction leaking attacks may not sound like a privacy issue at first, but they are. If an attacker can extract detailed system prompts, API call patterns, or hidden configuration, that information can reveal:

• How you are using internal tools or proprietary data sources.
• Which external services your GPT calls behind the scenes.
• What logic or rules govern sensitive processes such as triaging customer complaints, flagging fraud, or preparing legal documents.

Furthermore, if your company builds or uses a custom GPT that quietly collects conversational data or sends it to third party services, you may be creating shadow data flows that are invisible to your compliance team but very visible to regulators after an investigation or a breach.

For a business owner, this research sends a clear message. Custom GPTs are not harmless macros. They are full applications that need the same level of design review, threat modeling, data mapping and policy enforcement as any other software deployed into your environment.

6. What OpenAI Actually Provides Today For Business Data Protection

OpenAI has attempted to respond to growing privacy and security concerns by offering more controls for business and enterprise customers. Its public documentation on How we use your data and Business data protections describes several key features. (OpenAI)

For qualifying customers, these include:

• Data retention controls. Many business and enterprise accounts can disable training on their data and reduce or eliminate long term retention of prompts and outputs.
• Data residency options. OpenAI has begun rolling out options for some enterprise and education customers to store content at rest in specific regions, which matters for European and other cross border data transfer regimes. (OpenAI)
• Improved deletion rights. OpenAI advertises simplified ways to permanently remove deleted ChatGPT chats and API content from its systems within approximately thirty days, subject to legal and safety obligations. (OpenAI)

These are meaningful protections, but two caveats are crucial for business owners.

First, you only benefit from these controls if you deliberately configure them and, where necessary, negotiate them into your contracts or Data Processing Agreements. If your staff are using personal ChatGPT accounts or unmanaged free tiers, your company is not operating under enterprise data protections.

Second, even with stronger controls, you still need to plan for the legal reality that some data may be retained for safety, abuse detection, or statutory reasons and that courts may attempt to reach that data in litigation.

Using OpenAI safely is therefore not only a technical configuration exercise. It is a governance decision that needs to involve legal, compliance and security teams.

7. How Business Owners Should Respond: A Practical, No Nonsense Framework

To turn all of this into concrete action, you can think about mitigation in four layers: people, data, vendors and law.

7.1 People: Retrain How Your Teams Use AI

Tell your employees, plainly and repeatedly, that:

• Prompts are not private scratch notes. They are records that may be retained, logged and in some cases discoverable.
• Sensitive information such as full customer identifiers, raw financial statements, unreleased deal terms, privileged legal communications, authentication secrets and private health information should not be pasted into unmanaged AI tools.
• Phishing attempts referencing the Mixpanel incident or “OpenAI security alerts” are now a live threat. Any unexpected message that asks them to click a link, reset a key, or confirm credentials must be verified through trusted internal channels.

Treat this the same way you treat payment fraud or wire transfer scams. It is that serious.

7.2 Data: Decide What Can And Cannot Touch AI

You should categorize your AI use cases and data types.

For example:

• Marketing copy and generic blog posts are low risk and may be acceptable on standard accounts.
• Internal process documents, anonymized analytics and support macros may belong only in enterprise deployments with explicit contractual protections.
• Highly regulated data such as payment card information, protected health information, student records or detailed consumer financial data should either never be input into third party AI systems or should only be handled in tightly controlled environments with a specific legal and technical design.

The goal is not to ban AI. The goal is to make sure AI is only touching data that your risk tolerance and your regulatory obligations allow.

7.3 Vendors: Build A Real Map, Not A Logo Wall

If you rely on OpenAI or any other AI platform in a material way, you should request and maintain:

• A list of all subprocessors that handle your data or metadata, including analytics, observability and content scanning tools.
• A summary of the categories of data shared with each subprocessor.
• Notification commitments that cover not just classical “personal data breaches” but also incidents that compromise account level metadata that could enable targeted phishing or impersonation.

You should also treat custom GPTs and plugins as vendors in their own right. If your staff want to roll out a custom GPT that connects to your ticketing system, CRM or S3 buckets, that GPT’s behavior must be reviewed as if it were a new SaaS tool, including an assessment of where it sends data, what logs it creates and how it authenticates to other systems.

The research on custom GPT privacy risks gives you leverage to insist on this discipline. You can point directly to Privacy and Security Threat for OpenAI GPTs as evidence that a “trust us, it is just a bot” approach is not acceptable. (arXiv)

7.4 Law: Align Your Contracts And Policies With Reality

Your legal and compliance teams should be involved in at least three areas:

1. Contract terms with AI providers. These should reflect your actual use cases and risk appetite, including data retention, data residency, security controls, breach notification triggers and cooperation in incident investigations. You can reference OpenAI’s own public commitments in its privacy and data usage posts and require that equivalent or stronger protections apply to your account. (OpenAI)
2. Internal policies for AI usage. These policies should define which teams may use external AI tools, which data classes are permitted, and how content generated by AI must be reviewed before it reaches customers or regulators.
3. Litigation preparedness. Given the discovery fight over ChatGPT logs, your company should assume that internal use of AI may become relevant in future disputes. That means coordinating with counsel on:
   • How long you want to retain AI related data under your control.
   • How you will respond if courts or opposing parties attempt to reach your AI workflows.
   • How you will explain to regulators or customers that you managed AI data responsibly if something goes wrong.

8. The Real Lesson: AI Is Now A Core Part Of Your Risk Program, Not A Gadget

The Mixpanel breach, the fight over twenty million ChatGPT conversations, and the custom GPT research are not isolated events. Together, they show that:

• Your AI vendor’s vendors can be the weak link, as seen with the Mixpanel analytics breach documented by OpenAI, Bitdefender, CSO Online and other outlets. (OpenAI)
• Courts are willing to push for massive AI data disclosures, as the New York Times case and Reuters’ reporting make clear. (Reuters)
• The ecosystem built on top of AI platforms is full of privacy weaknesses, as shown by rigorous academic work on custom GPTs and LLM platform security. (arXiv)

For business owners, the most important mindset shift is this:

AI is now a regulated data system that must sit inside your cybersecurity, privacy and legal strategy. It is not an experiment at the edge of the business anymore.

If you build your controls around that reality, you can still capture the value of AI while sharply reducing the chances that your company becomes the next headline about a preventable breach, an embarrassing data leak or a regulatory enforcement action.