A Debate about Supervising Artificial Intelligence at Work

[00:00:00] Welcome to the debate. So, what if the massive productivity gains promised by generative AI are actually just an illusion? Right, like what if instead of freeing us from the grind, we've just secretly invented a brand new exhausting form of manual labor? Exactly. I mean, usually when you picture the automated workplace of the future, we imagine this assembly line of gleaming machines doing all the building. And the human workers are finally free to put down their tools, step back and focus on the big picture.

[00:00:29] Emancipation from the routine, right? But then you step into the modern knowledge economy and that factory floor looks, well, completely different. Yeah, it really does. The machines are building the car, sure, but the humans are frantically running behind them on the assembly line with magnifying glasses and tiny screwdrivers, desperately trying to catch every loose bolt and backward component before the car leaves the factory. Which is the exact opposite of stepping back. It is an incredibly intense, hidden form of continuous manual labor.

[00:00:57] And today we are establishing the reality of this exact phenomenon. It's a practice that has quietly consumed the modern knowledge worker, an emerging necessity known as botsitting. Botsitting, yeah. We're talking about the hidden, largely unmeasured human labor of providing context for supervising, reviewing and correcting generative AI output. Which brings us to the central disagreement for today's show. Right. The debate today centers on a fundamental question.

[00:01:23] Is structurally formalizing and budgeting for this botsitting the necessary path to capturing AI's true value? Or, does treating humans as permanent AI supervisors inevitably erode the very expertise and vigilance required to oversee these systems safely? We are grounding this discourse in recent research on human automation interaction, employment surveys, and documented organizational case studies regarding AI integration.

[00:01:48] And this material paints a very complex picture of what actually happens to human cognition and institutional capability when we ask human beings to essentially babysit algorithms all day. So my position today is that oversight is a permanent structural feature of human automation systems. It's something that must be formalized, risk tiered, and made completely visible in order to capture genuine net productivity. And I'll argue that shifting human roles from core creation to continuous oversight triggers the ironies of automation.

[00:02:17] It degrades human capability, institutionalizes automation bias, and creates dangerous organizational vulnerabilities that frankly no amount of formal budgeting can fix. Well, to set the foundation for my position, we have to recognize a fundamental truth about AI in the workplace. Its value is fundamentally a net figure, not a gross one. Okay, explain what you mean by that. So when leadership buys an enterprise AI license, they are sold the gross figure, right?

[00:02:43] They see the hours of drafting, summarizing, or coding that the machine will instantly eliminate. The promise of 11 hours saved. Exactly. But they are completely blind to the cost side of that exact same ledger. Let's look at the Glean Work AI Index data from 2026. This was a massive survey of 6,000 digital workers. Right, a very robust data set. Yeah, and respondents estimated that AI saves them about 11 hours a week,

[00:03:07] which sounds incredible on paper, but they also reported spending about 6.4 hours a week just making that AI usable. Wow. Over six hours. That is 6.4 hours of feeding it context, reviewing its output, and debugging its hallucinations. It's just a massive invisible tax on the workforce. That invisible tax is the core of the problem. If we rely on Susan Lee Starr and Ansem Strauss's classic sociological research on invisible work,

[00:03:33] we know that when labor goes unmeasured and unnamed in the organization, it leads directly to overburdened staff. And vastly overstated ROI. Yes. So the thesis I'm putting forward today is that the only sustainable path forward is deliberate organizational design. We have to measure the net value. We must include expected review time in our capacity models. And we have to explicitly name oversight and job descriptions. But does just naming it actually solve anything? It does. We actually have proof this works.

[00:04:02] Look at the Permanente Medical Group. When they successfully deployed Ambien AI scribes to listen to patient visits and draft clinical notes, they didn't just pretend the AI was perfect and pocket the time savings. They treated it like a rough draft. Exactly. They formally designed the workflow to treat the output as a rough draft. Physicians are scheduled and expected to review and edit that draft before it enters the official medical record. They built an architecture of oversight. Well, I come at it from a very different angle because I believe formalizing bot setting is a deeply flawed long-term strategy.

[00:04:32] I mean, it fundamentally ignores severe human psychological limitations. How so? We are biologically and cognitively terrible at passively monitoring highly reliable systems for rare errors. The human brain simply isn't wired for it. The research on automation bias and complacency is incredibly sobering here. You're talking about the complacency effect. Yeah. Consider the Dratch et al. study published in the journal Radiology. Researchers asked radiologists to read mammograms accompanied by AI-suggested categories. Right.

[00:05:00] When the AI suggestion was deliberately manipulated to be incorrect, the accuracy of even highly experienced radiologists plummeted to 45.5%. Wait, nearly a flip of a coin? Worse than a flip of a coin. For professionals, experienced specialists, people who have spent a decade looking at these scans, were pulled substantially toward the wrong answer simply because the machine suggested it. The anchoring effect of a computer suggestion is undeniably powerful. It's overwhelming. My thesis is that you cannot simply budget away the degradation of human vigilance.

[00:05:29] It's a biological constraint. But if we don't budget for it… Hang on. Let me just add this. By turning knowledge workers into full-time proofreaders of AI output, organizations are systematically stripping jobs of their core psychological rewards. Okay. The job characteristics argument? Right. Hackman and Oldham's job characteristics model. They found that human motivation and job satisfaction require three specific things. Using a variety of different skills, having autonomy in how you work, and task identity.

[00:05:58] Meaning you see a task through from a blank page to a finished product. And you're saying bot sitting removes those? Completely. When you turn a creator into a proofreader, you strip all three away. The work is deadened. Ultimately, converting creators into passive monitors hollows out the organization's foundational expertise. It leaves a shell of a workforce that's entirely dependent on the machine they are supposed to be supervising. Ah, yeah. I disagree entirely with that framing. Specifically because it ignores the immediate crisis workers are facing right now.

[00:06:27] I wouldn't call it ignoring the crisis. Well, anyone listening right now who has spent an hour trying to coax a chat bot into formatting a spreadsheet correctly or rewriting a generated email because the tone was bizarre, they know exactly what this invisible tax feels like. Sure, the friction is real. Right. If we accept that this oversight labor is happening, which the data overwhelmingly proves it is, the first step has to be accurately measuring productivity so organizations can plan effectively.

[00:06:53] If we don't formalize this, we are leaving workers to drown in unacknowledged labor. Just look at the SHRM 2026 data. It shows that directors and above are spending about six hours a week correcting AI output. That is a massive chunk of a senior leaders week that simply does not exist on any official spreadsheet. True, it's unmeasured. If we measure net value rather than gross time savings, we can accommodate this informal workload planning. We stop asking people to do two jobs in a time allotted for one.

[00:07:24] I understand the desire to relieve that pressure. I really do. But have you considered that by formally budgeting for this time, you are institutionalizing a corporate illusion? An illusion? How is measuring actual time spent an illusion? Because you are validating the premise that the AI is actually speeding us up when in many cases the net productivity gain is a complete mirage. We actually have randomized controlled data on this illusion of speed. You are referring to the open source developers study? Yes, the Becker et al.

[00:07:51] Study of experienced open source software developers. Researchers watched these developers use AI coding assistance. And the developers overwhelmingly believe that AI sped them up by 20%. Believed being the operative word there. Exactly. Because when the researchers actually measured the task completion times against a control group, the AI tools increased completion time by 19%. Which comes down to the hidden time spent prompting the machine, waiting for the output, and fixing the inevitable logical errors. Right.

[00:08:19] But we have to ask why the developers felt faster even when they were moving slower. Because generating text feels like work. It comes down to human psychology. Watching a machine generate lines of text or code triggers a sense of forward momentum. It feels like progress. Conversely, human thinking, staring at a screen, working out a complex problem in your head, feels like friction. It feels like a roadblock. That makes sense. So I have to push back on your framework here. Are we budgeting for actual productivity?

[00:08:45] Or are we just formally budgeting for the time it takes humans to clean up what researchers call work slop? Okay, that's a bit reductive. Wait, so your solution to AI generating massive volumes of garbage is to just like build an Excel column for garbage clean up time? That brings us to the real issue. Who takes the fall when the garbage inevitably slips through? Well, first of all, we aren't budgeting for the cleanup of garbage. We're budgeting for the navigation of what Fabrizio De Laquan's colleagues at Harvard Business School call the jagged technological frontier. The jagged frontier.

[00:09:15] Yes. AI performance is incredibly uneven, and it isn't always obvious where it will fail. On one task, it performs at an absolute genius level. On a seemingly identical task, just a millimeter outside that invisible frontier, it fails catastrophically. Like the BCG consultant experiment. Exactly. That proved this perfectly. Consultants using AI saw massive quality gains on tasks inside the frontier, but when given a task deliberately designed to be outsided, they did worse than those not using AI at all.

[00:09:45] Right, because they trusted the machine blindly. Because that frontier is invisible and constantly mifting. That's why organizations must adopt a risk-tiered approach to verification. But how does risk-tiering solve the psychological illusion of speed we just discussed? By explicitly acknowledging that not all AI output requires the same level of bot sitting, and mandating structural accountability where the stakes are highest, you tier the checking based on the consequence of an error. Okay, but mandating it how? For high-stakes outputs, we can and must mandate human accountability.

[00:10:15] Look at the legal system. Following the disastrous highly-publicized cases of lawyers submitting AI hallucinated case law in federal court, U.S. District Judge Brantley Starr in Texas issued an order. I remember this. Yeah, he required attorneys to formally certify that any AI-drafted language was checked by a human using traditional, authoritative legal databases. Similarly, the California Judicial Council adopted Rule 10.430, requiring court staff to take reasonable steps to verify AI output. So sign a piece of paper saying you checked it?

[00:10:44] It proves that human accountability can be successfully mandated for high-stakes outputs if we build the institutional scaffolding for it. See, that fluently written prose is exactly the trap. Mandating accountability on paper fails spectacularly in practice because reviewing AI-generated text is uniquely deceiving to the human brain. Deceiving how? When a human writes a flawed report, they're usually surface cues. We see awkward phrasing, logical leaps, grammatical errors, or, you know, an inconsistent tone.

[00:11:13] These act as friction detectors. They wake up our critical thinking. Sure, bad writing makes us read more carefully. But generative AI doesn't do that. It produces prose with total, unwavering fluency and supreme confidence. And cognitive science tells us that the human brain equates cognitive fluency with truth. It removes the traditional friction of poor writing that usually triggers our skepticism. Precisely. Because the AI writes perfectly, it bypasses our biological friction detectors, masking the factual voids beneath.

[00:11:42] So standard review processes, which were designed to catch human errors, are fundamentally mismatched for catching AI hallucinations. But process still matters. Let's look at the Deloitte Australia case. This was a massive, highly respected consulting firm, producing a welfare report for the Australian government worth roughly $440,000 Australian dollars. A very high-stakes document. And Deloitte has incredibly rigorous quality assurance processes. They have peer reviews, they have multi-stage editing, they have partner-level sign-offs.

[00:12:12] Accountability mandates on paper everywhere you look. Right. Yet the final published report contained completely fabricated references and an invented judicial quotation. It was only caught after publication by an outside academic. Which is embarrassing, yes. The human review process has failed because asking a human to painstakingly verify every single flawless-looking citation is a task human attention simply cannot sustain. But that is a failure of execution. It's not a failure of the concept of oversight itself. I'd argue it's both.

[00:12:40] The reviewers at Deloitte were likely doing standard read-throughs, looking for narrative flow and typos. They weren't conducting independent primary source verification of the citations. This is exactly why we need to formally define what oversight means for AI specifically. Which circles back to my point about institutionalizing the labor. You have to train people how to review this specific kind of output. But this brings us to the deeper systemic issue that training alone cannot fix.

[00:13:07] If standard review processes fail because human attention wanes when monitoring a usually reliable machine, we have to examine what this dynamic does to human expertise over a career lifecycle. Okay, let's look at that. This is where Lizanne Bainbridge's concept of the ironies of automation becomes absolutely critical. Back in 1983, she pointed out a terrifying paradox. By automating the routine parts of a job, we deny workers the exact practice they need to keep their skills sharp for the complex parts.

[00:13:33] The idea being that you can't be a good supervisor of a system if you've never actually done the foundational work the system is doing. Yes. And applied to knowledge work today, the irony is devastating. The expertise needed to catch a subtle, complex AI error is built by doing the routine, drafting, coding, and analyzing that AI has now taken over. But they're still working with the material. Are they? If we turn junior staff into prompt engineers and full-time bot setters, they never struggle through the blank page.

[00:14:01] They never build the foundational cognitive maps of their discipline. I think that's a bit nostalgic for the old way of working. It's like relying entirely on GPS to navigate a new city. You might get to your destination faster today, but after a year, if the GPS breaks, you don't even know which way is north. We are destroying our internal cognitive maps. Okay, but... How can a junior worker possibly develop the expert judgment required to catch those Deloitte-style hallucinations a decade from now if they've never built a report from scratch?

[00:14:28] Well, I'm not convinced by that line of reasoning because it assumes that expertise can only be built through repetitive manual creation from a blank page. It's how human learning works. I challenge the premise that starting from scratch is the only way to build a cognitive map. Instead of individuals endlessly struggling alone to learn or endlessly bot-sitting in isolation, organizations can build with systems engineers call quality upstream. Hold on. Quality upstream sounds like corporate jargon.

[00:14:56] If junior workers aren't drafting the documents themselves, how do they physically learn the job? Give me a concrete example of how this builds expertise. Okay, it looks like shifting oversight from an isolated, passive individual to a shared, proactive systemic process. Look at Morgan Stanley's Wealth Management Division. What did they do? Instead of just giving their financial advisors a chatbot and telling them to verify everything,

[00:15:19] which leads directly to the vigilance fatigue and cognitive fluency traps you mentioned, they use curated context and daily regression testing. Testing the AI itself. Right. They don't just hope the AI gets it right. They test the system against a vast array of real-world scenarios before deployment. They maintain a standing set of complex test questions, and they run a daily regression suite to check for compliance and accuracy. Okay, that's better than nothing, but… The expertise of the human is elevated.

[00:15:47] We shouldn't be training reviewers on basic task execution from scratch. We should be training them on specific AI failure modes. You build expertise by teaching workers how to proactively hunt for the machine's boundaries and break the system, not by forcing them to operate passively as if the machine doesn't exist. But elevating the human to this, like, high-level quality assurance role abstracts them away from the tangible work itself. Which brings up a critical question. What happens to the humans left working at the boundaries of these highly automated systems?

[00:16:17] They manage the system. Let's use Robert Karasek's job demand control model to break this down. Karasek found that psychological strain in the workplace is highest when workers face high demands, but have low control over how they meet those demands. Okay. Bot sitting creates the ultimate high-strain environment. You have endless volumes of fluent AI output hitting your desk like a fire hose, that is the high demand, and you are held personally accountable for its accuracy. Yet you lack the time and underlying verification tools to easily check it. That is the low control.

[00:16:46] You are describing a badly designed system, driven by cost cutting, not a technological inevitability. I am describing the reality of the profit motive. When you strip away core human roles to save money, you create unsustainable operational strain on the workers left behind to manage the itch cases. This isn't theoretical. Look at McDonald's recently ending its AI drive-through pilot because of severe order accuracy issues. Or look at Klarna, the fintech company. The one that replaced customer service. Yeah.

[00:17:12] They initially boasted about AI replacing hundreds of customer service workers, but later had to publicly state they are ensuring customers can still reach a human agent. They had to reinstate human boundaries because stripping away the human core destroys worker well-being and customer trust. Sure. Think about the people left behind in the Klarna model. They were likely overwhelmed by the bizarre edge cases the AI couldn't handle, doing nothing but frantically apologizing for machine errors they had no control over. Well, on that point, we actually have a strong area of agreement.

[00:17:39] Humans must remain at critical boundaries and they must be empowered, not just used as shock absorbers for bad code. I'm glad we agree on that. But that is precisely why formalization and regulation are vital, rather than just throwing our hands up and saying automation ruins jobs. Look at Article 14 of the European Union's AI Act. The human oversight requirement. Yes. It legally mandates that high-risk AI systems must be designed so that human overseers have real, structural authority to intervene, override, and reject the output. Authority on paper.

[00:18:07] Again, which we've established often fails to translate to reality when throughput metrics are the only things management actually measures. No actual architectural authority designed into the software interface itself. If you implement Article 14 properly, you give the reviewer the institutional time to reject output without having to justify the delay to a manager who is focused only on throughput. That assumes a very enlightened management. You design the interface to explicitly show the system's uncertainty and site-specific sources to break that cognitive fluency trap.

[00:18:37] You rotate monitoring duties so nobody is staring at a screen for eight hours straight. Right. When designed properly, oversight doesn't destroy well-being. It elevates the human to a high-judgment, empowered role where they are the ultimate arbiter of quality, not a passive victim of a machine's output. I would argue that only works as long as the metrics governing their performance actually reward them for catching errors, rather than punishing them for slowing down the raw output volume.

[00:19:02] I completely agree, which requires making the labor visible and budgeted, just as I've been arguing from the start. Fair enough. To summarize my position, bot sitting is an inevitable, permanent evolution of knowledge work. The jagged frontier of AI means a machine will always need a human guide to navigate its blind spots. A guide that needs a lot of time.

[00:19:23] By making this hidden labor visible, budgeting forward and capacity models and risk-tiering our verification processes, organizations can capture real, sustainable net gains from AI while protecting their workforce. It requires deliberate, thoughtful design, but the path forward is through formalizing the oversight, not avoiding it. And to summarize my perspective, while AI certainly has undeniable utility in specific contexts, formally converting our human experts into continuous supervisors is incredibly dangerous.

[00:19:53] Because of the cognitive maps. Right. It courts the ironies of automation by denying workers the hands-on practice they need to maintain their cognitive maps and expertise. It risks institutionalizing automation bias, where we formally budget for an illusion of speed, while missing catastrophic errors because human psychology simply cannot sustain vigilance over a highly fluent, highly reliable machine. It's a huge risk. We are slowly eroding the foundational craft required to keep these systems safe. And treating humans like software patches is a recipe for institutional collapse.

[00:20:22] I think where we absolutely converge today is the recognition that treating gross time savings as actual, realized productivity is a dangerous corporate illusion. Completely. Any leader taking the vendor's promise of 11 hours saved without deliberately subtracting the invisible labor of context-setting, reviewing and correcting is flying blind. And we also clearly agree that unmanaged AI deployment creates severe legal and reputational exposure.

[00:20:45] Whether it's a fabricated consulting report at Deloitte or hallucinated case law in a federal court, human accountability cannot be automated away. The responsibility ultimately rests with the human at the boundary. Because the court of law and the court of public opinion will never accept the chatbot did it as a valid defense. Precisely. As we reflect on the immense complexity of this jagged frontier of human-AI collaboration, we have to look closely at our own daily routines.

[00:21:13] It brings us back to that modern assembly line we discussed at the beginning. The one with the tiny screwdrivers. Yes. We have to ask ourselves, in our own work, as we run behind these incredibly fast machines with our magnifying glasses, are we truly mastering our automated tools? Or are we quietly being consumed by the hidden labor of bot-sitting? It's a question everyone needs to answer. The nature of our work is changing beneath our feet. There is so much more to explore in how we design the future of work.

[00:21:40] And it's a conversation every organization needs to be having right now. And it's a conversation when we are doing it. We are going to be able to prepare a new model. And it's a conversation every day.