Following on from last week’s post about why AI rollouts don’t behave like software rollouts.

Every time we’ve moved house, I’ve looked at the packing companies. The main reason we’ve never used one isn’t just that they’re not cheap, but that is part of it. Generally I’ve also always talked myself out of it for perhaps a less obvious reason: packing the kitchen yourself may be slow and tedious, but it produces something useful that has nothing to do with boxes.
You end up knowing what you have. You find the thing you’ve been looking for since the last move. Through doing the packing process you make a hundred small decisions, such as keep, toss, that goes in the box we open first etc. And by doing this at the other end you know where things are, because you’re the one who put them there.

Pay someone and you get a beautifully packed kitchen. Labelled, wrapped, efficient. Almost certainly better than mine, and so much time and effort saved.

What you don’t get is the map.

I’ve been thinking about this today, while sitting on a long train ride across Japan during some quiet time on holiday. Pondering my own hesitation to dive into Copilot or Claude Cowork, and in particular how we’ve quietly crossed a line with AI at work and I’m not sure everyone’s noticed.

Drafting is not the same as doing

For the last couple of years, most workplace AI has been a very capable intern, writing drafts, summarising meetings and making suggestions. You then review, makes edits and decide whether to send it.

You’re the gate.

This is now changing with the tools now can open your files and change them. AI can now move things, delete things, and even send things. It can act across systems on your behalf while you’re doing something else entirely.

The intern has been handed a set of keys, and this is going to take time to get used to!

AI ‘having the keys’ is fine, provided somebody knows what’s in the house. Which brings me to the part most organisations would rather not look at, because we’re only human and tend to avoid hard things, or don’t have the time and resources.

Nobody packed these boxes

It is very common for organisation have permissions sprawl, and so many sites nobody owns. A folder from a 2019 restructure with the wrong access. A spreadsheet of salaries in a Teams channel set up for a project that ended.

This has been survivable for one reason: nobody could find any of it. The mess was protected by the difficulty of searching it.

Point a capable AI at that same environment and the protection evaporates. Not because the AI did anything wrong, because it’s respecting permissions exactly as configured. Which is part of the problem with the permissions were always wrong and we just couldn’t tell.

Now add the ability to change things, and the question stops being “what can people see” and becomes “what can something acting on their behalf modify, and would anyone notice?”. Eeeek.

This is the bit that gets skipped.
Nobody packed these boxes, because it’s different in that they accumulated, over years, through restructures and departures and projects that ended without anyone tidying up. And we’re about to hand over the keys to a house where we genuinely don’t know what’s in the back cupboards.

Nobody has worked out who is accountable

This is the one I’d want answered before anything gets switched on.

Something acts on your behalf. It gets it wrong. Who owns that?

Not in a blame sense but in a practical someone-has-to-fix-this sense:

• What can it do without asking?
  Read is different from write. Write is different from delete. Sending something externally is different again.
  
• Where’s the gate?
  Which actions need a human to say yes — and is that a real decision, or a dialogue box everyone clicks through by week two?
  
• Is there a record?
  If you can’t reconstruct what changed and why, you can’t fix it and you can’t learn from it.
  
• Who’s watching?
  In most organisations the honest answer is “the person it happened to, eventually, when they go looking for something.”

These are governance questions, not training questions. But I’d note, with feeling, that they tend to land on the training team anyway… usually around 4pm on the day something has gone sideways.

Checking has to be designed in now

There’s one more shift which really matters for anyone designing the enablement.

When AI drafts something, checking it is natural. It’s sitting there on your screen and you have to do something with it before it goes anywhere. The workflow makes you look.

When AI does something, there’s often nothing to check. The task is done and the file is updated. Whatever you asked for has been handled and through this process that natural moment of review simply isn’t there. Most likely it is the whole appeal that you didn’t have to be involved.

So verification stops being something the process prompts you into, and becomes something you have to deliberately build in: a step, a habit, a reason to look. If you think about this, it is not something really being taught. We’re teaching people how to get AI to do things, but we’re not teaching them how to notice when it has confidently done the wrong thing.

Over-trust is the risk that doesn’t look like a risk. It looks like adoption.

There’s a bigger question sitting underneath this one: what happens to our own judgement when the hard thinking gets handed over routinely, and whether the skill to spot a wrong answer survives not using it. That’s a whole post of its own, and it’s the one I want to work on next.

None of which means don’t

I want to be clear, because this can read as a list of reasons to stall, and stalling isn’t the answer either. The capability is real and so is the value.

It’s that “are we ready” is a broader conversation than a licence count and a security sign-off. It’s whether you know what’s actually in your environment and whether you’ve decided what an AI is allowed to do without asking. Whether you’d know afterwards, and whether checking is built into how people work, or left to chance.

Most organisations can answer maybe one of those, which is fine as long as you know which one.

I feel in some ways we are doing a similar cycle or phase to moving to the cloud years ago… for those organisations that did this. What was on file servers may have been copied and moved to the online libraries. Perhaps there was a big tidy up first because ‘lift and shift’. However in that phase, things stayed the same, and nothing behind the scenes had the ability to change without human review. The current innovations are so much more complex. 

While I don’t plan to move for a long time, I’ll probably pack the kitchen myself again next time. Not because I enjoy it, or because I am a super control freak (we don’t need that confirmed!), but because three weeks later, when I need the good measuring jug, I’ll know exactly which box it’s in.

For today, back to the train ride awaiting to see Mount Fuji out the window.