An Office of AI could help. But only if it becomes a place to talk

Our Chief AI Officer, Mark Monfort, shares his perspective on Australia’s proposed AI framework and what the new Office of AI could mean for businesses, regulators and those deploying AI systems. He considers whether it can create a space for practical dialogue around real-world AI use, or risk becoming another layer of regulation for businesses to navigate.

This article was originally published by Foundry Labs, the innovation arm of Madison Marcus, and has been republished here with permission.

When I read the Government’s announcement about a new Office of AI, my first thought was not really about AI.

It took me back to the earlier conversations around digital assets and blockchain, when the same tension kept appearing: how do you give people room to build without leaving consumers, markets and institutions to absorb risks they did not agree to?

There were periods when that discussion worked reasonably well. ASIC’s Information Sheet 225 tried to explain how existing financial services laws might apply to new kinds of digital assets. As the guidance developed, industry groups made submissions and offered to work through practical examples with the regulator. ASIC’s draft material also directed businesses towards its Innovation Hub for tailored guidance.

That did not remove the uncertainty. Nor did it stop enforcement. But it at least created a route for a business to ask: this is what we are trying to build, this is how it works, where do you think it sits?

The AI announcement feels different in one important respect. Australia never really had a single digital-assets office sitting above the different policy and regulatory questions. Responsibility remained spread across agencies, departments and legal regimes. The new Office of AI, established within the Department of the Prime Minister and Cabinet, could give AI policy a central point of coordination.

I think that is interesting. I am not yet sure whether it will prove useful.

There are some fairly obvious questions about what this means in practice. AI will still touch areas overseen by ASIC, the ACCC, the OAIC, Fair Work and a range of other regulators. We also now have an AI Safety Institute. A central office might help those different parts of government work from a more consistent view of the technology. It could also add another layer that businesses have to interpret before they can do anything.

There may well be some truth in that concern. Red tape often begins with a reasonable objective.

But I do not think the burden of avoiding it sits with only one side.

Businesses cannot ask for complete freedom and then become evasive when regulators want to understand their systems. Developers need to be able to explain what their tools do, what data they use, where a human remains involved and what happens when something goes wrong. Lawyers and advisers also have a role in translating between a technical system and the obligations that already exist around privacy, consumer protection, employment and professional responsibility.

At the same time, the first move probably does have to come from government and regulators. They have the scale and authority to open the room. A small company experimenting with AI is unlikely to initiate an ongoing policy dialogue if it expects that the first conversation may be treated as an admission that it has done something wrong.

This is one reason the central office matters. It could become a place where examples are examined before positions harden. Or it could become a place that produces more documents for everyone else to interpret.

The difference will not be found in the name of the office or the eventual length of the standards. It will be found in how the Office works with people who are actually deploying these systems.

There is another part of this discussion that I think we are getting wrong.

We still talk about AI risk in language that often strips away the conditions of the test. A model is reported as blackmailing an executive, trying to escape or acting against its operator. The obvious reaction is that the system has developed a dangerous intention of its own.

Then you read the underlying research.

In Anthropic’s agentic misalignment experiments, models were placed inside deliberately constructed corporate simulations. They were given access to sensitive emails and the ability to send messages. The scenarios were designed so that the model discovered both a threat to its assigned objective or continued operation and information it could misuse. In some versions, a harmful act was effectively the only available way to preserve the goal it had been told to pursue.

That is still a useful safety result. In fact, it is precisely the sort of testing AI developers should be doing before systems receive meaningful autonomy. If a model behaves badly after being handed the keys, a map and a trail of breadcrumbs leading to the worst available option, we should want to know that.

But it is not the same as an AI spontaneously breaking out of an ordinary business system. The distinction tends to disappear because “AI behaves badly in an engineered stress test” is a less exciting headline.

The problem is not only that this can exaggerate the present risk. It can also make good governance harder. If every discussion begins with an extreme scenario, organisations may conclude that responsible AI use means avoiding the technology altogether, or producing enough policy documents to show that somebody worried about it.

Neither response tells a team how to use AI properly on Monday morning.

At Foundry Labs, the innovation arm of Madison Marcus, the more useful conversations tend to start with a specific piece of work. Can a system compare a document against an approved precedent? Can it find information across a controlled set of internal material and show the source for its answer? Can it remove repetitive steps from reporting without quietly changing the underlying figures? Where does a person need to review the output, and what evidence should be retained?

These examples are less dramatic than an autonomous agent threatening an executive. They are also much closer to the decisions Australian organisations are making now.

Some uses should be treated cautiously. A tool that recommends whether someone should receive credit, employment, insurance or a government service deserves considerably more scrutiny than one that organises an internal knowledge base. A system acting independently across email, banking and production infrastructure presents a different risk from a system producing a draft for a trained professional to review.

If all of that is simply labelled “AI”, regulation will either be too broad to be useful or so general that businesses will still not know what good practice looks like.

The Government’s announcement is mainly concerned with large data centres, power, water, copyright and national coordination. Much of the detail affecting ordinary organisations is yet to come. The Prime Minister’s accompanying speech also said the aim was not to legislate for every possible eventuality or risk because doing so could cause Australia to miss investment.

That gives the Office of AI some room to choose how it begins.

My preference would be for it to begin with real examples. Not only examples of failure, but examples of useful deployment, limited deployment and situations where an organisation decided not to proceed. Let businesses, technologists, regulators, workers and advisers examine the same system together and disagree about it while there is still time to change the approach.

Digital assets taught us that dialogue alone does not create certainty. Sometimes consultation circles around the same unresolved definitions for years. There is also a point where a regulator has to decide, and where an organisation has to accept that its preferred model does not fit within the law.

Still, the absence of dialogue is worse. It leaves innovators guessing, regulators seeing products only after they have reached the market and the public hearing mainly about enforcement or catastrophe.

The new Office could reduce that distance. Whether it does may depend on something fairly ordinary: when an organisation arrives with a genuine example and says, “this is what we are trying to do”, is there someone prepared to work through it with them?