From “shadow AI” to structured use

The use of generative AI has largely grown from the bottom up, and statistics from Deloitte provide a picture of how.

According to their survey, 77 per cent of employees who use AI at work do so in models that have not been developed or procured by their employer. 52 per cent use publicly available free services, while 25 per cent pay out of pocket for AI tools they consider necessary to get the job done. Only 19 per cent say they use exclusively AI that the employer has developed or procured.

The consequences are already visible. In the Cisco 2024 Data Privacy Benchmark Study, 48 per cent of companies report incidents where confidential data has been uploaded to public AI services. 68 per cent worry that sensitive information will end up in the public domain or with competitors, and 27 per cent have therefore completely banned public AI services at work.

The data shows that the problem is not isolated mistakes, but recurring patterns of behaviour. Many organisations using generative AI exhibit high‑risk behaviours, and a large share of the prompts sent to AI models contain potentially sensitive information. The risk landscape is characterised by everyday situations where internal documents, source code, customer data or strategic material are shared with external services just to “try things out”.

An increasing number of companies are therefore trying to move usage from “shadow AI” to controlled, secure platforms in their own environment. We see this in the strong demand for help in establishing enterprise‑grade AI solutions close to the customer’s data, with access control, logging and encryption, instead of letting sensitive information leave the organisation’s control in public services.

AI as a trusted insider

The classic view of insider threats has focused on people with legitimate access who abuse it, intentionally or unintentionally. With generative AI, a new type of “trusted insider” is emerging

AI systems are often given broad, automated access to email, chats, documents, code repositories and business systems, sometimes broader than any individual human user, but without equally mature control mechanisms.

Generative models are also being integrated directly into collaboration tools, development environments and customer‑facing systems. AI agents can not only suggest actions but also carry them out. Data connections are set up quickly to realise business value, often before there is a full overview of which information is being exposed.

The result is that a misconfigured AI system can collect, summarise or forward large amounts of sensitive data in a very short time, while its activity in the logs can resemble legitimate operations. This shifts security work towards identity and access for both humans and machines, rather than focusing solely on user accounts.

In practice, this means that more organisations are now applying principles such as “least privilege” to AI agents as well, with separate service accounts, clearly delimited data sources and continuous review of access patterns. We see it as crucial to combine our expertise in AI development and cyber security in order to design such solutions as part of architecture and implementation, rather than adding them as a thin layer afterwards.

AI is strengthening the attackers’ toolkit

Generative AI is changing not only how organisations work, but also how attackers operate. 38 per cent of security leaders rank AI‑enabled ransomware as their biggest concern. 82 per cent of organisations believe that phishing emails have become harder to detect when generated using generative AI, and 60 per cent report that they have already experienced attacks where deepfake technology was used.

It is no longer about poorly spelled mass emails. With the help of generative AI, attackers can create tailored messages in the recipient’s language, imitate tone of voice and refer to relevant internal projects, especially when some information has already leaked. As more organisations open up their data to AI platforms, the value of a successful intrusion increases.

This is driving a need for more advanced detection that takes AI‑generated patterns into account. Solutions where machine learning is used on the defensive side to identify anomalous behaviour in both email flows and authentication patterns are becoming more common.

There is growing demand for combining generative AI with traditional security analytics to detect abnormal events faster, and to build automated incident response where AI support helps security teams analyse logs, correlate events and propose actions.

Model manipulation and data poisoning

As organisations move from merely consuming off‑the‑shelf AI services to training or fine‑tuning their own models, new risks arise deeper in the stack. One area is data poisoning, where training data is manipulated in order to steer the model in an undesirable direction – via open data sources or internal systems where an attacker gains influence over which data points end up as training material.

Another area concerns targeted prompt attacks and attempts to gradually extract a model’s knowledge or internal rules. Through carefully designed prompts, an outsider can try to gain access to information or behaviours that were never intended to be exposed.

At the same time, surveys show that 60 per cent of leaders and 41 per cent of staff admit that they feed AI tools with internal data, which increases the risk that models are trained on material that should never leave the organisation’s own environment. 59 per cent of Swedish IT security leaders explicitly see generative AI as one of the greatest security problems in the coming years.

To manage this, security needs to follow the entire model lifecycle. Instead of focusing solely on the user interface, organisations must work with governance of data sources, traceability in training pipelines, version control and protection against unauthorised access to models in production.

Governance, data and capability – three foundations

To reduce risks without losing pace in development, three main areas are emerging.

The first is governance: clear principles and practical guidelines for how generative AI may be used. This includes defining which types of data may never leave the organisation, which approved services exist, and how follow‑up and incident management around AI should work.

The second is the data foundation. Without an up‑to‑date data catalogue and effective classification, it is difficult to know what can be exposed to AI systems. The work ranges from identifying where information is stored to implementing classification models, encryption and segmentation in cloud and hybrid environments.

The third area is capability and culture. Generative AI is easy to adopt, which means both good and bad behaviours spread quickly. Security training therefore needs to be tied directly to employees’ daily work, with concrete examples of safe versus risky ways of using AI.

Towards a more controlled AI everyday

Today’s figures point to a risk landscape where high‑risk behaviours are common, policy frameworks are often absent or incomplete, and control over data is insufficient. At the same time, there is a shift under way where more organisations are moving from experiments and isolated initiatives to structured investments in secure platforms, better data governance and integrated security throughout the AI lifecycle.

In this work, technology partners with experience of both advanced AI development and safety‑critical environments play a central role. By combining architecture, data management, application development and cyber security, companies like HiQ can help organisations move from ad‑hoc usage to a controlled AI‑driven everyday reality, where productivity and innovation can grow without compromising security.

Curious about how your organisation can adopt AI safely and sustainably? Get in touch and let’s talk.