Authors

Angus Lockhart
Senior Policy Analyst

Christelle Tessono
Policy and Research Assistant

Contributors

• Nina Rafeek Dow
  Zaynab Choudhry
  Suzanne Bowness

Funder

This work is funded by The Canadian Digital Media Research Network

Tags

• AI
• Innovation + Tech
• Surveillance and Privacy

Share

Background

Generative AI is defined as the set of AI systems that “create new content in response to prompts, based on their training data”.1 This content can include , but not be limited to, text, video, audio, images, and code, for example. Although OpenAI’s ChatGPT raised the technology’s public profile when it launched in 2022, generative AI tools are not a recent development. Instead, they stem from decades of research and innovation at the intersections of computer science, linguistics, and statistics.2 The first types of generative technologies were language models. 

Developed in the early 1950s, language models consisted of rule-based systems that relied on linguistic rules to process language. In the following decades, statistical models were developed using probabilistic methods that determined the likelihood of a word sequence in a given context (e.g., determining the likelihood that the word “rainy” could follow the word “day”).3 In the early 2000s, advancements in computing infrastructure allowed researchers to develop systems that could analyze large sets of various types of data. By the mid-2010s, decades of foundational theoretical research, combined with powerful computing infrastructure, allowed the creation of more sophisticated computational techniques to analyze significantly large data sets. The growing accessibility of consumer electronics (e.g., phones, laptops, and tablets), increased quality of internet connection, the digitization of many aspects of our professional lives, and the rise of social media platforms. These factors have all converged to allow the creation of generative AI tools like ChatGPT, Perplexity AI, and Midjourney.     

The arrival of generative AI coincides with an information ecosystem that is dominated by a small number of powerful social media platforms. Information about current events domestically and internationally is delivered through algorithmically-driven methods. These algorithms rapidly share information and prioritize user engagement to drive ad revenue. This has led to misinformation (the unknowing spread of false information) and disinformation (the intentional spread of false information to serve political objectives, cause harm, generate revenue) to undermine the integrity of information ecosystems.4 Moreover, extremist content has proliferated on social media as well, which has created unhealthy digital environments. GenAI tools make it easier for users to develop their own content, which opens the door to malicious uses that can further spread misinformation. 

Considering these issues, various jurisdictions are grappling with how to develop the necessary regulatory oversight mechanisms to ensure that social media platforms are held accountable. In the following section, we outline the scholarship dedicated to understanding the integrity of information circulating on social media platforms, and discuss how information is created, disseminated, perceived, and subsequently moderated in the context of generative AI.

Using Generative AI to create content

With regards to content creation, there are various computational techniques that can be adopted to generate text, images, audio, and videos. These include simple editing software such as Photoshop, meme generators,5 and now genAI tools like tools like Dall-E and Stable Diffusion. In light of the popularity of genAI, traditional editing software companies have now added AI capabilities to their products (e.g. Adobe).

While editing images and other media content is not a recent phenomenon, genAI tools increase the average user’s ability to generate original (fake) content, or modify existing content, due to the ease of use, accessibility and ability of these tools to create realistic content easily, quickly, and to a high level of realism. According to Sensity AI, a company that tracks malicious use of generative AI through deepfake detection software, by 2024 there were nearly 3,000 dedicated tools for face swap, lip sync, face reenactment and AI avatars available. Additionally, there are over 10,200 tools for image generation and 11,018 for voice generation and cloning.6

Disseminating generative AI content online

Scholars have studied how information appears on social media feeds. In recent years, particular attention has been paid to how automated systems are used by platforms to create engaging feeds for users. For instance, researchers have identified that “recommender systems” are used by platforms like Facebook, X, and TikTok, to match content with user preferences and help users discover new interests.7 The use of recommender systems has been found to influence user behaviour to optimize user engagement.8 They have garnered a lot of scholarly attention as researchers have also investigated the harms that emerge from them. For instance, research has demonstrated that recommender systems can exacerbate the spread of false news online,lead users to extremist content, enable cyberbullying, and promote toxic content that triggers mental health issues.9 As a result of these harms, scholars and technologists have worked towards building technical interventions that centralize harm reduction and fairness in the design features of recommender systems.10 

Factors contributing to user trust of content on social media platforms

Researchers have focused on understanding the factors that build a sense of trust in information among social media users. Notably, researchers have found that how information is presented to viewers influences their perception of its accuracy. For example, Ahmed (2021) conducted a US-based survey and found that captioning fake videos on Instagram decreased the proportion of participants who perceived the content to be accurate by 18 percent and reduced their likelihood of sharing the content by 15 percent.11 Yet, to date, there have been few studies attempting to understand how the public perceives AI-generated information.12 

There is also a growing body of scholarship analyzing the impact of media “provenance” — a term for the creator or source originating the content—on user perception of trust and accuracy. Notably, Feng et al. (2023), presented a mock Twitter-like feed to 595 participants based in the United States and United Kingdom. Participants were asked to rate the trust and accuracy of the content they saw on the feed containing a clickable pop-up box with various types of details about the source of the information. They found that providing source information through the pop-up boxes lowered the participant’s trust in deceptive, or fake, content shown to them.13

Content moderation practices on social media platforms

To address issues of misinformation and disinformation, social media platforms began to establish trust and safety teams that developed a wide range of content moderation policies and practices. As part of their arsenal, platforms moderated content manually through data workers and through automated methods powered by algorithmic systems.14 Data workers will review content and flag that it as harmful or as going against the platform’s guidelines. Moreover, this manual moderation is also used to train algorithmic systems to identify and subsequently take down content. However, research has demonstrated that platforms do not moderate content effectively.15 For instance, in a study that assessed the efficacy of Facebook, YouTube, and TikTok policies for political advertising, the authors found that platforms unevenly applied their policies across their respective platforms.16

In recent years, platforms have adopted a wide range of technical tools in an attempt to create healthier information ecosystems. These include, but are not limited to, labelling posts (e.g. using tags, identifying that it is sponsored/monetized content, political, etc.), inserting community notes, and adding descriptions, and/or account verification for public figures. These tools are meant to help a user decide whether to trust the source of information, or identify misleading information.

While the creation of trust and safety teams was a promising start in the effort to moderate content, platforms such as X (formerly Twitter) and Meta have significantly scaled back these initiatives. 17 With the increased use and popularity of content generated with AI systems on social media, and the reduction in content moderation, malicious actors may see this as an opportunity to increase misinformation and disinformation. 

Effectiveness of labelling techniques

Researchers have sought to assess the efficacy of labelling techniques. For instance, a study by Lewis et al. (2022) found an increase in users’ ability to detect deepfakes when labels were provided. However, despite detection moving from 11 percent to 22 percent with the aid of labels, the majority of participants were unable to identify a deepfake video.18 In another study of social media users in Italy, Iacobucci et al. (2021) found that describing the threat of deepfakes to participants, along with providing a definition of a deepfake ahead of viewing manipulated video increased their correct identification of fake content by 28 percent and decreased sharing intention by 57 percent.19

In Epstein et al. (2023), the authors attempt to discern the public’s understanding of nine different labelling terms.20 The authors recruited thousands of participants from the United States, Mexico, Brazil, India, and China. The researchers found that the labelling terms vary in their effectiveness at accurately representing the moderation goals. That is, for content that is intended to mislead users, labelling terms “deepfake” and “manipulated” were consistently able to represent the content’s misleading nature to users. Whereas labels such as “AI-generated,” “Generated with an AI Tool” and “AI manipulated” were perceived as useful for content that is made with AI, however, it did not push users to consider whether the content was truthful or misleading.

In Gruzd, Mai & Soares (2024), the authors used ModSimulator, an open-source tool that creates an interface similar to Facebook’s, to understand the effectiveness of two types of warning labels: footnotes and the “blur” filter. In terms of content, they created posts with false claims about the Russia-Ukraine war and presented the experiment to 1,500 participants. The researchers found that both warning levels were effective at decreasing user engagement with the false posts, with no significant difference in effectiveness.21 In Lu and Yuan (2024), the researchers conducted two experiments where they had 2,808 US participants watch deepfake videos of politicians expressing opposing views on climate change, with some participants viewing the video with AI disclaimers and others without the disclaimers. They found that AI disclaimers significantly enhanced viewer’s recognition of the content as deepfake parodies.22

Generative AI content labelling approaches taken by social media platforms

In light of the increased use of generative AI tools to produce social media content, efforts have increased in recent years to identify synthetic images and videos. Methods such as watermarking and labelling posts have been introduced by platforms. Some technology companies have been collaborating in self-governance efforts. For instance, the Coalition for Content Provenance and Authenticity (C2PA) is a project that seeks to develop global technical standards for content provenance. The C2PA has many members, including corporations such as Google, Intel, Sony, Meta, Microsoft, Amazon, Adobe, and OpenAI.23

In Table 1, we outline the various approaches that popular platforms have adopted to label AI-generated content on their platform.

Table 1

*24 **25 ***26 ****27 *****28 ******29

Generative AI governance in Canada

Canada has faced a series of challenges in passing AI legislation. Most significantly, despite introducing the Artificial Intelligence and Data Act (AIDA) part of Bill C-27 in June 2022, the Act was unable to move past committee consideration before parliament was prorogued in January 2025.30 As a result, Canada does not have a federal statutory framework governing AI systems. However, there is a patchwork of strategies, regulations, codes, standards, and directives developed by public-sector departments and agencies for governing their respective uses of these technologies.31 Policy initiatives that cover the public sector tend to come from the Treasury Board Secretariat (TBS), while private-sector initiatives come from the department for Innovation, Science and Economic Development (ISED).

While most AI governance initiatives tend to focus on all AI systems, there is also a dedicated effort in focusing on generative AI systems. For instance, in September 2023, ISED launched the Voluntary Code of Conduct on the Responsible Development and Management of Advanced Generative AI Systems. Intended for the private sector, the Code outlines a series of principles for companies developing and deploying generative AI systems, regarding safety, transparency, fairness, and human oversight. They were initially developed as an interim measure while awaiting formal regulations through the proposed Artificial Intelligence and Data Act (AIDA) part of Bill C-27; however, with the Bill failing to pass prior to the recent prorogation of parliament, the future remains uncertain for private-sector governance of generative AI systems.32 

There is also a Guide on the Use of generative Artificial Intelligence developed by the Treasury Board Secretariat, but that applies only to how government institutions use generative systems. The Guide consists of principles and best practices that public servants should follow to responsibly use generative AI technologies.33 Through the Standards Council of Canada, the federal government is in the process of developing AI and data governance standards.34 The Digital Governance Council has also developed voluntary industry standards for AI.35

Moreover, the federal government’s efforts at regulating social media platforms have not only been hampered by prorogation, but they also do not focus on content moderation practices. In particular, Bill C-63: Online Harms Act, was tabled in February 2024 by the Minister of Justice. It sought to establish a regulatory framework for social media platforms by requiring them to comply with a series of duties and disclosure practices. The Bill sought to regulate seven categories of harm: content that sexually victimizes a child or revictimizes a survivor, intimate content communicated without consent, content used to bully a child, content that induces a child to harm themselves, content that incites hatred, content that incites violence, and content that incites violent extremism or terrorism.36 While important to consider child safety and violent extremism as part of the scope of online harms, other types of harms brought on by misinformation and disinformation need to be considered. Furthermore, the proposed approach would not compel platforms to engage in content moderation practices such as labelling and fact-checking.

In short, when it comes to genAI, the federal government has developed a variety of voluntary governance mechanisms in lieu of legislative statutes. While proposed laws such as Bill C-27 and Bill C-63 had the potential to address the increased use of genAI and its harmful effects, the limitations of the bills, and now the prorogation of parliament, have cut short those efforts.

Compare this to Europe, where there has already been significant action taken to govern the use of AI—specifically the EU’s Artificial Intelligence Act (AIA).

Having come into force in August 2024, the AIA seeks to govern AI systems by establishing sets of responsibilities and obligations to developers based on four categories of risk: 1) unacceptable risk, high risk, limited risk, and minimal risk (see Figure 1 for examples of systems in scope for each risk category). General purpose AI systems (GPAI) are also part of the scope of the Act and have separate requirements and obligations for developers.

As part of the scope of the AIA, companies that make AI systems that can be used to generate deepfakes and synthetic media must ensure that their tools label outputs as generated with AI. This may include techniques such as watermarks, or any other approach prescribed by regulation.37 Additionally, the providers must establish mechanisms that prevent the misuse of their tools, as well as measures to address any harms that may arise from their misuse. In light of the recency of AIA’s adoption, it is hard to assess the effectiveness of these measures—only time will tell. However, the requirements demanded by this Bill represent a significant step towards the labelling of AI-generated content online.

The EU’s Digital Services Act (DSA) seeks to protect fundamental human rights and promote a safe, fair, and open online ecosystem for users. It does so by regulating app stores, social networks, content sharing, online travel and accommodation platforms.38 Compared to the AIA, generative AI systems are not explicitly accounted for in the DSA. For instance, search engines must follow a set of obligations and produce documentation about the mitigation measures that they’re adopting. However, genAI systems may be used as search engines by users (i.e. retrieve information from the internet) which blurs the line between a standalone genAI system and traditional search engines.39 This distinction is difficult to draw because tools like ChatGPT can also be used to generate images, audio and video, instead of generating text that was developed by retrieving information online.40

As a result of the lack of specificity about generative AI in the DSA, regulators in charge of enforcing it will need to develop regulations that address this gap. For instance, regulators have already compelled search engines (Bing and Google) and online platforms (Meta, Snapchat, TikTok, YouTube, X) to provide information about the mitigation measures they are using to address the risks associated with generative AI.41

Survey Findings

In addition to the survey experiment on the impact of various labelling approaches, we also tested Canadian residents’ familiarity with deepfakes and who is most exposed to them. This included both their self-reported familiarity with deepfakes, but also how well they could define deepfakes when asked to. We also benchmarked exposure to deepfakes specifically against exposure to misinformation more generally seen online.

Familiarity

Most people are at least somewhat familiar with deepfakes—only 18 percent said they were not at all familiar. Unsurprisingly, there are huge differences across age groups. Among those 16 to 29, 35 percent are very familiar with deepfakes while among those over 60, that number is only 12 percent. Men of all ages are also more familiar, as are people with a university degree.

Those who describe themselves as more politically centrist are less likely to say they are very familiar with deepfakes—only 18 percent of those who place themselves between 4 and 6 on a scale of 1 to 9 from politically left to right say they are very familiar with deepfakes. This is lower than 26 percent of people on the right who are very familiar, and 30 percent of people on the left.

However, despite claiming to be familiar with deepfakes, Canadian residents describe the concept of deepfakes or synthetic images in a wide variety of ways.

Most respondents who said they were familiar with deepfakes could come up with a definition that was at least partially correct - touching on at least one element of what constitutes a deepfake. A total of 73 percent were able to at least in part correctly describe deepfakes—although that includes three percent who gave a very specific example of deepfakes rather than defining them. However, 13 percent of respondents who thought they were familiar with the concept actually ended up describing misinformation generally. Out of all respondents—including those who said they don’t know what a deepfake is, 45 percent of Canadian residents were able to define deepfakes correctly.

In defining deepfakes, 32 percent of respondents mentioned anything related to artificial intelligence—either as a core part of deepfakes, or simply one avenue to create them—while many others simply described any media that had been altered or manufactured. While this suggests a close tie between deepfakes and AI technologies, it is clear that not all Canadians immediately make that connection.

Exposure

Canadian residents are exposed to a significant amount of fake content online—ranging from misinformation to deepfakes across a huge variety of topic areas. Our survey suggests that Canadian residents reported seeing what they believed to be deepfakes less frequently than what they believed to be fake news, but about as often as they see fake news they believe to be true. One in five Canadian residents say they see synthetic media or deepfaked content online a few times a day, 47 percent of Canadians see deepfakes on at least a weekly basis, and 70 percent report seeing it at least a few times a month. Comparatively, 17 percent of Canadian residents see fake information about the news or current events they believed to be true at the time, but later find out was fake multiple times a day online and 67 percent see this type of content at least a few times a month.

Exposure to deepfakes is largely consistent with age—we find that just under half of Canadian residents across age groups report seeing synthetic media or deepfakes multiple times a week. However, older residents (60 and above) are more likely to be unsure how often they see deepfakes, potentially signalling that older residents have a harder time identifying whether or not content they see online is synthetic or real.

In general, those who use the internet more often are both more familiar with and more exposed to deepfakes. Only 61 percent of all respondents said they were at least somewhat familiar with the concept of deepfakes while 72 percent of those who use YouTube were familiar and 80 percent of those who use X were familiar. Exposure to deepfakes is highest on TikTok and Instagram, with 78 percent of TikTok users and 76 percent of Instagram users saying they see deepfakes at least a few times a month.

Conclusion

Canadians want to know when they are looking at AI-generated content, but the current approach by platforms has been haphazard—often implementing the lowest effort option without appropriately considering how effective that solution would be. YouTube, TikTok, and Meta platforms already allow users to flag their content as AI-generated and then leave a small label to indicate that to readers. Others like BlueSky and X (formerly Twitter) have no labels for AI-generated content. However, no platforms to date offer the full screening method for labelling AI that we tested in this project.

This research suggests that users are no more satisfied with the small label approach than they would be with no labelling approach at all. These small labels do not limit the reach of AI-generated content, they do not reduce the likelihood of this content being believed, and they don’t reduce the likelihood of the content being shared.

If platforms want to implement an AI content labelling regime that would actually limit the reach of AI-generated content, or leave Canadian residents feeling that AI-generated content was being labelled at all, a more effective approach would be to cover up the AI-generated content until users explicitly click to view it. 

This leaves open the question of how platforms should best identify AI-generated content on their platforms. While some have started implementing schemes to identify it automatically, many platforms still rely on users self-declaring that the content they are posting is AI-generated. This has little impact when those uploading content are doing so with the intent of deceiving viewers.

When content is uploaded to deliberately deceive viewers, self-regulation by platforms proves largely ineffective. Despite surveys showing that Canadians frequently encounter synthetic and deepfake content on social media—and many Canadian residents saying they want to be informed about its presence—platforms have chosen not to implement transparent labelling that would help users identify such content.

This situation is compounded by the absence of a statutory governance regime in Canada. If this country implemented statues similar to the ones in the European Union, that would mandate clearer disclosure practices. Without legal requirements compelling platforms to adopt more effective transparency measures, their current self-governance practices continue to fall short of addressing the concerns of Canadian users. This regulatory gap underscores the need for policy intervention to ensure digital platforms better protect consumers and promote informed decision making.