Authors

Tiffany Kwok

Christelle Tessono

Contributors

• Nina Rafeek Dow
• Mariana Rodrigues 
• André Côté
• Suzanne Bowness

Funder

This project has been funded by the Office of the Privacy Commissioner of Canada (OPC); the views expressed herein are those of the author(s) and do not necessarily reflect those of the OPC.

Tags

• AI
• Innovation + Tech
• Surveillance and Privacy

Share

What is Generative AI?

The whole wide world of generative AI

The growing use of generative AI tools follows a decades-old trend in digitization, whereby an increasing number of activities are mediated through digital platforms.

According to the Organisation for Economic Cooperation and Development (OECD), generative AI can be defined as AI models that “create new content in response to prompts, based on their training data.”5 GenAI applications can be used to generate text, code, images, audio, and video. There are an increasing number of companies that are developing or adopting generative AI tools in their products. These technological developments can be experienced across multiple sectors, for example in healthcare (e.g. patient case notes, clinical diagnostics, etc.), in the banking sector (e.g. as client-facing chatbots), in the financial sector (e.g. as virtual assistants for market research), and in education (e.g. for curriculum development and student assessment). However, for the purposes of this report, we will focus on genAI tools that are directly present in everyday lives of Canadian children and teenagers through K-12 education as well as on social media platforms.

In recent years, digitization has led to the use of big data analytics to optimize student performance at school.6 Technology is usually adopted for the following purposes: information search (e.g. digital libraries), school management (e.g. communications with parents, teachers, and students), and teacher educational tools (e.g. quizzes, automated proctoring, or grading). Right now, the use of genAI in schools is growing. An increasing number of companies have developed technologies for learning purposes, or are now adopting genAI applications as a part of their existing products. Generative AI companies are also adapting their technologies to operate in educational contexts.

In terms of uses of genAI in social media, we know it is currently being to train and serve as chatbots,7 to create content on platforms (video, images, text, audio, etc.), as well as to train algorithms from metadata and other types of user-generated content. Another related context that we believe is important to note, but are not within scope of this report, is in video game development.8 Furthermore, generative AI tools are increasingly being used in the provision of mental health support such as crisis help-lines.9 Companies will develop chatbots using the transcripts from help-line phone calls to train their systems.

Table 1: A non-exhaustive list of popular uses of generative AI in education and on social media platforms

GenAI and privacy harms

Recognizing privacy harms specific to children and teenagers is vital as genAI tools continue to become more widely used. Virtual learning environments currently being adopted in education feature information technology infrastructures that enable continuous data collection—which include, but are not limited to, student records, grades, geolocation, browsing habits, and personal information.10 In the absence of technical and policy guardrails preventing how data is collected and subsequently used, student information can be sold to a third-party and used for purposes that schools, teachers, and students did not consent to or are unaware of.11 Even if anonymized, student data collected by genAI tech providers can be re-identified and used for other purposes (malicious or commercial) by a third-party or the provider themselves as re-identification techniques have increasingly become more sophisticated.12 For instance, a study conducted by Na et al. (2018) showed that a machine-learning algorithm was able to re-identify approximately 80 percent of children and 95 percent of adults based on de-identified physical activity data.13 This indicates the ease with which sensitive data can be exposed to advertisers, governments, and strangers.14

In recent years, a growing number of institutions have experienced data breaches from cybersecurity attacks. For instance, the Toronto District School Board uses PowerSchool, a cloud-based program to store student and staff information. In January 2024, PowerSchool was hacked, which led to sensitive personal data from 1985 to 2024 to be compromised, including medical information, home addresses, health card numbers, student names, and dates of birth among others.15 GenAI technology providers are not immune to this problem. Other cases indicate that the risks are present, such as Character.ai’s data breach of usernames, voices, and chats due to internal error,16 and now defunct educational technology company AllHere’s mismanagement of sensitive student data, as called out by a whistleblower.17

The adoption of digital technologies in the classroom undermines traditional norms of intellectual privacy and safety.18 Prior to digital technology adoption, classroom discussions, assignments, performance indicators, and other relevant metrics, didn’t leave the classroom or the school. Students were free to discuss without concern that what they contribute to the classroom would not be used in other contexts. Now, the classroom extends to wherever the student accesses it, with their laptop, tablet, and even smartphone. Digital technologies are constantly tracking their performance, as well as having access to information outside the classroom such as their geolocation and lifestyle habits (e.g. extracurricular interests, nutrition, browsing history, etc). The large amounts of data collected about students enables unprecedented forms of surveillance. As illustrated in Figure 1, through the adoption of digital technology in the classroom, data flows between not only students and their teachers, but also between the educational technology platforms, providing data to third parties, and potential adversarial actors looking to use data in harmful ways. This raises a range of risks to the individual—from having their data sold to marketers for targeted advertising,19 to having data ”profiles” created that have the potential to follow students into adulthood, despite changing privacy preferences.

Figure 1: This model depicts data flows between students, instructors, schools, platforms, third parties, and adversarial actors. This model was developed by Cohney et al. (2021).20

Moreover, social media platforms have a wide range of privacy implications that may affect the children and youth who use them.21 With the growing adoption of genAI, the risks of privacy violation and data breaches increase. For instance, platforms such as Snapchat have launched conversational chatbots that act as companions to their users. The information that is disclosed by youth through these conversations can be used by platforms for a variety of undisclosed purposes.22 For example, although there are mechanisms in place to “collect consent,” the process by which it is collected is often unclear, so user data is not collected via informed consent. Further, we do not know whether this data is de-identified and anonymized.

Harms beyond privacy

Other types of harms beyond privacy are also important to mention. For instance, researchers have raised concerns about the degradation of pedagogical conditions. This concern builds on the decades-long digitization of education, whereby a teacher’s autonomy is compromised through the displacement of grading and curriculum delivery onto automated decision-making tools. Furthermore, since a majority of these tools are developed by for-profit technology providers, there is also a concern that pedagogical decisions are being shaped by these corporate interests instead of the public interest.23 In this case, genAI would accelerate this trend due to the technology’s advanced capabilities.

Experts we spoke with raised concerns that genAI is eroding educational norms and values such as academic integrity, attribution, and originality. Other concerns include the degradation of critical thinking skills of students due to heavy reliance on generative AI tools to interpret texts.24 Regarding its application in school settings for skills identification and career counselling, there are concerns that students won’t be adequately evaluated, which could be detrimental to their professional development.

Beyond AI’s impact on learning and academic performance, experts also raised concerns about the hyper-personalization of education afforded by genAI tools. Hyper-personalization refers to the set of technical approaches that analyze student learning patterns and capabilities to develop exercises and interfaces that ensure better engagement and performance outcomes. Specifically, experts argued that these technologies will facilitate the isolation of youth and increase challenges in developing strong social bonds with their peers. This could render youth even more vulnerable to phishing and catfishing scams as well.

Finally, and this is an issue that predates genAI tools, there are risks of ingrained biases and potentially discriminatory outputs, as genAI tools may not capture all lived realities, and histories of marginalization due to an overrepresentation of dominant Western epistemologies. For instance, this may look like adopting derogatory language such as slurs, creating contested socio-political framings (e.g. not recognizing a particular Indigenous nation), or assuming gendered representations of work (e.g. only presenting men as doctors) to name a few examples.25 Moreover, “hallucinations”, the term for chatbot responses that contain false or misleading information, are also significant problems with genAI technologies.26 This may lead to not only misunderstanding issues and academic material, but may also expose students to harmful practices (e.g. if a user asks about nutritional or health advice for instance and gets a non-standard answer or advice).

On social media platforms, genAI can harm youth when bad actors use it to create child sexual abuse material (CSAM) deepfakes of minors. There are growing cases nationally and internationally, where bad actors will use tools like open-source image platform Stable Diffusion to create this harmful imagery.27 It is also important to note that both youth (e.g. classmates, teammates, peers, etc), as well as malicious adult actors, can engage in this harmful behaviour.

There has also been an increase in the popularity and use of conversational chatbots. Examples of companies include Character.AI and Replika. These tools are often designed to optimize engagement, and as a result may lead to users developing deep emotional connections that open the door to a variety of mental health challenges. For instance, in October 2024, a 14-year-old American tragically died by suicide after developing an intense emotional attachment with a companion AI.28

Best Practices for Safeguarding Children and Teen Privacy

Privacy design principles as a common foundation

Privacy design principles can offer a common foundation for incorporating privacy considerations into the design and use of technologies like genAI. While privacy design principles are non-binding and less detailed than design codes and laws and regulation, design principles offer a systematic, proactive approach to ensuring that privacy, data protection, and user rights are embedded in policies and technology products.

“Privacy by Design” was a framework developed by former Ontario Information and Privacy Commissioner Ann Cavoukian in the 1990s, with the aim to embed privacy into the design specifications of technology.66 The framework is based on seven foundational principles:

1. Proactive not reactive; preventative not remedial
2. Privacy as the default setting
3. Privacy embedded into design
4. Full functionality - positive-sum, not zero-sum
5. End-to-end security - full lifecycle protection
6. Visibility and transparency - keep it open
7. Respect for user privacy - keep it user-centric

Privacy by Design has been recognized across Canada and internationally as one of the most widely used privacy frameworks both in the public and private sectors, and has been applied to information technologies, organization practices, and information architectures. Adherence to principles is often up to interpretation, and not legally binding. The unequal adoption of privacy design principles across sectors and users calls for firmer action and standards in best practices to implement said principles.

In addition to Cavoukian’s principles, a variety of organizations have also released privacy design principles to guide their work, many of which are values-based. From prioritizing fairness, accountability, security, and transparency, companies committed to these principles have taken action to increase clarity of privacy policies and their terms of service, and school boards have issued guidelines for safe and ethical AI use in institutions.

The Privacy by Design principles offer an intuitive, common foundation for understanding the contributors to privacy as it relates to AI systems used by youth. They can more directly guide the development of AI governance models and AI products for communities of policymakers, technologists, and privacy practitioners. The following sections will explore best practices for protecting youth privacy geared to each of these groups.

For policymakers: Youth interests as a priority for genAI governance

With the exception of Ontario's nascent Bill 194, there is no legal framework for governing youth privacy and genAI specifically in Canada today. For policymakers, genAI’s rapid deployment—and the vulnerabilities for specific populations like children and youth—create urgency in establishing governance frameworks, with important lessons for Canada from other jurisdictions that are further ahead.

Youth privacy should be specifically recognized in AI governance. Privacy-by-design should be mandated for companies and the development of all tools and features. Youth understanding of privacy differs from that of adults, requiring specific measures to be put in place to truly obtain consent and to understand what is being agreed to.67 Whether this is established through simplified privacy dashboards and prompts,68 or a complete ban on the use of dark patterns in the development of a tool,69 youth privacy requires additional attention and design measures beyond foundational privacy-by-design for adults.

Future iterations of privacy laws should set standards for clear consent mechanisms, transparency, and data collection, use, and processing. These standards should be put in place with accountability mechanisms and appropriate penalties. Regulators should be timely and consistent in establishing a standard of acceptable risk levels. Establishing codes of practice that are specific to sectors that develop and deploy genAI tools can be one avenue for timely, actionable support for technologists and teams innovating new technologies. Specific to youth, limits to data collection should be established and enforced, akin to the EU’s GDPR limitations on appropriate data collection and processing. These measures will ensure that companies and technologies cannot indiscriminately collect data for training purposes.

Policymakers and governments must take an informed, participatory approach by working alongside educators, child development experts, and youth to develop future governance and safeguards. This will ensure that aspects like standards for clear wording of privacy policies and terms of service, and mechanisms for obtaining consent are youth-friendly and conducive to protecting youth privacy. User testing for different types of privacy protection features with youth focus groups can help inform design decisions such as where, when, and how privacy notices should be displayed and written. Without involving youth in the development of policy interventions, any safeguards or policies will not adequately reflect the needs and lived experiences of young people. Regulatory sandboxes are a potential avenue for organizations to work alongside regulators in both the development of their technology and the testing of a policy intervention.

For practical tips, see the accompanying tool: Best Practices for Safeguarding Children’s Privacy with GenAI Tools.

For technologists and privacy practitioners: Privacy considerations in the development and maintenance of genAI tools

Generative AI technology poses a new challenge for the preservation of privacy. For technologists and privacy practitioners, there are risks to understand and manage, but also proactive steps that can be taken.

Privacy can be unknowingly compromised by sharing personal or sensitive information with a chatbot. Australia’s Office of the Victorian Information Commissioner (OVIC) published a report on the use of ChatGPT by a child protection worker, who had entered a significant amount of personal information relating to the child.70 Concerns revolve around the “black box” of how inputted data is used to train generative AI models, the potential data leakage risks,71 and the opportunity for cybersecurity attacks to retrieve inputted data.

Cyber attacks pose a significant threat, with the potential to compromise children’s privacy. With 90 percent of successful cyber attacks on generative AI systems resulting in sensitive data leakage, action must be taken at every stage of development and maintenance of systems to mitigate privacy risks.72 Our research found several privacy-related cyber attacks documented in literature and media, with the aim to leak data from the model, to bypass safety guardrails, or to poison the model’s training data. While there are a wide range of attacks, the cyber threats can be grouped by two target vulnerabilities: attacks executed through inputs from users, and through training data sets. Both are described briefly below.

Attacks executed through inputs from users have the potential to cause generative AI systems to produce outputs that go against safety guardrails, leaking personal data or hateful/illegal material. For example, “jailbreaking” and “prompt injection” attacks use specific, intentionally-crafted prompts to bypass existing safety and moderation guardrails or content filters. These generate malicious outputs. They can include distribution of misinformation in its outputs, disclosure of sensitive information, and the execution of malicious code.73 Model inversion attacks can extract or infer individuals’ data used in the training data of the models. This inherent vulnerability created through user input has serious implications for personal data that may be accidentally entered into the systems.

Attacks executed by targeting the training data directly are akin to traditional concepts of cyber attacks. By polluting the model’s training data, all subsequent outputs of the system are poisoned—potentially spreading false or offensive information, or revealing restricted data. “Data poisoning” attacks allow malicious actors to change the behaviour of a generative AI system—with backdoor data poisoning attacks injecting mislabelled training examples into a training set, resulting in the model learning the attacker’s intended output.74 For example, hundreds of poisoned datasets were discovered to have been uploaded onto the Hugging Face platform—a platform used for machine learning model collaboration –that have the potential to enable attackers to inject malicious code and impact many other models if downloaded and used.75

Developers of generative AI tools can take proactive steps to protect the privacy of users. Developers can make use of privacy-enhancing technologies (PETs) like data synthesis algorithms, differential privacy, and homomorphic encryption.76 Although these technologies are not foolproof,77 they are mitigating measures to implement as foundational protections. Data should be pre-trained and fine-tuned, should receive regular evaluations of its output filter, and should be minimized, anonymized, and de-identified.78 For technologies that are youth-facing, efforts should be made to use synthetic data (artificially generated data rather than data produced by real-world events). If youth data is collected, easily comprehensible notices should be made at the initial access point of the tool, and data processing and time limits for retention should be transparent, accessible, and held accountable by a dedicated governing body.

Throughout the development of the technology, feedback and reporting mechanisms should be put in place for testing users to provide input to developers’ further improvements of the model and safety assurance.79 Feedback should continually be collected from users, even post-deployment. In the case of tools that may also be used by youth, companies should make efforts to create accessible and age-appropriate feedback mechanisms as well. This will provide helpful insights into any features that need to be adjusted or removed if in violation of privacy protections.

Technologists should prioritize ongoing maintenance of genAI tools.80 This can include use of privacy impact assessments (PIAs), risk-assessment frameworks, executive accountability processes, and data protection impact assessments (DPIAs). Companies should also resource proactive monitoring for new trends and potential emergent harms that may elude existing enforcement systems. In the rush to build and introduce new features and products to the market, technologists and companies should recognize that prioritizing privacy-by-design from initial research and development stages will build a strong foundation for future products and its users to thrive.

For educators and parents: Protecting youth privacy in their exploration and use of genAI tools

For school administration, institute privacy impact assessments (PIAs) and develop robust cybersecurity. These safeguards are important in determining which technologies to approve for use in schools. Before approving and procuring any technologies for student and educator use, PIAs should be conducted by a privacy expert/officer. School boards interviewed for this study including the Ottawa Catholic District School Board and the York Region District School Board. Each had their own PIAs, vetting approaches, and tools to explain different “levels” of approvals for educator use. While each school board had different grading methods and allowances for educator use of certain genAI tools, both stated the commitment to using PIAs before releasing genAI tools was non-negotiable.81 This allows for longevity of the products procured, and takes the weight off of the educator’s shoulders in assessing these tools. A dedicated cybersecurity team and robust cybersecurity plans should also be developed and kept updated, to proactively protect schools from new cybersecurity risks that may be introduced with any newly approved technologies.82

School boards looking to approve and implement new technologies should refer to legislation (for example Bill 194 in Ontario), and privacy-by-design principles. Beyond privacy risks, educators should also be informed of harms and risks associated with the use of genAI tools (e.g. potential for biased outcomes, hallucinations),83 and be trained to act to reduce the risk of these harms (e.g. start conversations with students about why an output was generated; engage in critical thinking and fact-checking of outcomes).

In addition, school administration should review a school’s resource availability and capacity to pilot tools prior to adoption, to consider structural power imbalances and inequity. Ask questions like: Can teachers co-develop the testing process for the tools? Can parent-teacher associations (PTAs) and students contribute to and contest this process? These questions will help school boards meaningfully assess tools that educators and students will use.

For teachers and students in schools, foundational digital and media literacy training can be extended to include AI literacy. School boards can provide opportunities for educators to learn about the board’s plan to implement and approve different technologies, and to be informed about capabilities, privacy risks, and tips before using the tools. Resources like Stanford’s CRAFT AI Literacy offer a great foundation for educators to help students explore and question AI. The Dais’ Digital Literacy Toolkit (set to release in March 2025) offers digital literacy lesson plans for educators on topics like deepfakes, synthetic media, misinformation, and disinformation.84

Offer standardized guides and “explainers” about genAI and privacy. When updated iteratively to reflect the rapid change in the technology, these guides can offer a do-it-yourself resource for educators, students, and parents to use and reference. School boards like the Ottawa Catholic School Board85 have developed their own guidance, creating both internal and externally accessible resources on their websites. The GenAI and Youth Privacy Tipsheet paired infographic likewise is one of these quick resources for educators. Resources like TeachAI’s “AI Guidance for Schools Toolkit” provides an easy-to-use framework for schools looking to adopt AI tools responsibly and transparently.86 Ensuring that educators are familiar and comfortable with the introduction of these tools enables students to be fully supported and to understand the safety precautions (e.g. not entering personal information or information beyond what is necessary) around using genAI tools. Training options and support to use alternative technologies should also be made available for educators who may not yet feel comfortable using these tools.

Whether through education or personal contexts, youth are increasingly interacting with genAI technologies in their daily lives. Tools offered to students and educators provide potential benefits, from personalized and iterative lesson plans and AI tutors to chatbots offering human-like interactions and companionship to users. For teachers and education administrators as well as for parents, there are steps that can be taken to mitigate the privacy risks that come with the use of these tools.

While at home, parents should practice the following three steps to manage privacy risks in the use of genAI tools by their children (and their own use).

1. Read and understand the privacy notices of genAI tools you or your child might use. While some privacy notices may be simpler to read than others, ensuring that you and your child understand and are comfortable with the data that will be collected, and how it will be used and stored, is crucial to fully providing consent before using a tool.
2. Stay up to date with your digital, media, and AI literacy skills. Knowing at a high level how generative AI works, as well as its capabilities, its risks and harms (e.g. bias, data collection, hallucinations), and how you can safely and effectively use the tools to your benefit will allow you to support your child’s exploration and use of the tool. Staying up to date will equip you with the skills to navigate newer emerging features and products. Some of these resources can be found at this footnote.87
3. Talk with your child about genAI tools and features they use on a regular basis. Regardless of the tools your child uses, whether integrated genAI features on social media platforms like Snapchat’s My AI and Meta AI via Facebook or Instagram, chatbots like Character.ai, or large language models (LLMs) like OpenAI’s ChatGPT, Google’s Gemini, or Microsoft’s Copilot, knowing which tools your child is using means you can guide them in thinking through the opportunities and risks associated with each tools, and to advance in your AI literacy together.

For practical tips, see the accompanying infographic: GenAI and Youth Privacy Tipsheet.

Summary

• Privacy-by-design principles are a useful foundation for the development of genAI tools. In relation to genAI, companies should anonymize or pseudonymize data, minimize data being collected, conduct regular audits and third-party vetting, and clearly communicate what types of data are collected and for what purpose.
• Policymakers should prioritize and make specific arrangements to protect youth privacy. Privacy protections should be accessible to youth, have accountability mechanisms in place, and be co-developed with youth and relevant subject-matter experts.
• Technologists are core decision-makers in the development stage of any technology. Make use of, and actively review effective privacy-enhancing techniques, conduct ongoing maintenance in the form of privacy impact assessments (PIAs), and provide agile feedback and reporting mechanisms for users.
• Educators and parents play a key role in a child’s development and understanding of technology. Be digitally fluent by understanding the capabilities and limitations of the technology to guide youth in their exploration and use of genAI tools.