Data Science

Mapping the human screenome: Our vast digital lives

Olivia Peterkin — Thu, 12 Jun 2025 23:39:51 +0000

Mapping the human screenome: Our vast digital lives Olivia Peterkin Thu, 06/12/2025 - 16:39

In this episode of School’s In, Professor Nilam Ram discusses the Human Screenome Project, a research study that explores cell phone usage and what it says about the way we process information.

July 10, 2025

By Olivia Peterkin

What does your cell phone usage, and the content you consume, say about you and how your brain works? And how can this information be used to better our lives?

According to Nilam Ram, a professor of psychology and communication at �� University who is studying the digital lives of adults and teenagers through the Human Screenome Project, the possibilities are limitless.

“The digital world is humongous and there’s so many different parts and places of it that are interesting and contain all kinds of great information about so many things, way more than even exist in our current physical universe,” said Nilam Ram, a professor of psychology and communication at �� University.

Ram joins hosts GSE Dean Dan Schwartz and Senior Lecturer Denise Pope on School’s In as they discuss his research and the light it’s shedding on digital switching and the human brain. They also discuss the difficulty in finding a strong connection in how cell phones affect mental health.

“The way in which your smartphone use is related to your mental health is very different from the way that my smartphone use is related to my mental health,” Ram said. “I may use my smartphone more when my mental health is low, you may use your smartphone less when your mental health is worse. So the same relation can exist in different directions for different people.”

“Even in a four-week period, the relationship between my anxiety and my smartphone use may be different one month than it is from another month,” he added. “So it’s very hard to make a general statement about how this type of behavior is related to fluctuations in mental health.”

They also discuss the ethics of algorithms that gauge user interest for commercial purposes, and ways that a different set of algorithms could instead be used for good.

“Our hope is that we would be able to provide a set of algorithms that have a different purpose than trying to increase engagement or purchasing and would be more interested in how it is that I'm doing as a person and the goals that I would like to achieve.”

“We’ve spent a lot of time thinking about the potential dual use possibilities here, both for good and for evil and yes, we are trying to promote the good side of the whole piece, but being aware that the evil side is there,” Ram said. “And so we’ve reached this conclusion that we sort of have a moral obligation to try to do this research in order to provide a different way of looking at the data than a profit-motivated company might,” he said.

Transcript

Nilam Ram (00:00):

Finally, we have a device that allows us to be able to express the switching our brains were designed to do.

Denise Pope (00:09):

Welcome to School's In your go-to podcast for cutting-edge insights in learning. From early education to lifelong development, we dive into trends, innovations, and challenges facing learners of all ages. I'm Denise Pope, senior lecturer at ��'s Graduate School of Education and co-founder of Challenge Success.

Dan Schwartz (00:33):

And I'm Dan Schwartz. I'm the Dean of the Graduate School of Education and the faculty director of the �� Accelerator for Learning.

Denise Pope (00:42):

Together, we bring you expert perspectives and conversations to help you stay curious, inspired, and informed.

Dan Schwartz (00:52):

Hi, Denise.

Denise Pope (00:53):

Hi, Dan.

Dan Schwartz (00:54):

I have a good question for you.

Denise Pope (00:56):

Okay. I need a good question. That's good.

Dan Schwartz (00:58):

So imagine you could look at my cellphone and see everything that's happened, say over the last day. Would you be able to tell anything about me if you looked at it? Like, "Boy, he's just looking at airline tickets."

Denise Pope (01:15):

Yeah, I think I'd be able to tell a lot about you.

Dan Schwartz (01:18):

So what do you think you'd discover? Because I know the answer, I actually know the answer.

Denise Pope (01:24):

Yeah, I know you know the answer 'cause it's your phone, right? No, I think that Dan Schwartz is probably not so much a phone guy because you're on your computer for work for much of the day. So my guess would be that, this is a total guess, during the day Dan Schwartz is not on his phone a lot, maybe answering a quick text like, you know, who's picking up food for dinner or a quick text how much longer are you going to be in your meeting 'cause we have someone else coming in. I think those kinds of things are happening on your phone. But I would not peg you as someone who, the minute there's a break, you pick up your phone and you start scrolling and looking at dog pictures. Am I right or am I wrong?

Dan Schwartz (02:02):

You are, but I can tell you more specifically what you would learn.

Denise Pope (02:05):

Okay.

Dan Schwartz (02:06):

You would look at my chats and you would see that almost all my chats are, "I'm really, really sorry, I'm 10 minutes late."

Denise Pope (02:12):

Yes.

Dan Schwartz (02:12):

Just every single one.

Denise Pope (02:13):

That's what I was guessing, yeah.

Dan Schwartz (02:15):

And then you know what kind of person. I'm apologetic, but I don't do anything to fix it.

Denise Pope (02:20):

Let's flip it for just a second. If you looked at mine, I am a news junkie and you would learn that about me. And I text often with my children and my husband. It's a very easy way for us to communicate during the day, just like a quick little thing, but neither you or I are like, "let's go shopping on the internet for the five minutes we have in between meetings." I don't see that happening.

Dan Schwartz (02:45):

You are correct, but I think it is interesting that you could figure something out about people. We have a guest who is really working on this.

Denise Pope (02:54):

Yay.

Dan Schwartz (02:55):

His name is Nilam Ram. He's a professor of communication psychology at ��, and he leads something called the Human Screenome Project, a great name.

Denise Pope (03:03):

Oh my God, I love that name.

Dan Schwartz (03:05):

Yeah. No, it's really good. And he's looking at people's use of digital media from moment-to-moment across the lifespan.

(03:12):

So welcome, Nilam. It's good to have you here.

Nilam Ram (03:15):

Thank you very much for having me. It's a pleasure to be here.

Dan Schwartz (03:17):

So my memory of this, The Screenome Project, was that somehow you convince people to let you take a screenshot of their cellphone every five seconds for like two months, something like that?

Denise Pope (03:31):

Wow.

Dan Schwartz (03:32):

Is this right?

Nilam Ram (03:33):

It's amazing. We install software on study participants' telephones. We take screenshots every five seconds and we've done some people even all the way up to a year now. We see everything that they travel through, all the different aspects of media that they get exposed to, things you guys were talking about with the text messages, cats, dogs, all that's in there.

Dan Schwartz (03:56):

Wow. So before we get to what you found, why did people agree to do this? Lik, is this a new-generation that everything they do is public, they don't care. Like the chances I want you to follow me around and take a picture of me every five seconds is zero.

Denise Pope (04:17):

You're probably not the candidate of choice, Dan.

Dan Schwartz (04:20):

Well, just what is the psychology of someone who says, "Sure, my life's an open book and in fact I want you to see every single page of it?"

Nilam Ram (04:29):

Well, I think that there are some general trends in the way that we are viewing privacy of ourselves and our own behaviors. But there's also a thing about contributing to science and trying to understand both how people behave in the world and also how is it that I behave in the world and can, if you learn more about me, you can help me optimize my life and help me not be late to every meeting, to provide the reminders at the right time. Like we would be able to automatically try to provide that kind of help if we learn what those behaviors are.

Denise Pope (05:02):

It's for the good of science, Dan.

Dan Schwartz (05:06):

Yeah, yeah. So before we get to findings, this is an incredible amount of unstructured data, like a picture every five seconds. There's no chance a human is going through all that, right. You would lose your life for every participant 'cause it would take you five seconds per screen to look at it. So you have a machine to do this somehow?

Nilam Ram (05:29):

You're absolutely right. It's lots and lots of data we've collected now up to, I think, it's three and a half centuries' worth of people's smartphone usage.

Dan Schwartz (05:35):

Jesus. Wow.

Denise Pope (05:35):

Oh, that is crazy. That's crazy.

Nilam Ram (05:40):

And so in order to be able to look through that material, we are employing computational techniques in order to be able to articulate. So say, give an example, we think that Dan is a dog person rather than a cat person.

Denise Pope (05:51):

You would be correct, Nilam, yes.

Nilam Ram (05:54):

So we go through the images on the screenshots and we look to see okay, how many dogs are you exposed to in the pictures and how many cats and then we can try to do a classification of you as a dog or a cat person.

Denise Pope (06:07):

I just want to point out one thing though 'cause if you looked at my phone, things that you say out loud, so I might not be, I get a lot of exercise videos. I am not an exercise crazy person, but because I work out with a friend and we talk about this, my phone picks it up. And so you might think that I'm a crazy exercise person, when really I'm just once or twice-a-week walking with a friend and we're talking about it. So I worry about that a little bit. Are you picking up actually if Dan's a dog person or if Dan's just in situations that talk about dogs and then the phone algorithm picks that up?

Dan Schwartz (06:42):

Everybody around me is a dog person and I really hate them 'cause I'm a cat guy.

Denise Pope (06:51):

Okay. You know what I'm saying, Nilam, right?

Dan Schwartz (06:51):

Yeah.

Nilam Ram (06:51):

Yeah. So I think it's one of these classic person context debates about how much of the behaviors which we see and express are related to us as people or related to the context that we're moving in and out of. And I think in this project we've sort of literally taken the screen, which is the membrane that connects us to a set of algorithms and digital context, and we're taking pictures of that screen, that membrane.

Dan Schwartz (07:17):

Yeah, that's a great way to describe it. I really like that.

Denise Pope (07:19):

Yeah. So you're learning about the algorithm in Dan's life that goes along with Dan's life-

Nilam Ram (07:24):

As well as about Dan.

Denise Pope (07:24):

Yes, yes, yes. That's important, I think that's important. And also, did everyone understand that why we think The Human Screenome is such a cool name? I just want to make sure that that's clear because there is something called The Human Genome Project, and I'm assuming that's how you came up with Screenome. Yes, Nilam?

Nilam Ram (07:40):

Yes, that's right.

Denise Pope (07:42):

And just so folks know, what does The Human Genome Project do? It maps the genes and so, all the genes, which is a lot and so The Human Screenome is mapping all the screens that you're on basically in a day.

Nilam Ram (07:58):

Yeah. One difference between the two projects is that we do like to think of the genetics and the genome as being static, like it's what you were given when you were born, what's fixed. In contrast, the screenome is constantly changing. Every day we're looking at different types of material, moving in and out of different spaces. So in that sense, it's a much more dynamic ome or ome mixed than genomics.

Denise Pope (08:22):

So cool, I love that. Cool.

Dan Schwartz (08:25):

All right. So give us a taste of one of the things you found.

Denise Pope (08:29):

Yeah.

Dan Schwartz (08:29):

I've been stalling just to build up the anticipation.

Denise Pope (08:30):

Suspense, yeah.

Nilam Ram (08:33):

One of the interesting things is that these devices allow us to very quickly move from one application or type of content to another application. So I can be looking at the news and also answering a text for my kids, all within just a few seconds of each other. So people are switching back and forth between these different applications on average on their smartphones every 10 or 11 seconds. And that's an average. So there are some that are shorter than that, and also some which are quite a bit longer. But on average, it's switching every 12 seconds, which means we're able to do many, many things within an hour on our smartphones 'cause there's so much material to switch back and forth between.

Dan Schwartz (09:13):

So all this switching, is there some implication? So here, this is a very far analogy, but people ask me what it's like to be dean of the school. And the answer is it is very fast, things keep changing. And the consequence is I don't really get to savor anything. I just get one thing and then I move on to the next. So all this switching world of less sort of appreciating what you have because you move on so quick or should I not care?

Nilam Ram (09:44):

It's an interesting hypothesis. I wouldn't be able to say directly of whether people are doing less savoring or more savoring. I would put it in the context of people have multitasks since humanity began. So take a mother is doing six different things simultaneously with their child and the rest of the house and the rest of the family and work at all at the same time. And so the devices are allowing for greater expression of that, or maybe we can say better measurement of that because so much of those behaviors are now digitally mediated.

Denise Pope (10:16):

So moms are going to finally get the credit for juggling, those pictures of a mom with 18 balls in the air, now you'll actually show this mom actually is doing a million things at once and go mom, yeah?

Dan Schwartz (10:29):

So I don't know what I think about the 10-second sort of average. Help me think about that.

Denise Pope (10:38):

What are you asking, Dan? What do you mean?

Dan Schwartz (10:40):

Well, I sort of want to say 10 seconds. Oh, that's good. That's interesting. But it's like, okay, this is really bad. It's like when I look at the stars in the sky on a clear night and I look at them and I go wow. And that's about as far as I get thinking about it. It's like wow, that's a lot. So help me go beyond wow in the 10 seconds. Give me something to hang on to.

Nilam Ram (11:02):

Well, I think that our brains and minds and bodies are built in order to be able to process lots and lots of information. And that includes looking out for lions that are running across the savanna and that might eat us, as well as for the moments of the stars and appreciating the beauty of the universe that we live in. And all of that is simultaneously happening in the world and it's happening all within this screen, all within 10 seconds of each other. So now we can move between these, what we used to think, were very different kinds of content much more quickly, and that could lead to a greater interconnection between the different parts of our lives. So our work life is not separate from our home life because I'm doing the texting and the e-mailing all simultaneously within each other. So we're getting greater and greater connections, one hypothesis is we're getting greater and greater connections among the different varieties of life experience that we have.

Denise Pope (12:03):

I think that's so true. If you think about even just the newspaper, right, because I used to get an actual newspaper and I would read it in the morning and then I would go to work and I wouldn't be turning on news or getting bings or dings on my phone. I'm saying this as a news junkie. And then if I wanted more news, I'd have to wait till the 5 o'clock version or 6 o'clock when I got home. And now, you get notified like urgent, alert, so-and-so did this in the Senate or whatever and it's constantly throughout the day

Dan Schwartz (12:31):

And that's good?

Denise Pope (12:32):

No, I don't know. Well, that's what I'm saying. I think there's some good happening with connections. It's great that I can just answer a quick question from a colleague who just needs something super quick and they don't have to wait till the end of the day and I'm not disrupting their workflow. And yet, there's a lot of other stuff that's disrupting workflow-

Dan Schwartz (12:52):

I have to ask the question.

Denise Pope (12:52):

Okay.

Dan Schwartz (12:54):

So there's a lot of concern that the big social media companies have algorithms that make you sort of want to stay on. And so this behavior where I'm switching to some extent is rewarded. And so it's driving me to switch more than I used to. Can you track someone and say, "Oh, this person didn't switch very much, but over time I noticed because they started using XYZ, they're switching more and more?" And by the end, Denise is checking her cellphone all the time to see what's happening in Washington, DC which she doesn't really care about anyway.

Nilam Ram (13:33):

Well, so we see evidence of the switching. We don't see evidence that the switching is happening differently than it is in other domains or in physical world compared to the digital world, that this is a log normally distributed thing, which means that there's a lot of short switching and there's some long segments as well. And that's exactly the same distribution of switching across different tasks that we see in the physical world as well, same distribution. So we're not seeing that it's different, it's just that the technology is facilitating the expression of neural processes that allow us to do switching at a rate at which their brains can do. So in that sense, it's finally we have a device that allows us to be able to express the switching our brains were designed to do, is another way to look at the question.

Dan Schwartz (14:29):

So one of the interesting things about genome is we have lots of genes, but it's the context that determines whether they express themselves.

Denise Pope (14:39):

Oh.

Dan Schwartz (14:39):

And so the environment you're in will activate a gene and then it'll cause some change. And so that sort of feels like the cellphone story, that it's kind of there, but the context is releasing my responses to it to a large extent as opposed to me initiating a tremendous amount.

Denise Pope (14:59):

Everyone, this can't possibly just be me, is just flooded all the time with things. I was talking to my mom about buying a dress and then the next thing I know, all of these dress ads show up. I didn't look for dresses even, right? It was just literally out in the ether.

Dan Schwartz (15:18):

Yeah. So I intentionally do not use my cellphone for certain things 'cause I just don't want the context to intrude.

Denise Pope (15:26):

Like what? Wait, this is so cool. Tell me.

Dan Schwartz (15:28):

Well, I don't shop on my cellphone, for example.

Denise Pope (15:32):

Oh.

Dan Schwartz (15:32):

Because once you do that, you get ads that keep showing up.

Denise Pope (15:36):

Yes, this is my problem ,and I could care less about it. So wait a minute-

Dan Schwartz (15:40):

So I'm trying to control the environment that's just on the other side of the membrane of the cellphone so that I can-

Denise Pope (15:45):

Look at you. That's amazing.

Dan Schwartz (15:48):

Who knew? Of course, I have no Facebook, no LinkedIn, nothing.

Denise Pope (15:53):

You're like one of those-

Dan Schwartz (15:54):

Go away world.

Denise Pope (15:57):

That's interesting. I don't think I could do that because it's just so much more convenient to shop on your cellphone-

Dan Schwartz (16:03):

I know, I know.

Denise Pope (16:04):

... or I look at recipes all the time and then all of a sudden, I'm just getting tons and tons of recipes and shopping lists and things like that. So there's certain things that you just don't do on your cellphone. You use your cellphone only for certain types of behaviors?

Dan Schwartz (16:18):

Yeah, yeah.

Denise Pope (16:19):

So you text?

Dan Schwartz (16:22):

I text, I check the airline apps. I like looking at the exercise stuff, how many steps.

Denise Pope (16:28):

Oh, you mean tracking your own steps?

Dan Schwartz (16:31):

The problem is if I'm at the dentist and I'm waiting an hour before I go in and all I've got is my cellphone, it's tough not to start going in and then I'm thinking if I do that, you start getting feeds. I don't want it.

Denise Pope (16:42):

Yes. And the big brother is watching you and all of a sudden...

Dan Schwartz (16:46):

How many times can I check the weather?

Denise Pope (16:47):

So, Nilam, this is so fascinating. Now, am I right to know that you have just adults in this study or have you included some young people?

Nilam Ram (17:00):

No, we've also included young people. We have adolescents that are age 13-plus, and we also have, for a subset of those adolescents, we also had their parents participating. So in some cases we are looking at both the parents and the adolescents.

Dan Schwartz (17:12):

Wow.

Denise Pope (17:13):

That is so cool. That is so cool. Okay. Can you give us any insights into findings of young children or younger 13 through 18 and versus adults and their parents?

Nilam Ram (17:25):

We haven't done yet the analysis to do the contrast between the two. One short answer is everybody's on their smartphones a lot. It's not necessarily the case that the adolescents are on more, often more than the adults, um are.

Denise Pope (17:39):

Interesting.

Nilam Ram (17:40):

And that the behaviors are in part related to each other, but also not related to each other so that the types of content that adolescents are engaging with is slightly different than the types of content that adults are engaging with.

Denise Pope (17:58):

Do you mean like TikTok, some kids spend more time on certain apps than adults do? Is that what you mean?

Nilam Ram (18:06):

Yeah. So there are some app differences, but there are also the types of tasks, which we are engaged with are slightly different. So one of the predominant aspects of adolescents is peer relations and engaging with peers and trying to understand those peer relationships, whereas adults are more engaged with work and family relations compared to the peer relations.

Denise Pope (18:29):

That makes sense.

Nilam Ram (18:31):

And that's true both on the smartphones and off the smartphones.

Dan Schwartz (18:33):

Yeah. Yeah, that makes a lot of sense.

Denise Pope (18:34):

That totally makes sense.

Dan Schwartz (18:36):

So as a basic question, and this is really fascinating, just sort of how do people allocate their time? Are they good at it? Things like that. Are there practical applications of what you're doing? You've been able to detect early warning signals for kids, for example, that you follow them and you sort of say, "Uh-oh, this kid's heading in a bad direction," or something like that?

Denise Pope (19:00):

Wow.

Nilam Ram (19:01):

Yeah. So that's the long-term dream is to be able to take this relatively continuous data streams of people engaged in their natural daily lives and to be able to learn patterns that exist in there and then to help people optimize those trajectories. So would we eventually be able to, if we learn the patterns, provide time-sensitive and context-sensitive interventions like text messages that would interrupt somebody's flow in order to return them towards the flow which they would like to achieve?

Dan Schwartz (19:34):

Oh, that's nice. That's a nice application. So a lot of people are worried about cellphones and various aspects of mental health. So are you getting something there 'cause being able to send a text to a kid that sort of says, "Stop what you're doing, go this other direction," you're sort of doom-scrolling kind of thing.

Nilam Ram (19:53):

Yeah, so the literature generally about the relationship between social media use and mental health has been very mixed. On average, across all of the studies, it's basically, there's zero association between those two. And yet at the same time though, the empirical evidence is showing no relation or intuitive senses say, "Oh, there is a relation between my smartphone use and my own mental health." So why aren't we finding it? And in the studies which we have done, the analyses we've done so far is that we can find those associations, but they're very idiosyncratic. The way in which your smartphone use is related to your mental health is very different from the way that my smartphone use is related to my mental health. I may use my smartphone more when my mental health is low, you may use your smartphone less when your mental health is worse. So the same relation can exist in different directions for different people.

Dan Schwartz (20:52):

So having all the data in the world turns out to be complicated to find a clean hypothesis. Is that sort of the answer?

Denise Pope (21:02):

You were hoping for like yes, mental health goes down when your phone use goes up at a certain point or something. Is that what you were hoping for, Dan?

Dan Schwartz (21:10):

Yeah. If your home screen is orange, lookout, something simple.

Denise Pope (21:14):

It's not that simple, it sounds.

Nilam Ram (21:17):

Yeah, I would say that it's definitely more complicated and much more nuanced and much more idiosyncratic. Even in a four-week period, the relationship between my anxiety and my smartphone use may be different one month than it is from another month. So that complexity is all embedded there. And now that we have the data to be able to dive in and really look at those nuances, we're finding them everywhere. So it's very hard to make a general statement about this type of behavior is related to fluctuations in mental health.

Denise Pope (21:50):

I think there's going to be a bunch of teenagers saying, "Yay," if we have any teenage listeners because I think so much of what you're hearing in the news and from certain cities as you say is like, this is bad, this is bad, this is bad and kids are just really struggling as they see the party play out online that they weren't invited to or they see everybody looking beautiful all the time and all of this. So you're basically saying, "Wait a minute, it's a little bit more complex than that." Yeah. Am I understanding that?

Nilam Ram (22:23):

Yeah, much more complicated. Sure there are moments when I feel as though I have been left out of something that my peers are doing, but there's also right next to them, 15 seconds later, a moment that my friend says something very nice to me-

Denise Pope (22:37):

Online?

Nilam Ram (22:38):

... and I need that support. Online, right. So all of this is happening within these 15-second snippets back to back. So it's not all bad and it's not all good. It's all mixed together and very complicated.

Denise Pope (22:51):

It's so hard as a parent because you are just so worried about your kid and this is so new and you hear so much stuff. I mean just even you saying that makes me think, it goes back to the development of you want to know your kid's friends, you want to know who they're hanging out with, virtually and/or in person, 'cause you want your kid to be able to say, "Hey, no, you're great," right after they're feeling down.

Nilam Ram (23:17):

Yeah. Yeah, and that's hopefully what our eventual goal would be, is to be able to deliver those types of interventions in that moment when they're needed the most.

Denise Pope (23:26):

It's also a little big brother-y, I have to say, right? We're always watching you, but I guess people are always watching anyway. That's how I get my little videos sent to me is the algorithm is listening, right? So maybe it's not that different, I don't know. Sorry, I just had to throw that in there.

Nilam Ram (23:42):

Yeah. I think it is true that the algorithms are now working with us in many ways, and our hope is that we would be able to provide a set of algorithms that have a different purpose than trying to increase engagement or purchasing and would be more interested in how it is that I'm doing as a person and the goals that I would like to achieve.

Denise Pope (24:03):

But I know you're a good guy, Nilam, and so you would be doing things for positive, but I can also see people doing things to mess with people. I don't know. It does scare me, but you're right, it's already out there, so it's not you.

Dan Schwartz (24:16):

Denise. I have a fitness app and it says get up and walk. I've been sitting too long. Does that seem evil to you?

Denise Pope (24:27):

No. That's what I'm saying. We can think of all the good things, right? But I could also, you know, have you both read 1984? You can also be like go out and consider buying a gun, you know, and here's all the cool things about owning a gun, and if you're underage, this is how you get a gun, and now here's how you make a gun and I- I mean, I don't know, I'm watching too many true crime stories clearly on TV, but right for all the good, I am a little bit worried about that power. You must think about this ethically.

Nilam Ram (24:58):

Oh, yeah. So we've spent a lot of time thinking about the potential dual use possibilities here, both for good and for evil and yes, we are trying to promote the good side of the whole piece, but being aware that the evil side is there. And one of the reasons that our participants are engaged with us as a research institution is that we are providing a public good, which is not motivated by profit in order to understand human behavior and how it's working. And so we've reached this conclusion that we sort of have a moral obligation to try to do this research in order to provide a different way of looking at the data than a profit-motivated company might.

Dan Schwartz (25:42):

Yeah, no, that's important. So if there's a narrative out there that you would like to correct or get people to hedge on, what would it be about cellphone use?

Nilam Ram (25:56):

Yeah, I would think that cellphone use is not bad. We're carrying them around for reasons that they are useful to us in many, many different ways. And they're useful for me in a different way than they are for each of you and for each of their listeners. And to sort of celebrate that heterogeneity in the ways in which they're being used, and to try to leverage them as technological innovations that can help people optimize their lives in ways that are good for them.

Denise Pope (26:24):

Can I ask a silly question? I have a silly question to ask. You don't have to say the person's name obviously, 'cause of privacy, but has there been a person where you're like, "Oh my gosh, this person is literally on their phone all the time." Like, who wins the prize for most on their phone all the time?

Dan Schwartz (26:44):

Bob.

Denise Pope (26:44):

That's what I want to know.

Dan Schwartz (26:45):

Bob.

Denise Pope (26:48):

Don't tell me their name, but are you surprised by just the sheer amount of time people are on their phones or is this like, "No, Denise, everybody's on their phone all the time?"

Dan Schwartz (26:56):

So can I give a story on this?

Denise Pope (27:00):

Yeah, then I want Nilam to answer.

Dan Schwartz (27:01):

I know, we have guests who are experts, whatever.

Denise Pope (27:05):

Yes. Hello, yes.

Dan Schwartz (27:07):

So I had an undergraduate who wanted to work with me in my lab, and so she comes and she asked me a question, and while I start to answer, she gets onto her cellphone and starts doing something. And I stop talking and it takes her about 5 seconds, 10 seconds to realize I've stopped talking. And I said to her, "You realize that you asked me a question. As soon as I started answering it, you stopped paying attention. You went..." And her response to me was, "Get used to it. I'm the new generation."

Denise Pope (27:36):

Yes. So did not hire her.

Dan Schwartz (27:38):

So I appreciate Nilam saying it's good and bad, but there are social norms where I want to say, "Sorry, you lose" like the kids at the dinner table all on their cellphone, I want to say no.

Denise Pope (27:49):

And adults and adults.

Dan Schwartz (27:51):

Yeah. Yeah. No adults-

Denise Pope (27:52):

It's bad. It's bad. Okay. Wait. So Nilam-

Dan Schwartz (27:56):

Sorry. Sorry. I had to get it out.

Denise Pope (27:57):

Okay. That's good, Dan. That's good. I'm glad. Do you feel better? That's good.

Dan Schwartz (28:02):

I do feel better.

Denise Pope (28:02):

So, Nilam, have you been shocked by anything in that sense?

Nilam Ram (28:07):

To this particular point, I would say that the adults are also doing many of these same types of behaviors, surprisingly more than we give ourselves credit for doing those types of both good and bad behaviors on the phone.

Dan Schwartz (28:18):

Sorry, what did you say? I was just checking my e-mail.

Denise Pope (28:21):

Yeah, yeah, yeah. No, adults are poor role models when it comes to this, a hundred percent. The kids are the first to point that out.

Nilam Ram (28:28):

Yeah, and rightfully so. I think the surprising thing for me is that just the sheer range of media world, the digital world is humongous and there's so many different parts and places of it that are interesting and contain all kinds of great information about so many things, way more than even exist in our current physical universe. And so being able to see that heterogeneity and the very creative ways in which people are engaged with all of that heterogeneity that is relevant to their own personal lives in meaningful ways, that's been what's surprising to me.

Denise Pope (29:10):

That's a nice way to end.

Dan Schwartz (29:12):

Yeah, that's really cool.

Denise Pope (29:12):

That's a really nice way to end. So okay, so Dan, I'm going to put you on the spot as I love to do. What are some takeaways from this episode that you're walking away with?

Dan Schwartz (29:21):

Boy, I think the theme that came through over and over in my head was there's a lot of variability in the way people use these devices. And so it's going to be really hard to say good or bad. So one person may use it style one, another person uses it style one. For the first person, style one was really good. For the second person, it was really bad. And so it's going to be really complicated to find different pathways through the media world we're in to be able to make recommendations.

Denise Pope (29:56):

Yeah. We're looking to you as the future hope and help for all of us folks out there who are struggling with, you know, how can we emphasize the good Nilam.

Dan Schwartz (30:07):

This was great. This was great.

Denise Pope (30:08):

Thank you so much for being here. And thank all of you listeners for joining this episode of Schools In. Be sure to subscribe to the show on Spotify, Apple podcasts, or wherever you tune in.

(30:18):

I'm Denise Pope.

Dan Schwartz (30:20):

Sorry, I was checking the newsfeed. I'm Dan.

Podcast

split

GSE area

Faculty and Research

Chatting about chatbots: How AI tools can support teachers

Olivia Peterkin — Mon, 04 Nov 2024 19:08:58 +0000

Chatting about chatbots: How AI tools can support teachers Olivia Peterkin Mon, 11/04/2024 - 11:08

Data Science | Innovation | Teaching

On this episode of School’s In �� GSE Assistant Professor Dora Demszky discusses how chatbots can be used to give teachers feedback.

November 7, 2024

By Olivia Peterkin

While much has been said about the potential positive and negative effects of generative artificial intelligence (AI) in education as it relates to students, less has been said about how AI tools can be used to support teachers.

�� Graduate School of Education (GSE) Assistant Professor Dora Demszky, whose research combines machine learning, natural language processing, linguistics, and input from educators, is currently working on a project called M-Powering Teachers that provides feedback for teachers in the classroom.

“It’s really rooted in the idea that we want to empower teachers,” said Demszky, who teaches education data science at the GSE. “We’re not trying to tell them what to do. We’re just providing them with opportunities to reflect on what they did.”

The M-Power tool (the m stands for machine) utilizes natural language processing to analyze verbal classroom interactions and provides formative feedback to teachers.

“A lot of the feedback is actually more just providing them with things that they did, highlighting things and moments in their lesson for them to reflect on and asking them good reflection questions and goal-setting questions so there’s less opportunity for risks or for error,” she said.

Demzsky joins hosts GSE Dean Dan Schwartz and Senior Lecturer Denise Pope on School’s In as they discuss artificial intelligence as a tool for positive feedback and support for educators. Her research focuses on developing natural language processing methods to support equitable and student-centered instruction.

In the episode she explains how her team is trying to identify practices like cultivating growth mindset, using supportive language, and building on student ideas as focal points for teacher feedback and professional learning.

“We know from the literature — like, decades of literature — that when students feel heard, when they feel that their ideas matter and that their teachers are building on it rather than just funneling them to a very specific answer, that really facilitates learning,” says Demszky. “So we identify practices that are related to that, like building on ideas, mindset-supportive talk, asking questions that probe students’ thinking, and then we build algorithms.”

Never miss an episode! Subscribe to School’s In on Spotify, Apple Podcasts, or wherever you get your podcasts.

Transcript

Dora Demszky (00:00):

Because of these technical limitations of these algorithms and our current philosophical approach, we are focusing on the good only.

Denise Pope (00:09):

Today, we're discussing a very hot topic, one that we will revisit this season again and again and again because it is just that hot, AI in education. More specifically, we'll be talking about how artificial intelligence tools can improve feedback for teachers. We're always, as educators, looking to get more feedback to improve our practice, right Dan?

Dan Schwartz (00:31):

I don't know, Denise. Maybe it's just you who wants that feedback. I think I'm pretty good.

Denise Pope (00:36):

All right, Dan. Well, let's get into our episode and find out.

(00:42):

Welcome to School's In, your go-to podcast for cutting edge insights in learning. Each episode, we dive into the latest trends, innovations, and challenges facing learners. I'm Denise Pope, senior lecturer at �� Graduate School of Education and co-founder of Challenge Success. And I'm with my co-host, Dan Schwartz, dean of the �� GSE and faculty director of the �� Accelerator for Learning.

Dan Schwartz (01:11):

We're very fortunate today. We have someone who's state-of-the-art thinking about how to support teachers, how to get them feedback. This is Professor Dora Demszky at the �� Graduate School of Education. She combines machine learning, natural language processing, linguistics and input from educators. And one of the projects I want to talk with her about is how can chatbots, basically the computer, provide actionable feedback to teachers? Give me the model. I'm a teacher and I'm in the middle of teaching a class and suddenly in my earpiece it says, do this instead. What's our vision?

Denise Pope (01:52):

What? Oh my God, that's crazy.

Dan Schwartz (01:54):

Well, I just made that up. I have no idea if this is what-

Denise Pope (01:57):

Oh, oh, okay.

Dora Demszky (02:00):

Yeah, yeah, I can clarify that. So it depends on the setting, right? So are we talking about physical classrooms or face-to-face instruction where oftentimes yes, giving them real-time feedback can be really distracting. So one component is about the what, like what are you actually telling them to do? It has to be good feedback. The other question is the when and the how, which are just as important. So in the context that we've deployed it, it's been oftentimes online context with novice tutors or teachers and they get feedback after they taught. So you teach your lesson, your lesson gets recorded, which is pretty simple in online context. Then we run the analyses on your lesson and then you get some insights on it.

(02:47):

In physical classrooms, it's pretty much the same thing. The difference is that you upload, you use a device to record your lesson, and then you upload that to your app or something and then you get feedback afterwards. We have thought about real time, but as you say, it can be very distracting.

Denise Pope (03:05):

How do you know that the quality of the feedback that the bot is giving, is it called the bot, I don't know, is good? Are you checking that? Are you behind that? Is it Dora really giving the feedback?

Dora Demszky (03:18):

Great question. So we call this Empowering Teachers. That's the name of the tool that we built. It's really rooted in the idea that we want to empower teachers. We're not trying to tell them what to do. We're just providing them with opportunities to reflect on what they did. So kind of like a Fitbit, when you run and it gives you some metrics, it doesn't necessarily tell you, oh, you should burn this many calories a day, or you should run this fast. It's so person dependent and so context dependent that you are the only person that knows.

(03:51):

So that may answer some of their question about how do we know it's good? So we first obviously validate it in different ways, validate it by having teachers, experts look over what our models are predicting and what are their outputting, do they seem right or seem wrong? But also, a lot of the feedback is actually more just providing them with things that they did, highlighting things and moments in their lesson for them to reflect on and asking them good reflection questions and goal setting questions so there's less opportunity for risks or for error.

Dan Schwartz (04:32):

Let's take the Zoom example. It's online. I've got three or four students that I'm talking to. Afterwards, it'll come back and it'll say, you said to that student they should try harder. How do you think that student felt about that? I mean, what's an example of some feedback?

Dora Demszky (04:51):

So you mentioned, what you just brought up is something that has to do with growth mindset, which is something we worked on you know, "try harder". The idea that intelligence is not something that's fixed, that you can improve your brain and your skills can constantly developing if you are working on it. But just telling them to try harder may not be the best way to do that, but actually recognizing that things are challenging and giving them the tools and strategies to improve their skills. Anyways, that's a tangent, but we are trying to identify these types of practices like growth mindset, supportive language, or building on a student's idea. So we know from the literature, like decades of literature that when students feel heard, when they feel that their ideas matter and that their teachers are building on it rather than just funneling them to a very specific answer, that really facilitates learning.

(05:47):

It creates that type of a collaborative process that you mentioned at the beginning that it's not just the teacher lecturing and the students passively receiving things, but instead, students are active participants. So we identify practices that are related to that like building on ideas, mindset-supportive talk, asking questions that probe students thinking, and then we build algorithms. It's not really a chatbot. These are algorithms that can identify moments when you do that and then highlight them for you to revisit, to think about, what did I do in this moment when that led to this great example, and could I do this more? Where are there missed opportunities?

Dan Schwartz (06:36):

Is the approach generally to find good examples and tell the teacher what you did here was good, as opposed to saying, don't ever do that again?

Dora Demszky (06:48):

Exactly. There are multiple reasons for it. Just pedagogically speaking, it's better to celebrate the good, and also especially for novices. So a lot of the contexts we're working, we are working with novice instructors. For them, it's really important to give them the positive feedback early on so that they can build on it and grow and not feel like, oh, I'm just doing everything wrong.

(07:12):

Second of all, it's actually extremely hard to reliably be able to tell this was wrong because as I mentioned, everything is so context-dependent. Maybe your model cannot actually identify the teacher's true intention with let's say not building on a student's idea at the moment. Maybe they were just trying to get ideas from everybody before then synthesizing it. There are different contexts where we may not necessarily want to directly build on what the student said right away, and that's just really hard and complex to algorithmically detect.

Dan Schwartz (07:53):

There's a study that occurred, I don't know, 25 years ago, and it was about clinical psychologists, but what they discovered was clinical psychologists who'd been doing it for 20 years were no better than clinical psychologists in their first year of practice.

Denise Pope (08:10):

Oh. That's not good at all.

Dan Schwartz (08:12):

Right. They weren't getting very good feedback. You give a diagnosis or a prescription to your patient, they never come back, and so they don't know whether it works or not. So do you think the same thing's true with teachers? I've been teaching for 20 years. Have I just gotten better and better or is that just too hard of a task to improve massively?

Denise Pope (08:36):

I mean, I think it completely depends on the context. If you are getting zero feedback and you're just doing the same thing, and a lot of people who haven't had really good training often teach the way that they were taught, and we know that that's kind of antiquated at this point. People believe that talking and lecturing is the best method, which we know is not true. And kids sitting in rows, spitting back answers. Don't act surprised, Dan, I know you know this.

(09:05):

So I would say it really depends, and I think what it depends on is feedback. And if you think of a clinician maybe in the office with just a patient, I could see how that would be hard because nobody's watching them. Nobody is following up with the patient. And in some sense, that's kind of like teachers because alone. Usually they're a single adult in the classroom, and so it might be hard to know what you're doing well and what you're not doing well. So I think it depends is my answer.

Dan Schwartz (09:35):

Okay, so that's a nice hedge. I think another thing for teachers is they can experiment with curriculum so they can create conditions that help them get feedback. So when I teach my courses in college, let's say I teach intro stats, it's a pretty established curriculum, but every couple of weeks, I'll try something brand new and see how it works out.

Denise Pope (10:01):

Well, I mean actually what you just said is very true for when I was teaching high school, my third period class always went better than my first period class because with a new lesson, you're running the lesson for the first time and then you're like, okay, next time do this a little differently. By the time you get to sixth period, you're like, we got this down.

Dan Schwartz (10:21):

Really by the time I get to sixth period, I'm bored.

Denise Pope (10:23):

Okay, well, you're also already exhausted.

Dan Schwartz (10:25):

I've mastered it.

Denise Pope (10:27):

You do forget what you've said, how many times you've said it, right? You do forget that.

(10:37):

So okay, I'm not that techie. So I'm going to ask a question that might be kind of embarrassing, but if someone is programming, it's not a bot, whoever or whatever is giving the feedback, right? So how does this thing that's giving the feedback know that, oh, that's academic press, pushing someone to ask or build on a question. Oh, that's growth mindset. How is this all working, without getting too techie, without getting too techie? Sorry.

Dora Demszky (11:05):

Of course. Yes. Our general approach is the traditional way you train machine learning models is you get training data. Our training data includes a bunch, thousands of transcripts of talk between students and teachers that then we label manually with experts, not all of them, but a subset of them for these particular moves or examples that we're interested in. And then we train the model on this data to learn to predict those same labels.

Denise Pope (11:41):

That's so cool. And how accurate is it, because I know you hear about hallucinations in AI and all this stuff, like how accurate is the model?

Dora Demszky (11:49):

For this type of classification, I would say it's moderately accurate. I want to stress something which that humans disagree a lot. These are very subjective things like these are not just like, oh, is this a cat or a dog? These are examples that are very context-dependent, subjective practices as I mentioned. And the agreement between two experts is also in the moderate range. So what we are finding is that our models are able to approximate that level of agreement that humans have between one another, but they're not perfect because yeah, it is subjective. So we're trying to mitigate that by obviously hedging and saying that this is not perfect, and also by coming out with the positive examples, which obviously have a lot lower risk for harm than providing the negative examples.

(12:39):

I do want to add though that I know from Dan your work about the importance of contrastive cases, teaching people by showing them contrast of, oh, this is good, and in contrast, this is not so good. So ultimately, it would be really great to incorporate something like that, but just because of these technical limitations of these algorithms and our current philosophical approach, we are focusing on the good only, but we hope to get there at some point.

Dan Schwartz (13:06):

Focusing on the good will work if you help people see which part of it was good. So you show me a three-minute video of my lesson and you say, that's really good, and I'm sort of like, which part? So some way just to get people to see what's the thing. So the reflection prompts are probably really important here.

Dora Demszky (13:26):

So we are very specific. We actually identify a single conversational turn with some limited context. This is what you said, showing them that built on a student's idea. Students said, I added 30 to 70, and the teacher asked, where did the 70 come from? This is just a really simple example, but of building on and improving the student's thinking. So we give them these examples and then ask them, what were some strategies that you used in this context to build on a student's contribution and what else could you do next time? And which one of these strategies you might want to use again next time, thinking about your next lesson topic, for example, like ratios, making it up, it's depending on the context.

(14:12):

A lot of what we work in is math or computer science, like STEM learning environments. And then we give them resources that can facilitate these reflections. Here are some examples, for example, from other instructors. In the case of this online course, there were several instructors teaching the same curriculum the same period of time. So we could actually give them examples from their transcript, de-identified obviously that were really great examples that they can incorporate in their own reflection, like maybe I want to use a question like this next time.

Dan Schwartz (14:47):

So you're also matching across different instructors that allows you to do a lot.

Dora Demszky (14:52):

And they can share their reflection with others, so that's another motivation. Pro-social motivation for them to reflect is not just self-serving, but they can actually opt in to share that with others and see what others are reflecting. So it's kind of a shared community for professional learning among instructors.

Dan Schwartz (15:16):

That's great. So could you do this for students? I'm sort of imagining student groups.

Denise Pope (15:21):

Students in a study group. Often it's like the blind leading the blind, but if they had this extra help, this machine that was saying, hey, you're on the right track, keep doing, I mean, it becomes sort of like another teacher.

Dora Demszky (15:36):

Yeah. So this is something we're working on thinking about, especially focused on a specific population of students, multilingual learners who we know are often underserved, especially in a monolingual teaching environment where they might not have the vocabulary, but not just the vocabulary, just like they might have been taught about different concepts in different ways. And currently, it's really hard for a single teacher to support all of them and their separate language backgrounds and needs. It would be really amazing if these AI technologies, which we know are supposedly amazing for different languages and dialects, still to be tested in this particular context, but it has a lot of potential for doing exactly that. Like you said, kind of a collaborative assistant in maybe a group discussion.

(16:24):

So my work, a lot of it takes the stance of always having the teacher in the loop is because of multiple reasons. One, because the technology is not entirely reliable and it poses some risks if you directly interface it with minors and kids. And two, because I really worry about the inequities it could create to replace a teacher with an AI bot and then certain kids just not ever having access to a human teacher, so I just worry about the incentives that creates. But I think there could be different structures where an AI bot like that could really facilitate group work or peer work or even just a single student working on the problem through the teacher or even by themselves, but with some type of a supervision.

Dan Schwartz (17:17):

So I could imagine having a unit on cooperating right, and so you have three kids working together and there's an after action review where the AI sort of says this bit where you said, I like your idea, how about if we do this was really good. I don't feel like that's chasing the teacher out of the room. It seems like it could be super helpful to kids who have a lot of trouble giving feedback. And what do you think, Dora?

Dora Demszky (17:49):

Yeah, I think that specifically focused on these collaborative norms, it could be super, super helpful. There's something about that a group, like CU Boulder has actually created, it's called COBE. It currently, I don't think it intervenes in the conversation, but it listens and then identifies these moments of collaboration when students engage with each other's ideas or when they fail to do so. And then those insights are surfaced to the teacher and back to the students.

Dan Schwartz (18:20):

Is there anybody who could do that for my home life?

Denise Pope (18:26):

What you just said to this person who you care a lot about was really good. Say more of that. And then you see the person in the background going, hey, bot, tell Dan to do the dishes. Tell him that's good. Doing the dishes is good. Look, Dora, can I just say one of the coolest things about this I think, because teaching is a really lonely profession. You talk to teachers all across the US, it's very rare that they get supervised, that they get any kind of feedback that someone's actually in their classroom for more than 15 minutes with a sort of high level like, hey, good job. Keep it up. Right?

(19:02):

And when you tell teachers that you want them to go and look in other people's classrooms and take the time and actually build in time for teachers to watch teachers, it's a little bit awkward. They don't know how to give feedback to their peers. It's not part of that culture. So you've kind of taken the ouch out a little bit and you've also allowed more camaraderie to happen.

Dan Schwartz (19:28):

That's an interesting question. So these automated feedback systems in medicine, the doctors didn't want them.

Denise Pope (19:33):

Yeah.

Dan Schwartz (19:35):

So Dora, when teachers see this are they like thank you so much, or are they sort of like, sorry, I do what I do, I don't appreciate it. What is the response to this?

Dora Demszky (19:48):

I would say just like with every tool, everything, there's variation. So they're going to be the power user. So I want this for everything. Can you add this feature and that feature and can look at my historical data and can it give me suggestions for X and Y and so many things? And there are people, I think honestly we haven't really heard teachers saying I don't want feedback on my teaching. What we hear though is when they disagree with something or when they find something to be inaccurate, and that oftentimes comes from inaccurate transcription, which is a bottleneck that we are facing and we're working on addressing. It's really hard to transcribe speech accurately, especially in a noisy classroom environment, and there are going to be mistakes. And then it propagates down to the feedback.

(20:40):

But yes, even if you take all of that into account, just like I don't use a Fitbit when I run, I just like that to be my thing. I like when I'm not measured by anybody. I think that it's normal that some people want more or less of this. And so I don't think a teacher would say I don't want this ever because I don't think that's a good attitude towards professional learning, but maybe not as often as some others.

Denise Pope (21:11):

I love it. I'm so excited about it. So I mean, I can imagine someone listening saying, how can we get this? Coming to classrooms soon. Are we a year out from everybody using it? And then we'll wrap this.

Dora Demszky (21:25):

So good question. We have a beta version of the tool that is already public. It's not currently ready for wide release, but anyone can sign up on the wait list. We have a wait list. If you Google my name and you can find this project page Empowering Teachers, and you can sign up to the wait list there. As a researcher or a teacher or educator in any capacity, this tool could help you. There are also commercial tools out there like TeachFX, which pretty much does what I described to you already, and they have thousands of users across the country and abroad. And they incorporate many of the models that we have developed, so they're really great.

Dan Schwartz (22:12):

Dora, thank you so much. It's great to see into the future and what's possible.

Denise Pope (22:18):

I know, I know. It's super, super exciting. Thank you. Thank you. Especially the information on where we can start to access some of these tools as educators. I can't believe they're already here, right? I mean, it's not something that's out in the future. Okay. Dan, I wonder, we started this out saying that you didn't really want feedback, and now I want to know, do you think you can use some of this feedback now that you know all the things that AI can do?

Dan Schwartz (22:45):

You mean about all the things I'm doing right? I'd love it, especially if it gave me points for everything I got right, it gave me a couple of points.

Denise Pope (22:53):

Yes, because everything is a competition for you, Dan, so you are motivated by the points. I will say this, I do actually like that. It's almost like caught you doing good, one of those exercises that we do with the kids. It's meant to really shine a light on the good and to find out where we as educators are doing right, and to highlight it in a way that gives you a chance to grow. So in that sense, Dan, you're not that far off. I mean, I think you're right. It's very positive and it would have a lot to tell you and others about what they're doing well.

Dan Schwartz (23:30):

No, I like that. It's hard to know, and teachers do want the feedback, especially if it's sort of saying, this is the right behavior, keep it up. And I like the idea that eventually, this tool could be used to help students get feedback about what they're doing right. I think that's really an exciting, exciting possibility.

Denise Pope (23:49):

Totally. Totally. A hundred percent agree. Really, really good stuff. Thank you again Dora for this great conversation, and thank all of you for joining us on this episode of School's In. Remember to subscribe to our show on Spotify, Apple Podcasts, or wherever you tune in. I'm Denise Pope.

Dan Schwartz (24:05):

And I'm Dan Schwartz. Denise, did I get that right? Can I get some feedback, some points for that?

Podcast

split

Faculty mentioned in this article: Dan Schwartz , Denise Pope , Dora Demszky

�� initiative reveals racial, gender, other disparities in Brazilian schools

Carrie Spector — Tue, 29 Oct 2024 17:36:40 +0000

�� initiative reveals racial, gender, other disparities in Brazilian schools

Carrie Spector Tue, 10/29/2024 - 10:36

�� researchers are adapting digital reading and math assessments for use in Brazil, to provide teachers with more timely feedback on learning differences. The research is part of an initiative to identify inequities in Brazilian K-12 schools. (Photo: Diego Lima)

Data Science | Education Policy | International Education

A team led by �� education professor Guilherme Lichand surveys students and educators in Brazil, identifying inequities in the school system.

October 31, 2024

By Carrie Spector

Earlier this year, when the Brazilian government reported its official data on the number of K-12 students nationwide with disabilities or learning disorders, �� education scholar Guilherme Lichand thought the figure seemed off.

According to data from Brazil’s annual school census, less than 4 percent of K-12 students have some type of physical or learning disability. But Lichand expected the actual number would be much higher, given U.S. rates and the international average.

Lichand, an assistant professor at �� Graduate School of Education (GSE) who is originally from Brazil, runs an initiative called Equidade.info (Portuguese for equity), which collects data on K-12 schools in Brazil through interviews with students, teachers, and administrators across the country.

Spurred by the dubious census figures, the team recently investigated the prevalence of physical and learning disabilities in the country's K-12 schools – arriving at a rate of at least 12.8 percent, more than three times the official statistic.

“There are differences in socioeconomic backgrounds that make it less likely for some kids to get a medical diagnosis,” said Lichand, who also co-directs the �� Lemann Center, a center housed at the GSE that supports Brazilian scholars at �� and initiatives to improve the Brazilian educational system. “But the government data is what determines how school resources are allocated. If you’re a policymaker making decisions based on incomplete data, kids are not necessarily going to get the support they need.”

Equidade.info, a project of the �� Lemann Center, analyzes data from a representative sample of schools throughout Brazil on a variety of issues, from chronic absenteeism and reading proficiency to racial relations and students’ sense of belonging.

GSE Assistant Professor Guilherme Lichand

Some of the initiative’s findings so far:

About a quarter of K-12 students report being a victim of bullying during the previous 12 months. Females report higher rates than their male peers, and non-white students report higher rates than white respondents.
Students’ sense of feeling welcome at school decreases as they progress from elementary to high school, with Black students reporting lower rates overall than their white classmates.
The prevalence of child labor among the student population is eight times what official statistics indicate.
Internet speed in schools is much lower than official data indicates, with a significant disparity between public and private schools.
More than half of teachers report witnessing cases of racial discrimination in schools – data cited in a recent push by Brazilian legislators urging the Ministry of Education to address racism in the country’s educational system.

In the country’s annual school census, information on many key markers of inequality is absent or incomplete, said Lichand, who is also a faculty affiliate of the �� Center on Early Childhood and the �� King Center on Global Development, and a fellow at �� Impact Labs and the �� Institute for Advancing Just Societies.

“Without precise information about differences along dimensions like race, gender, and disability status, education policies often magnify inequalities instead of alleviating them,” he said. “We need better data to design more equitable policies. So we established a different system for collecting it.”

Marta Vitória, an enumerator for the state of Sergipe, conducts a survey with a student for Equidade.info.

What the school census doesn’t capture

In Brazil, most of the Ministry of Education’s information on the state of K-12 schools comes from the annual school census, which solicits information on all 49 million students at 160,000 schools nationwide. Census responses are supplied by school staff, ideally but not necessarily drawn from information provided by students and their families.

“The only way you can get that much student-level data is to sacrifice quality in the way that you collect the data,” said Lichand. “One can only hope that what the schools are providing about these 49 million students is accurate.”

Given the lack of oversight for the process, Lichand questions the reliability of the data, especially where factors like race and disability status are concerned. These are complicated constructs by any measure, he said, particularly if the designation isn’t self-reported but determined by a school official or other figure.

With race, for example, Lichand said Equidade.info has found a discrepancy for 30 percent of students between what they report their race to be and what the school reports. Schools also have the option of not reporting students’ race in the first place, an option that’s prevalent in the data, Lichand noted.

“Until 2014, there was no information on race for half of the students in the country,” he said. “More recently, the government has made efforts to have schools provide this information, but still at least a quarter of students are missing race data.”

Lichand’s data collection model for Equidade.info goes deeper than the school census with in-person surveys of a representative sample of more than 200 schools from all of Brazil’s 27 states. The sample reflects key characteristics of the school population nationwide – urban and rural schools, public and private, technical and academic – serving students of all backgrounds, including indigenous populations. The researchers use statistical methods to ensure the results mirror the universe of K-12 students, teachers, and schools.

College undergraduates in each state are recruited and trained to serve as enumerators, visiting local K-12 schools every two months to conduct tablet-based surveys or run tests.

“Our enumerators can collect data that the school census doesn’t capture,” Lichand said. “They can ask students to self-identify their race. They can have students do tasks that indicate certain abilities, like reading fluency and comprehension. They can measure the school’s internet speed objectively, by connecting to the wifi network and running a speed test app.”

Because the enumerators visit the schools every two months, the surveys can include questions related to issues of the moment, like a mass school shooting or the wildfires currently raging in Brazil. “We can ask principals how many days of school they’ve lost to climate events, what kind of adaptation measures they have in place,” said Lichand. “We can be really responsive to what’s going on, and generate timely data that’s useful for policy and the public debate.”

The initiative’s website provides dashboards to visualize key insights from the data, particularly around gender, racial, and regional disparities, and all of the information is shared with the Ministry of Education.

Timely feedback on learning differences

Lichand is also working with other GSE researchers to adapt new U.S. digital reading and math assessments for use in Brazil, not only to support research but also to provide teachers with timely, personalized feedback on learning differences so they can more quickly support students who fall behind.

A Nov. 8 conference at the GSE will explore the potential of using the two ��-developed technologies in Brazil: the Rapid Online Assessment of Reading (ROAR), led by Associate Professor Jason Yeatman, and the �� Mental Arithmetic Response Time Evaluation (SMARTE), led by Professor Bruce McCandliss.

Lichand, an educational economist and entrepreneur, hopes to replicate the model he created for Brazil with Equidade.info for other low- and middle-income countries in the Global South, especially in Latin America and Africa.

“We know that high-quality data is in demand,” said Lichand, who is frequently featured in Brazilian media discussing Equidade.info’s findings. “As researchers, the nature of our work is to put knowledge out there and hope that it makes its way to change policy. Now we are starting to work closely with officials to be sure we can meet their specific data needs, creating pathways to support equitable policies.”

Júllia Kessia (in forefront), an enumerator for the state of Alagoas, poses for a photo with a group of young students she surveyed for Equidade.info.

Research Stories

split

GSE area

normal

Faculty mentioned in this article: Guilherme Lichand

A data-centered approach to education AI

Carrie Spector — Mon, 25 Mar 2024 17:17:27 +0000

A data-centered approach to education AI

Carrie Spector Mon, 03/25/2024 - 10:17

GSE scholars are working to engage teachers and students in developing ethical, inclusive data practices for AI in education. (Photo: iStock / Gorodenkoff)

Data Science | Innovation | Technology

Useful models need better data. �� scholars explore a path to ensure domain experts have more fruitful conversation.

March 1, 2024

By Dylan Walsh

The promise of machine learning in education is tempered by the standard concerns: biased data, privacy violations, inequitable outcomes, and so on. There remains a significant gap between how the technology is imagined and how, in fact, it is deployed and used.

“We are pioneering a participatory AI approach with the goal of developing ethical, human-centered, and equitable AI solutions for education,” says �� Graduate School of Education (GSE) Assistant Professor Hariharan Subramonyam. “By engaging teachers and students from the outset, our goal is to ensure that our data practices are inclusive and representative of diverse learner experiences and needs.”

Subramonyam explored how gathering different stakeholders at the same table can help address these needs with Mei Tan, a doctoral student at the GSE; Hansol Lee, a PhD candidate in Education Data Science at the GSE; and Northeastern University Associate Professor Dakuo Wang. Working in the world of education, the researchers convened 10 meetings in which engineers, designers, legal specialists, teachers, and students discussed the training data specifications for four different machine learning tools. (These included measuring student engagement through images, career recommendations based on resumes, assessing drop-out risk, and automated essay grading.) In a new preprint paper, the researchers distill the results of this process into a framework that ensures the right mix of people are together and able to have productive conversations.

Subramonyam and Tan spoke with �� HAI to outline how their findings inform a more conscientious and effective method for designing machine learning algorithms in education and beyond.

I want to start with a contextual question: Can you talk about the difference between model-centric and data-centric AI practices?

Subramonyam: For a long time, we assumed that data quality basically doesn’t matter for these models, as long as you have a lot of it. Machine learning engineers and AI researchers would take data as a given and focus mostly on fine-tuning and improving the model. That approach hit a ceiling in recent years, and people started to realize that improvements in model performance required thinking about data quality. It’s also expensive to collect lots of data. So now we’re thinking about how to gather less data that’s higher quality.

In your recent education work, you look at a multi-stakeholder collaborative process that considered what data should be collected. What prompted this, and how did you think about who should be involved?

Subramonyam: Issues around data transparency — how it has been collected and labeled and so on — have been around for a while, but this all happens after the fact: The data is already collected, the model is already built. To me, that has always seemed like the warning label on cigarette packets. It’s an acknowledgment of something bad. In my lab, in the Graduate School of Education, I am interested in how we can be more proactive about addressing these downstream problems.

Tan: In the world of AI and education, there's a good amount of research stating the obvious, which is that a lot of AI tools don't serve the needs of practitioners. And there's a lot of worry about the harms that poorly built AI tools can have in education, with many of these harms residing in the quality of the data.

Given this, we wanted to take a front-end, data-centered approach where we involved domain experts, the people who can right that wrong of how we represent the world of education through data. When we thought about whom to involve, we obviously started with teachers and students. From the industry perspective, we were thinking about machine learning engineers, but also UX designers and legal specialists, given the privacy concerns around the treatment of minors.

In thinking further about this data and this framework of participatory design, the model that we'd like to advocate to the world is not one where the researchers or the developers of ML products define the domain experts. Rather, we should ask the community: Who do you think are the valuable stakeholders in this case? The teachers, for instance, often raised administrators and school counselors as people who could make important contributions.

These group dynamics aren’t simple. What are the challenges of getting a team like this to work together?

Tan: I want to emphasize that what we’ve done here is a proof of concept, because no one has put a group like this together before. What we watched unfold was a bit of an experiment.

It ended up that the machine learning engineers had to translate what a decision about data collection might mean for the model and, ultimately, for the end-user. It required a very experienced machine learning engineer to be willing and able to engage in that dialogue with a teacher or a student. We saw some do it well and others do it less well.

The same was true with teachers. There were some teachers who were very willing to engage with the decisions and technical details in front of them, and there were others for whom that burden of entry was too high.

We learned that for this process to work, we need scaffolding for both sides. We need to help teachers understand some of the basic parameters of a machine learning model and how the data that’s collected fits into the overall picture. And we need to help ML engineers do that work of translation and understand more about the domain they’re working in — education, in this case. There is often emphasis on this first step: helping non-technical experts understand the technical. There is much less focus and work helping technical experts understand the domain. We need to do a better job of that.

What are some of the problems that result from bad data, and what promise did you see in this approach?

Tan: There are all kinds of harms that can result from bad or biased data. To take just two examples, there’s research about adaptive learning systems that shows racial and gender biases in the way these systems evaluate a student’s understanding. There is also evidence that technology used to assess engagement based on an image of a student’s face is full of bias depending on the characteristics of the student’s appearance.

In our work, we saw how domain experts shape key variables that go into data collection and, that way, alleviate some of these concerns or errors. Representation is one example. When we currently think about representation in developing machine learning data sets, we think about demographic variables, whether there is equity across gender, race, and socioeconomic lines. But the domain experts had much broader ideas of representation when it comes to education. So, for example, is this a private or public school? Is this a small classroom or large lecture? What are the individual learning needs? Is there neurodiversity? What subject are we talking about? Is this a group activity? All of these play a massive role in the classroom and can dramatically affect a student's experience — and therefore the act of data collection and the models that you build.

Beyond education, what did this show about how to think through who's involved in a process like this and what kind of scaffolding can help it succeed?

Subramonyam: There are a few things we need to think about. The first is the incentive structure. In industry, product teams are created with efficiency in mind, which means they’re divided up to work in ways where machine learning engineers are incentivized differently from people in finance, and where workflows favor separation. Our first recommendation is that we need to bring all of these people together at the beginning of the decision process. It takes enormous effort to collect data and train models, so it’s best to get thing right at the outset.

The second is around infrastructure. We need to think about the kinds of tools that we need in order to support collaboration. One of my students built a visualization related to these ML models to support collaboration between data scientists and domain experts, where the visualization offloads some of the burden from the engineer who typically has to translate things for other people.

Tan: Another thing we need to think about is the fact that language and standards are not shared across these domains. Teachers want to see evaluation metrics about how much a student is learning. This is very common language among teachers, but it’s very hard for ML engineers to define in an equation. Meanwhile, engineers care about things like precision in the model, which means nothing to teachers. We need to establish groundwork so these different stakeholders can talk about the same thing without a bunch of roundabout discussions.

And, finally, I’d mention the need for continuous iteration. In this case, we brought everyone together for one moment in time, whereas the process of developing an ML model can go on for months. As things change, these stakeholders need to be brought back together to recalibrate the direction that things are moving. Our research shows the benefits of upstream collaboration, but more work is needed to understand how to sustain continuous engagement.

This research was funded by the �� McCoy Family Center for Ethics in Society. Dakuo Wang was a visiting researcher at HAI, supported by IBM Research.

This story was originally published by �� HAI.

Research Stories

daps

split

GSE area

normal

DAPS

Faculty mentioned in this article: Hariharan Subramonyam

‘Harnessing the power of data’: GSE’s Education Data Science program hosts inaugural symposium

Olivia Peterkin — Wed, 21 Jun 2023 20:05:08 +0000

‘Harnessing the power of data’: GSE’s Education Data Science program hosts inaugural symposium

Olivia Peterkin Wed, 06/21/2023 - 13:05

The first cohort of graduates from the EDS master's program (from left): Othman Bensouda Koraichi, Xunyi Gao, Sibei Zhang, Raymond Zhang, Iris Zhong, Ian Bott, Lucy Caffrey-Maffei, Priscilla Zhao, Laura Hinton, Mei Tan, Anna-Julia Storch, and Elena Pittarokoili. (Students not pictured: Shengan Chen and Akshatha Kamath.) (Photo: Ryan Zhang)

Data Science | Events | Faculty and Programs

The first cohort of graduates from the new master’s program presented capstone projects that offered data-driven solutions to problems in education.

June 21, 2023

By Olivia Peterkin

Othman Bensouda Koraichi was working toward becoming a data scientist in Morocco when he applied to a new master’s program at �� Graduate School of Education (GSE), driven by some of the severe educational inequalities he saw in his home country.

“I grew up in Morocco with this idea that we have a small elite that is really educated, and that the rest of the country is essentially struggling,” Bensouda Koraichi said. “On a day-to-day basis, you would see kids begging for money in the street instead of going to school.”

As part of the inaugural cohort of the GSE’s Education Data Science (EDS) program, Bensouda Koraichi enrolled because he wanted to explore ways to use data to address the inequities he saw. “I think that, as a country, Morocco lacks people with the combined knowledge of data science and education, which is essential to make the right decisions,” he said. “I want to be a leader in that effort.”

His capstone project, which centered on how machine learning can be used to predict and hopefully prevent student dropouts, was one of a dozen presented by the cohort’s new graduates at the first-ever EDS Symposium on June 15 at the Center for Education Research at ��.

Othman Bensouda Koraichi, MS ’23, wants to use his degree in education data science to address educational inequities in his home country of Morocco. (Photo: Ryan Zhang)

Other students’ topics included racial imbalances between public school districts' attendance zones, why data equity matters in educational program evaluation, restorative classroom management practices, and family stresses during the pandemic.

Throughout their time at ��, students took courses at the GSE as well as technical courses across campus, said Sanne Smith, director of the EDS program and a lecturer at the GSE.

“They have immersed themselves in the topics close to their hearts, ranging from early childhood education to the use of edtech in the classroom,” she said. “All the while, they have been thinking about how to harness the power of data to work toward positive change in our educational landscapes.”

Initially meant to be an 18-month program, several EDS students successfully petitioned to add an additional quarter so they could spend more time researching their respective projects.

Elena Pittarokoili, another member of the EDS class of ’23, investigated whether getting personal development certificates — think wine tasting or building sustainable relationships — has a positive effect on employment. Turns out, based on her project’s findings, it does.

“I want people to use this information as encouragement to start learning about whatever they’re interested in, whether it’s career-related or not,” Pittarokoili said. “As we move into the ChatGPT era, it will be difficult for people to be authentic and different. I think being driven by curiosity will become more and more important.”

Moving forward, Pittarokoili — who often includes the fact that she trained as a ballerina on her resume — will be joining Age of Learning, a California-based edtech firm, as a data analytics engineer on its efficacy research team.

“We spent this whole year working on this project,” she said. “So being able to share both my results and excitement with others at the symposium was an amazing experience. This capstone isn’t just a project for me, it’s a part of who I am.”

Raymond Zhang, MS ’23, presents his capstone project on peer review policies. (Photo: Ryan Zhang)

Laura Hinton, MS ’23, presents on data equity in program evaluation. (Photo: Ryan Zhang)

Othman Bensouda Koraichi, MS ’23, shares his work on a dropout early warning system. (Photo: Ryan Zhang)

Elena Pittarokoili, MS ’23 (right), presents her capstone project on non-career-focused online credentials. (Photo: Ryan Zhang)

Ian Bott, MS ’23, presents on STEAM curriculum priorities in K-12 schools. (Photo: Ryan Zhang)

Anna-Julia Storch, MS ’23 (left), sits with family at the symposium. (Photo: Ryan Zhang)

GSE Dean Dan Schwartz joins the symposium. (Photo: Ryan Zhang)

From left: GSE doctoral student Hansol Lee; GSE Assistant Professor Dora Demskzy; Sanne Smith, director of the EDS program; GSE Professor Daniel McFarland; and Klint Kanopka, PhD ’23.

Twelve of the 14 members of the first EDS cohort. (Photo: Ryan Zhang)

GSE News

alumni

edcareers

split

GSE area

school_news

Alumni

EdCareers

Faculty mentioned in this article: Sanne Smith

What happened to students who left public schools during the pandemic? New research tracks their paths

Carrie Spector — Wed, 08 Feb 2023 20:03:14 +0000

What happened to students who left public schools during the pandemic? New research tracks their paths

Carrie Spector Wed, 02/08/2023 - 12:03

(Photo: maroke / Getty)

Data Science | Education Policy | K-12

�� education professor Thomas S. Dee says the findings point to a need to refocus academic recovery efforts on younger students.

February 8, 2023

By Carrie Spector

The number of students enrolled in the U.S. public school system has plummeted since 2020, with roughly 1.2 million K-12 students leaving its rolls. Where did they go?

Pundits have speculated that the drop could be attributed to families switching to private schools or homeschooling, and to population changes occurring since the start of the pandemic. New data, gathered through a collaboration between �� researchers and the Associated Press, finds that these explanations only account for about two-thirds of students who have left public schools since the fall of 2019.

Despite a significant and sustained rise in homeschooling and private school enrollment – especially in kindergarten and early elementary grades – more than a third of the enrollment drop remains unexplained.

“We saw a historically unprecedented exodus from public schools during the pandemic, especially among students at the kindergarten and early grade levels,” said Thomas S. Dee, an economist and professor at �� Graduate School of Education (GSE). “We’ve begun to see the financial pressure this is putting on many districts, with discussions about closing under-enrolled schools and possibly laying off teachers. But there hasn’t been much focus on where these students went and what kind of learning environments they were experiencing.”

Dee partnered with Big Local News, a project of the �� Computational Journalism Lab, and journalists at the Associated Press to collect and analyze data about the paths students took out of public schools during the pandemic. “Education reporting is a high priority for local news organizations, and understanding enrollment data is key,” said Justin Mayo, a senior data journalist with Big Local News, which works with reporters to gather and process data on a range of issues.

The findings, published on Feb. 8, are summarized in a policy brief for the nonprofit Urban Institute and available, aggregated by state along with more detailed breakdowns, through Big Local News.

“Before this research, we knew virtually nothing about where these kids went,” said Dee, who is the Barnett Family Professor of Education at �� and a senior fellow at the �� Institute for Economic Policy Research. “Now we have some really important evidence to explain a portion of that exodus. But the fact that so much of the loss cannot be explained raises new questions about what will help students get their education back on track.”

Where the kids went

For their analysis, the researchers gathered state-level data on school enrollment – public, private, and registered homeschooling – for three school years, from fall 2019 through spring 2022. They also used U.S. Census Bureau estimates to identify the school-age population in each state over the same time period.

Only 22 U.S. jurisdictions – 21 states and Washington, D.C. – were able to provide adequate data on non-public schooling options such as private school and registered homeschools, the researchers said. The data they were able to assemble includes more than half of the school-age population in the United States at the onset of the pandemic.

Data from the available jurisdictions are representative of the nation, Dee said, in that their public-school enrollment declines mirrored the changes observed nationally.

Using the available data, the researchers found that 14% of students who disenrolled from public school over the three-year period went to private schools, while 26% switched to homeschooling. Another 26% of the loss could simply be attributed to a declining school-age population.

Given that the data tracked students into the 2021-22 school year, the finding on homeschooling was particularly striking, said Dee. For every newly enrolled private school student, two additional students remained enrolled in homeschooling for the 2021-22 school year, when most traditional public schools had returned to in-person instruction.

“We saw a run-up in homeschooling in the fall of 2020, which wasn’t surprising when you think about where we were then, when a lot of schooling was remote,” said Dee. “But I was surprised by both the magnitude and how enduring that initial run-up was.”

Population estimates indicate that the United States experienced historically slow growth during the pandemic due to COVID mortality, low birth rates, and a decline in international migration. This period also saw families increasingly relocating to other states due to rising housing costs and the persistence of remote work arrangements, which in some cases contributed to the public school enrollment decline.

“The states with the largest declines in public school enrollment – California, New York, Illinois – also saw the largest exodus of school-age children,” said Dee. “So at some level the reduction in public school enrollment wasn’t just a flight from public schools, it was a flight from communities.”

The researchers said they were unable to include an analysis of the racial and ethnic breakdown of the enrollment decline because of a lack of data from states addressing these demographics in non-public schools.

Expanding recovery efforts

To account for the remaining 34% of unexplained enrollment losses, Dee suggested several explanations: an increase in truancy, growth in the prevalence of unregistered homeschooling, and an increase in the number of families skipping kindergarten. The researchers found the unexplained declines were concentrated in states that do not require kindergarten attendance.

With much of the national conversation about pandemic learning loss focused on test scores for students in fourth and eighth grade, the findings indicate the need to expand recovery efforts for younger students, Dee said.

“The federal testing data that came out last fall showed the largest declines ever observed in math and, to a lesser extent, reading, and that has rightly motivated attention to serving older kids in public school to try to remediate that learning loss,” said Dee. “But the enrollment data are telling us that kids who are experiencing the most dramatic learning disruptions are too young to be in those testing windows yet.”

Efforts touted to support academic recovery from the pandemic, such as extended classroom time and high-dose tutoring, are largely targeted toward students in public schools, especially in older grades where testing typically occurs. Dee said the findings suggest the broader need for early screening tools that can identify learning setbacks years before students reach the age for large-scale assessments, as well as increased oversight of truancy and homeschooled children’s learning experiences.

Research Stories

split

GSE area

normal

Faculty mentioned in this article: Thomas S. Dee

�� researchers partner with New York State to map the landscape of educational equity

Carrie Spector — Fri, 11 Nov 2022 19:57:52 +0000

�� researchers partner with New York State to map the landscape of educational equity

Carrie Spector Fri, 11/11/2022 - 11:57

A five-year collaboration aims to show how policies can be targeted to combat inequities in the school system. (Photo: Gorodenkoff / Shutterstock)

Data Science | Education Policy | Poverty and Inequality

The Educational Opportunity Project at �� and the New York State Education Department will study the impact of education policies.

November 14, 2022

The Educational Opportunity Project (EOP) at �� University and the New York State Education Department (NYSED) have launched a five-year project to map the landscape of equity across the New York State education system.

The aim of the project, in which researchers will leverage 15 years’ worth of data from NYSED, is to provide actionable information about how past and current education policies and practices impact equity in the school system.

The research team – including �� Graduate School of Education professors Sean Reardon, Ben Domingue, and Francis A. Pearman – will analyze student, teacher, school, and district data to develop a set of equity indicators that New York can use to monitor students’ access to educational opportunities within the education system. The team will analyze these indicators to understand whether inequities in access to opportunities – such as early childhood care and particular features of a school or neighborhood – are connected to disparities in students’ academic outcomes, including test scores and graduation rates.

“School systems have tremendous power to address disparities in educational opportunity through decisions they make on a regular basis,” said Reardon, the Professor of Poverty and Inequality in Education at �� and director of the EOP. “Our hope is that this project will show how system policies can be targeted to combat inequities, and that it will serve as a model for other partnerships between researchers and state education agencies.”

In addition to measuring and mapping out patterns of disparities among student outcomes and opportunities to learn, the researchers will assess the reliability and feasibility of using measures of such opportunities in the New York State data system to monitor educational equity. They will also provide evidence about specific features of the education system – including state-level policies and local strategic planning – that lead to greater equality in student outcomes.

Over the coming years, progress updates from this effort will be shared with, and feedback sought from, policymakers and education leaders throughout New York’s education system. The goals are to provide transparent information about educational equity in New York, and to inform state and local collaboration intended to address disparities in access to learning opportunities.

The project was presented at the New York State Board of Regents meeting on November 14, 2022.

The Educational Opportunity Project (EOP) Research Team

Core Researchers

Sean Reardon, Professor of Poverty and Inequality, �� Graduate School of Education
Erin Fahle, Research Scientist, NWEA
Andrew Ho, Professor of Education, Harvard Graduate School of Education
Ben Shear, Assistant Professor, University of Colorado Boulder

Collaborators

Ben Domingue, Associate Professor, �� Graduate School of Education
Francis A. Pearman, Assistant Professor, �� Graduate School of Education

Advisory Board

Susanna Loeb, Professor of Education and Director of the Annenberg Institute, Brown
Christopher Edley, Jr., Interim Dean, UC Berkeley School of Education
Rucker Johnson, Professor of Public Policy, UC Berkeley
Carrie Conaway, Senior Lecturer on Education, Harvard Graduate School of Education

NYSED Leadership Team

Allison Armour-Garb, Special Advisor to the Executive Deputy Commissioner
Rose LeRoy, Director of Educational Data and Research
Jason Harmon, Deputy Commissioner, P-12 Operational Support
Zach Warner, Assistant Commissioner, Office of State Assessment
Alexander Trikalinos, Executive Director, Office of Educator Quality and Professional Development

The EOP-NYSED partnership and research activities are supported by grants from the Institute of Education Sciences, the Spencer Foundation, and �� Impact Labs.

Research Stories

split

GSE area

normal

Faculty mentioned in this article: sean reardon , Benjamin Domingue , Francis Pearman

�� research reveals a hidden obstacle for women in academia

Carrie Spector — Tue, 14 Dec 2021 23:34:54 +0000

�� research reveals a hidden obstacle for women in academia

Carrie Spector Tue, 12/14/2021 - 15:34

Analyzing nearly 1 million doctoral dissertations, researchers found widespread bias against research that simply seems feminine, even if it’s not about women or gender specifically. (Credit: RocketClips, Inc./Shutterstock)

Data Science | Diversity and Identity | Higher Education

A study finds that women are penalized for pursuing research perceived to be “feminized” – a bias surprisingly strong in fields associated with women.

December 16, 2021

By Edmund L. Andrews

For more than a decade, women have earned more doctoral degrees than men in the United States. Despite that, women still lag behind men in getting tenure, getting published and reaching leadership positions in academia.

Much of the research into why that might be focuses on structural barriers and explicit prejudice. But a new study by a team of researchers at �� Graduate School of Education (GSE) finds a widespread implicit bias against academic work that simply seems feminine – even if it’s not about women or gender specifically.

Analyzing nearly 1 million doctoral dissertations from U.S. universities over a recent 40-year period, the researchers found that scholars who wrote about topics associated with women, or used methodologies associated with women, were less likely to go on to get senior faculty positions than those who did not.

The issue wasn’t so much a prejudice against feminist studies or gender studies, which have expanded considerably since the 1970s. In fact, people who wrote their dissertations explicitly about women had slightly better career prospects than those who wrote explicitly about men.

The real problem was a more subtle bias against topics and research designs that were “feminized,” meaning they were more associated with traditions of women’s work. Scholars whose dissertation abstracts had words like parenting, children or relationship, for example, had slimmer career prospects than people who used words like algorithm, efficiency or war.

Even within a particular field, whether sociology or computer science, scholars whose dissertations were associated with women’s traditions in research had poorer prospects than those who wrote more “masculinized” dissertations in their respective fields. Despite changes in social norms and a growing number of women scholars over time, the researchers found the devaluation of women's research was more or less consistent throughout the 40-year period.

“Everyone emphasizes that academia is based on meritocracy, that everything is neutral and based on the scientific value of research,” said the study’s lead author, Lanu Kim, who led the research team as a postdoctoral fellow at �� GSE and is now an assistant professor at the Korea Advanced Institute of Science and Technology. “It’s somewhat fake, and it’s somewhat impossible. There can be differences in men’s and women’s research interests, and some topics are already associated with women rather than men. The process cannot really be neutral.”

The study was recently released online in advance of its publication in the January 2022 issue of Research Policy.

Uncovering patterns through AI

The researchers used natural language processing, a type of artificial intelligence used to study patterns in text, to analyze the abstracts of dissertations in every field from universities across the United States between 1980 and 2010.

To measure how “feminized” or “masculinized” a dissertation might be, the researchers tallied the concentration of words that had been used disproportionately by male or female doctoral candidates in previous years. This included words explicitly referencing gender, such as woman, man, her or him.

Beyond that, however, the researchers looked for words associated with women’s or men’s interests, even if the words in themselves had nothing to do with gender.

Among the terms with a strong association to women: school, teacher, child, parent, culture and participation. Terms strongly associated with men, by contrast, ranged from algorithm and efficiency to words connected with energy and electronics.

The researchers then measured academic prospects by looking at which of the scholars went on to hold senior faculty positions. Specifically, they looked at whether a scholar was later named as the primary faculty advisor on someone else’s doctoral thesis, which is a strong indicator of an emerging scholar’s long-run success as an academic.

Though there are many other measures of success, Kim and her colleagues wanted to know whether academic institutions implicitly penalize scholars for certain types of research.

Overall, only 6.3 percent of those who received PhDs went on to become faculty advisors, but women were about 20 percent less likely than men to reach that mark.

Notably, scholars who wrote dissertations explicitly about women had a slight advantage over those who wrote explicitly about issues for men. That reflected efforts by many universities to make up for lost ground after years of giving short shrift to women’s issues.

Scholars had poorer prospects, however, if they pursued topics and research designs more implicitly associated with women: Their chances of becoming a faculty advisor were 12 percent lower than average. Perhaps even more startling, the implicit bias was actually greater in fields that had strong traditions of research associated with women’s work in the academy, such as sociology, than in fields dominated by men, like mechanical engineering.

For scholars working in fields with a preponderance of research traditionally associated with women, women PhDs are more likely to suffer a triple disadvantage on the job market, the authors wrote. “They are penalized for being women, [for] not doing a PhD in a masculinized field, and [for] not adopting man-type research practices.”

“The troubling inequity we identify is one that women faculty have likely long suspected but continue to experience,” said Daniel McFarland, a professor at �� GSE and one of the study’s coauthors.

Kim and her colleagues confirmed that women are now modestly rewarded for research on women’s issues. But that progress, they concluded, is being overwhelmed by implicit biases.

“As a society, we've made outstanding progress over the last century in transforming higher education and science institutions,” said Daniel Scott Smith, a doctoral candidate at �� GSE and coauthor of the study. “But implicit biases against certain kinds of research undermines our current efforts to make the academy more diverse — in terms of who becomes university professors but also in terms of what’s considered valuable academic knowledge.”

Former �� GSE postdoctoral fellow Bas Hofstra, now an assistant professor of sociology at Radboud University in the Netherlands, is also a co-author on the study.

Research Stories

split

GSE area

normal

Faculty mentioned in this article: Daniel A. McFarland

�� study reveals how public schools’ reopening decisions during the pandemic influenced a drop in enrollment

Brooke Donald Gorlick — Tue, 10 Aug 2021 00:09:09 +0000

�� study reveals how public schools’ reopening decisions during the pandemic influenced a drop in enrollment

Brooke Donald … Mon, 08/09/2021 - 17:09

Data Science | Education Policy | K-12

Research led by �� education professor Thomas S. Dee examines the impact of school policies to go remote or in-person in 2020-21.

August 9, 2021

By Carrie Spector

Public schools nationwide saw a major drop in student enrollment during the pandemic last year, with many families switching to private schools where classes were still held in person, taking up homeschooling or – especially those with kindergartners – just skipping a formal school year entirely.

According to newly collected data from a collaboration between �� researchers and the New York Times, public K-12 schools in the United States experienced a loss of roughly 1.1 million students in the fall of 2020 – and districts that began the school year with remote-only schooling had significantly larger enrollment declines than those that offered face-to-face schooling.

In a new study, researchers at �� Graduate School of Education (GSE) and Big Local News, a project of the �� Computational Journalism Lab, examine the extent to which the enrollment decline was influenced by school districts’ decisions to hold classes in person or go remote.

The team at �� worked with journalists at the New York Times on a major report published Aug. 7, which analyzed the enrollment drop at 70,000 public schools across the United States. EdSource, a nonprofit media organization covering education in California, and the nonprofit Colorado News Collaborative (COLab) also participated in the project.

Here, study coauthor Thomas S. Dee, the Barnett Family Professor at the GSE and senior fellow at the �� Institute for Economic Policy Research, shares more about how the collaboration came about, what the team discovered and why the findings are important as districts prepare for the beginning of a new school year.

How did you get involved in this project?

It was very serendipitous – I actually met Cheryl Phillips, who directs Big Local News, at our local dog meet-up. She lives around the corner from me and our dogs, Isabel and Tank, were getting along with each other, so we just started talking. She said something like, ‘I do data journalism,’ and I said, ‘I do data.’ I mentioned a paper I was working on with some very early data on school enrollment declines, and she came back to me a couple of months later, saying she’d been talking to the New York Times and other reporters about doing something really ambitious on this issue.

So my doctoral advisee Elizabeth Huffaker and I got involved, thinking initially that we’d just have more comprehensive data to do a descriptive look at disenrollment. Federal data on school enrollment typically lags far behind the state-level data, so we all scoured state websites and reached out to state officials to get the data they report to the feds.

Then we realized that a private company called Burbio was tracking data on school reopening plans for a sample of school districts. The company generously shared those data, and we put it together with the disenrollment data we’d collected from the states.

In a nutshell, what did you find?

We examined a large sample of K-12 school districts over six years, from fall 2015 through fall 2020. Public school enrollment typically increases from year to year, but it fell sharply in the fall of 2020, and we found that districts that adopted remote-only schooling had significantly larger enrollment declines than those that offered face-to-face schooling.

All of the districts experienced declines last year; even those in our sample that were face-to-face saw a 2.6 percent enrollment decline. But those that chose remote-only had a 3.7 percent decline. In other words, going remote-only actually increased the enrollment decline by about 40 percent. The decline was particularly sharp in kindergarten.

Were you surprised?

I went into this genuinely uncertain about what we would see, because I just know from my own position as a parent, I was conflicted about keeping my kids home in front of a computer or sending them into a classroom where they might get sick or bring the virus home.

The enrollment declines attributable to schools going remote-only were also particularly large in rural areas, which is a little enigmatic to me. To the extent that parents were turning to other schooling options, my supposition is that there are fewer of these options in more rural areas. So there’s still a lot to be learned about this. For example, access to high-speed broadband and digital devices may influence how parents in rural communities judged the appeal of remote instruction.

What would you want school leaders to take from your findings?

I don’t want to plant a flag on whether or not they made the right decision last year about going remote or in person. From a policy perspective, schools were really caught between a rock and a hard place: There were serious concerns about developmental harm with kids not being face to face, and at the same time, concerns about creating a vector for COVID transmission by bringing them back. Unfortunately, with many communities experiencing low vaccination rates and the spread of COVID variants, it appears we are confronting the same trade-offs once again.

But I think this study has at least three important implications for school leaders. One is that our results provide objective evidence on the character of parents’ preferences – specifically that some parents, particularly those with the youngest children, held the offer of remote instruction in such disdain that they were willing to go so far as to disenroll.

Second, these results provide leading indicators of the teaching and learning challenges educators will face in the wake of the pandemic. The exact character of those challenges will turn on what disenrolled students experienced last year and where they go this fall. For example, children who skipped kindergarten last year and enter first grade this fall will have their first experience with more formal schooling and may present different “readiness to learn” profiles to their teachers. Alternatively, if the children who skipped kindergarten “redshirt” into kindergarten this year, their teachers may confront unusually large class sizes and a more mixed group of older and younger students. Meanwhile some older children are navigating the educational consequences of switching schools or even sustained truancy.

Third, these results also suggest an imminent fiscal challenge to public schools if student enrollment doesn’t rebound. My best guess is that many of these students aren’t returning. As parents confront the uncertainty of the coming school year and what their local public schools will offer, they may view their new accommodations as a safe harbor. Because the students who disenrolled are disproportionately younger, this effect may be long-lasting. These are all areas where we should direct continued attention as new data become available.

The study was authored by Thomas S. Dee, the Barnett Family Professor at the GSE and senior fellow at the �� Institute for Economic Policy Research; GSE doctoral student Elizabeth Huffaker; �� lecturer Cheryl Phillips, director of Big Local News (which partners with newsrooms on investigative projects and provides a platform for journalists to share and analyze data); and Eric Sagara, senior data journalist at Big Local News.

Research Stories

banner

GSE area

normal

Faculty mentioned in this article: Thomas S. Dee

�� launches new master’s degree program in education data science

Carrie Spector — Sun, 26 Jul 2020 15:45:54 +0000

�� launches new master’s degree program in education data science

Carrie Spector Sun, 07/26/2020 - 08:45

��'s new master's program in education data science will equip students to interpret data amassed by school districts, universities and tech companies. (Photo: Laurence Dutton/iStock)

Announcements | Data Science

The pioneering program will focus specifically on the use of data to solve educational challenges.

July 27, 2020

By Carrie Spector

When it comes to understanding how people learn, data is vital—and it’s become ubiquitous, as school districts, universities and education technology companies amass increasingly complex records of students’ behavior.

Few education professionals, however, have the skills to make the most of that data. And few data scientists understand the educational environment well enough to interpret that data or even to know the questions to ask.

In a groundbreaking step to bridge this gap, �� Graduate School of Education (GSE) is launching a new master’s program in education data science, the first program of its kind to focus specifically on the use of data to solve educational challenges.

“Without data, it’s nearly impossible to provide students, teachers and policy makers with the kind of feedback that supports continual improvement,” said Daniel Schwartz, the I. James Quillen Dean and Nomellini and Olivier Professor of Educational Technology at the GSE. “The new master’s program in education data science will produce the talent we need to take the vast amount of data now being generated and apply it to new possibilities in education.”

The 18-month master of science program is now recruiting students for its inaugural cohort, which will begin in the fall of 2021. The GSE is also introducing a track for PhD students interested in focusing on data science.

“With the volume and new shapes of data we’re generating now, we can get a much more detailed picture of how people learn.”

Sanne Smith

Director, Education Data Science Program

‘A detailed picture of how people learn’

The use of data has surged in all areas of research and industry in recent years, and education is no exception. The growing use of technology in teaching and learning has created massive data sets—and with that, opportunities to improve and customize education.

Consider the widespread adoption of online learning platforms, which is providing schools and tech companies with extensive data on the learning experience. Clickstreams, discussion forums, even video and voice recordings reveal information about how students interact with the content and each other.

At the same time, advances in brain science research are beginning to increase our understanding about students’ spectrum of abilities. And while many schools have long relied on “thin” measures of learning and engagement, such as test scores and attendance records, new tools make it possible to capture more nuanced and wide-ranging information about student learning and the effectiveness of different teaching methods.

“With the volume and new shapes of data we’re generating now, we can get a much more detailed picture of how people learn,” said Sanne Smith, a lecturer at the GSE and director of the new master’s degree program. “The infrastructure to collect and analyze the data is just starting to emerge. We need professionals who can mine this type of data, organize it, visualize it and analyze it.”

Part of a larger initiative

The master’s program grew out of a new strategic initiative at the GSE called Transforming Education with Data, which supports research and innovations in the emerging field of education data science and learning analytics. The GSE’s initiative is connected to a university-wide focus on data science that arose out of the Long-Range Vision to advance the tools of data science to respond to challenges in science and society.

The new degree program in education data science brings together researchers in many disciplines—not just education and data science, but also engineering, linguistics, sociology, psychology, political science and economics.

The particular focus on education also sets this program apart from others that prepare students more generally for a career in data science—a focus that supports students in developing technical skills as well as building a professional community.

“You learn best when you practice on something that you care about,” said Smith. “In this program, students are studying data science with a group of peers who are also passionate about education. At the same time, they’re creating a network of fellow professionals who also care about education.”

Combining academic coursework with real-world experience, the program begins with core courses in data science and education theory and practice before students move into more specialized tracks. During the summer, students will either work with faculty as a research assistant or take an internship with a firm or institution such as a school district central office, education policy think tank or an education technology company.

“These organizations need professionals who can analyze data and ask the right questions,” said Smith. “There’s so much untapped potential out there, in terms of different types of education data and the ability to use it to shed new light on problems. This program opens up new possibilities for the field.”

GSE News

split

GSE area

school_news

Faculty mentioned in this article: Dan Schwartz , Sanne Smith

Data Science

Mapping the human screenome: Our vast digital lives

Chatting about chatbots: How AI tools can support teachers

������������ initiative reveals racial, gender, other disparities in Brazilian schools

What the school census doesn’t capture

Timely feedback on learning differences

A data-centered approach to education AI

‘Harnessing the power of data’: GSE’s Education Data Science program hosts inaugural symposium

What happened to students who left public schools during the pandemic? New research tracks their paths

������������ researchers partner with New York State to map the landscape of educational equity

������������ research reveals a hidden obstacle for women in academia

������������ study reveals how public schools’ reopening decisions during the pandemic influenced a drop in enrollment

������������ launches new master’s degree program in education data science

�� initiative reveals racial, gender, other disparities in Brazilian schools

�� researchers partner with New York State to map the landscape of educational equity

�� research reveals a hidden obstacle for women in academia

�� study reveals how public schools’ reopening decisions during the pandemic influenced a drop in enrollment

�� launches new master’s degree program in education data science