[music]
-Hello, and welcome back to the ILO’s Future of Work podcast.
I’m Sophy Fisher.
Advances in generative Artificial Intelligence, or AI,
will have widespread implications for labor markets and production.
Whereas in the past automation
has been seen as primarily affecting low-skilled jobs,
such as those in agriculture or in factories,
AI is different.
Many say that the effects of tools
such as ChatGPT will be on more highly skilled jobs,
the so-called white-collar workers.
I have with me today two people
who are well qualified to throw light on this question.
Shakked Noy and Whitney Zhang
are both with the Massachusetts Institute of Technology,
better known as MIT, which is in the United States.
They’ve published a new paper,
Experimental Evidence
on the Productivity Effects of Generative Artificial Intelligence.
Shakked and Whitney, welcome.
Thank you very much for joining us on the podcast.
-Thanks for having us.
-Thanks for having us.
-First of all, why don’t you run us through the key findings of your paper?
-Yes, so the way we typically summarize it,
our paper has four main findings.
The first is that access to generative AI,
in this case, ChatGPT-3.5,
increased the productivity of medium-level white-collar professionals
on the sorts of writing tasks
they might typically encounter in their jobs.
It raised their productivity both by
allowing them to complete those tasks faster
and by slightly raising the quality of the resulting output.
The second finding is that
the productivity gains were larger for workers
whose initial ability levels were lower.
They gained about the same in terms of increased speed
as higher-ability workers but gained much more in terms of quality
of their output.
The third finding is that workers’ initial reactions
to encountering these technologies seemed to be pretty positive.
People were more satisfied
with the work that they’d done and with the task
when given access to ChatGPT
compared to when they were asked to complete the task
without access to that technology.
Then finally, when we followed up with the workers
who’d completed the tasks in our experiment
and asked them about their activities in their real jobs
two weeks and two months after the experiment,
a substantial fraction of them,
30-40% were using ChatGPT in their real jobs.
The percentage using it
was substantially higher in the treatment group
whom we’d introduced to ChatGPT compared to the control group
whom we had not introduced to ChatGPT,
suggesting that there are still frictions or barriers
to diffusion of the technology in real workplaces.
-Right. To sum up your first two findings,
essentially it makes higher-skilled workers quicker
and lower-skilled workers both better and quicker.
-Exactly, yes. -The consequences for this are
potentially greater productivity.
What do you think the consequences are for employment and wages?
-This, I would say, depends on a number of factors
that we can’t directly address in the experiment itself.
In particular,
if we think about things on both the demand and the supply sides,
if there’s this increased productivity
and workers are able to produce more than on the demand side,
there’s this question of,
are people going to demand these types of outputs more?
For example, if we think that the demand for writing,
or advertisements, or coding would substantially increase,
then this could suggest an increase in employment and wages.
Whereas if that demand is relatively inelastic, then perhaps
this would lead to a decrease in jobs overall.
Then just thinking about the supply as well.
One thing is that our tasks are around this very specific part of
what most people’s jobs are doing this kind of writing.
Whereas a lot of tasks in the real world
are much more loose and require a lot more judgment and planning,
and these other factors.
Then just also thinking about if this is 5% of someone’s job,
for example,
maybe I can take that 5% out and then replace that person entirely.
This just means that they have 5% more time
or they’re going to fill that time with other work.
It also depends to what extent these tasks
are the main part of someone’s job
or just some small part that’s then complemented by other pieces.
-The subjects in your experiment
were basically doing writing tasks, weren’t they?
Which required them to synthesize information
and then produce a piece of text.
That’s correct, isn’t it?
-That’s right. -What other skills do you see
this applying to?
Would it simply be that?
-I think it’s a potentially very flexible technology.
We’re only beginning to see the range of applications
that the generative AI systems can be applied to.
I think definitely writing tasks like the ones in our experiment,
in the sense that they’re relatively generic,
they’re not extremely technical,
are one main application for generative AI.
There’s also lots of other things.
It’s very good at writing code, for example.
It can be good at other things like summarizing documents,
synthesizing information,
maybe even as a cooperative tool for brainstorming or coming up with ideas.
It’s, potentially a technology with a lot of applications.
-I would think your findings might be of concern to high-skilled workers
who are hearing what you say because essentially,
someone could employ a lower-skilled worker
who would presumably be cheaper
and get almost exactly the same results as a worker
who might potentially be more expensive.
-I think limited to the specific set of tasks,
I would say yes.
I think, in some sense, that’s what you would find here.
At the same time,
again,
I think a lot of these times these tasks are bundled together.
Potentially, when you’re paying for these high-skilled workers,
you’re also asking in some sense for their skills and judgment
or interpersonal communication
or their ability to work with others on a team.
These aren’t things that I think ChatGPT right now would be great at assisting at,
and I think it’s still our important skills in the workplace.
Depending on what that bundle of skills looks like in the job,
I think that would ultimately predict what is going to happen.
Then I think the other thing, this relates back to
complementarity versus substitution more broadly,
is that here, these initial low performers are doing much better,
but they’re doing much better because they’re just taking this prompt,
putting it into GPT,
taking this output, and then submitting it is what we generally seem to be finding.
Then there’s also a question of,
well, do you even need these low-skilled workers in the first place?
Or can you just hire someone to do this prompting?
That would ultimately be more efficient.
In that case, I think it becomes even more unclear
what the impacts on inequality might look like.
-Did you look at all that the impact on inequality
or was that something that still requires more research depending on
how the technology is rolled out?
-We looked at an important input into the ultimate effects on inequality,
which is by how much it raises the productivity of lower-ability
compared to higher-ability workers.
There’s a common argument
that inequality has increased a lot over the past several decades,
at least in part because technologies like computers
enhanced the productivity of high-ability workers
more than they enhanced the productivity of low-ability workers.
Which gave rise to divergence in wages and earnings.
In this case, what we found, again, limited to a very specific set of tasks
in a very specific context
was the opposite, which is that
the productivity impacts were greater for lower-ability workers.
A bunch of other factors will determine
what the ultimate equilibrium effects of
these technologies on unemployment will be,
like some of the demand side factors and supply side factors
that Whitney has talked about,
how training and human capital acquisition will adapt,
how companies will restructure their workforces
and their models of production and things like that.
It’s really impossible to tell from current information
what the ultimate impacts on inequality will be.
-Higher productivity is often associated
with reductions in poverty and increasing wages.
In that sense, it might actually reduce inequality.
-Absolutely.
Productivity has been stagnant for a long time,
even as we’ve gotten fantastic new things like the Internet.
I don’t think there’s a consensus on why, for example, the Internet
hasn’t resulted in higher productivity growth than we’re currently seeing.
It suggests that things like
bottlenecks from other parts of the production system
or other factors might be important.
I wouldn’t be super confident from the fact
that generative AI raises the productivity of white-collar workers specifically,
that this will translate into the kinds of
broad-based productivity gains for the economy that would translate into
increases in living standards and reductions in poverty.
I think it’s too early to tell.
-I wanted to pick you up on one particular sentence in your paper,
which said that the experiment’s evidence suggests
that it largely substitutes for worker effort
rather than complements worker skills,
potentially causing a decrease
in demand for workers with adverse distributional effects
as capital owners gain at the expense of workers.
By which I think you mean that the people who have the money and the capital
will gain a higher proportion of the benefits.
Would that be correct?
-That’s right. -This action is taken to shape the impact,
be that through legislation or taxation or something like that?
-Right. I think this goes back to
what we were just talking about in terms of
what are these workers actually doing in our experiment,
and it’s that they’re taking these prompts,
they’re putting them into ChatGPT, and then they’re submitting these outputs.
It seems plausible that you could,
if that’s all that they’re adding value to,
then you could have someone where that’s their job.
You don’t need all of these workers to be doing that kind of task.
Who then takes all of the profits
that come out of doing this kind of output generation?
Then it’s the people who own this type of capital in the first place.
-Do you think it is also possible to shape the impact
and the way that this is applied through other kinds of institutions,
for example,
stronger worker representation, greater voice in the shaping of processes,
unions, things like that?
-Absolutely.
I think this is an area where there’s been a lot of speculation
because there’s a very large amount of anecdotal
or qualitative evidence about the ways
that unions or other forms of worker representation
are involved in shaping the way that technologies affect firms
and shaping how firms adjust their employment structures
in response to new technologies.
I would say there’s still very little solid evidence
on the interaction
between systems of worker representation and technology adoption.
You can tell optimistic stories
where worker representatives protect workers
and encourage firms to upskill them,
thereby ensuring that technology gets rolled out in a way
that really benefits everyone,
benefits the consumers through productivity gains,
but also benefits workers
through internal productivity gains and wage gains.
There’s also potentially stories
where worker representatives block the deployment
and development of new technologies
in a way that’s ultimately harmful for society
because they’re representing relatively narrow interests.
I would say there’s as of yet, no good empirical evidence
that would help us adjudicate between those two possibilities.
That definitely seems like a really important area of research going forward.
-I was also talking about those who might benefit.
I was also struck by a point you made in the paper
about, that it could increase options for people
who potentially are very creative or who have a lot of good ideas,
but simply have a problem expressing themselves
in ways that others can understand and digest.
For example, people who have problems with writing skills,
either because they’re very technological
or simply because perhaps they’re writing in a second or third language.
ChatGPT could help them literally get their message across.
-That’s right.
I think anecdotally, I’ve certainly heard from people
for whom English is not their first language that,
for example, ChatGPT has been really helpful
in helping them with academic writing or otherwise,
where before this was a bottleneck for them.
Now they’re much more able to communicate their research or other areas.
I think this is another way in which
GPT technologies could be a sort of leveling
of the playing field in that
people who might be really great with creativity or thinking of ideas,
but just have difficulty writing them in a way
that sounds like professional
or academic style can now use the technology
to make their writing more sound like these other aspects.
I think this could be potentially great
for both these workers
as well as just for innovation as a whole, potentially.
-I was also struck by one of the other main conclusions
that you mentioned,
which was about the most subjective outcomes, e.g.
the job satisfaction
that a lot of your subjects in the experiment
found using ChatGPT gave them equal or greater job satisfaction.
Whereas traditionally in the past, people have felt that
forms of automation have reduced job satisfaction.
Do why they felt that the job was more satisfying
and whether that feeling lasted
or whether it was simply a hangover from the experiment?
-Right. To some extent, we definitely think that
this reflects people’s first reactions to an exciting new technology
rather than necessarily something fundamental and permanent about the way
that the technology affects their enjoyment of their work.
When we followed up with people two weeks and two months after the experiment,
the people who were in the treatment group
were much more likely to be using ChatGPT in their real jobs,
but they weren’t more satisfied
with their real jobs than people in the control group.
Now, that doesn’t necessarily mean
that ChatGPT won’t raise job satisfaction in the long run.
It could be that it’s just too small of an impact on a two-week
or two-month time horizon
relative to your base prior level of satisfaction in your job.
We don’t find evidence
that it’s permanently improving job satisfaction.
As to why it raised job satisfaction in our experiment,
one possibility,
again, is that people are just really excited
to discover a powerful new technology.
Another possibility is just,
ChatGPT helps with many elements of tasks that might be the least enjoyable.
Editing,
translating your ideas into a rough draft.
The tasks that we gave people were not terribly exciting or stimulating.
People may in general have found,
writing an e-mail
or writing a short report to be a boring task.
In that case, anything that helps you finish that task more quickly
might raise your job satisfaction.
You could tell a story where basically,
ChatGPT is speeding people up
or helping them expend less effort precisely in the areas
that they least enjoy working and in the types of tasks
that they least enjoy doing.
Again, I would say there’s no conclusive evidence
on that from our experiment.
-In that case, research always raises as many questions
as it answers.
Where do you want to go next with this?
What do you see as the key questions that have been raised by
what you find so far?
-I think the next big thing is really to try to move into the field
into a setting where the tasks are much less well defined,
where we’re requiring much more of workers
to use a broad bundle of skills,
where there’s many of these organizational-type concerns
that we aren’t able to address an individual level experiment.
To see what productivity impacts of workers look like there.
I think there’s just also interesting work to be done in looking at
how organizations are figuring out how to adapt to these workflows.
Shakked and I have been looking into some more qualitative stuff
where it seems like companies are really
seeing this as an exciting tool, but not quite sure of
how do I actually get my workforce to start using this?
What are some areas
where this might really improve productivity in my organization?
What are good use cases and what are bad use cases?
Of course, there’s all sorts of regulatory
and legal things as well that further complicate things.
I think just having a broader look at the ecosystem as a whole,
I think is the exciting next direction.
-Yes. In other words, the difference between what happens in the lab
and what happens in real life.
Shakked and Whitney, thank you very much for joining us.
That was extremely interesting.
Another look at the effects of generative AI on the world of work.
Unfortunately, that is all we have time for
in this edition of the podcast.
Once again, thanks to Shakked Noy
and Whitney Zhang for joining us and talking about their paper.
If you want to know more about it,
you can find a link to it on the webpage for this podcast on the ILO’s website.
For now, thank you very much
for listening and please join us again soon.
For now, goodbye.
[music]
