Here, off Canada’s Pacific
coast, researchers are hoping to make a long-held
dream come true. Behavioral biologists and
IT experts have teamed up to create programs
aimed at deciphering acoustic signals
from animals. Could artificial intelligence
help to identify patterns in the sounds
made by marine mammals? Will we soon start
to understand what it is that whales
are talking about? The coastal waters between
Alaska in the U.S. and Canada’s Vancouver
Island are where a group of killer whales or “orcas”
spend their summers. A team of scientists from the
DEEP AL research expedition are preparing to embark.
“DEEP AL” stands for “Deep Learning Applied
to Animal Linguistics.” Computer scientist Elmar
Nöth from Germany’s University of Erlangen has
spent years working on automatic recognition for
human speech patterns. Can the same methods be
adapted to animal languages? Underwater microphones
embedded in tubes serve as the
expedition’s ears. For three summers, teams
of computer scientists and biologists have set
out to record orca calls and document
whale behavior. Rachael Cheng from the
Leibniz Institute for Zoo and Wildlife Research
in Berlin is looking for patterns between behavior
and animal vocalizations that could help to
decipher whale language. I assume they would
exchange information. They may have a very, very
different system which may not fit into our
prototype of a language. Around 300 orcas are identified
as "Northern Residents" — as they spend each
summer along the coast of Alaska and northern
British Columbia. They live in small family groups
and are very communicative. Do different families
use different dialects? And is it possible to discern the
meaning of individual calls? Here the researchers
lower the highly sensitive microphones
into the water. The eight hydrophones
can record sounds up to a frequency
of 100 kilohertz. That’s far higher pitched than
what is audible to the human ear. Thanks to the network of hydrophones
and acoustic triangulation, the researchers will
later be able to calculate the positions
of the whales. To avoid disturbing the
whales with engine noise, the research trimaran is
equipped with an electric motor. While visibility is usually
limited underwater, sound waves are transmitted
over considerable distances. That’s why a
communication system with loud calls is
clearly beneficial. They’re very tight together,
with frequent direction change. Circling. And body twisting.
Plus a lot of echolocation. They are socializing,
like you will see. They bump into each other and
rub, and frequent twisting, and then jump
onto each other. It sounds like they
talk about the plan “What are we going
to do next?” Orcas only spend about five percent
of their time at the surface, which makes systematic
observation quite difficult. The research team
uses drones to document the behavior
of the animals. Diving expeditions with whales
are prohibited in Canada. The scientists are
looking for the smallest meaningful
units of communication. Which whale is calling,
and which one answers? Are some sounds repeated
more than others? Biologist Elysanne Durand
examines the recordings. Each call comprises a series
of brief pulses which sound like melodic
curves to human ears. Each shift in sound
could be meaningful. While recording the whale
calls, the researchers also document the behavior
of the animals. So there’s four individuals in
this group here that are circling, there were four back
there with the male, there is one I can see out of the
corner of my eye coming towards us, and the mom
and calf. So that’s 8, 10, 11
individuals minimum. The more data available, the
easier it is to "train" deep-learning programs to
decipher whale language. It’s therefore a major advantage
for the researchers that over 20,000 hours of orca calls
have already been collected — more than for any
other animal species. Whale researcher Jared Towers
has been tasked by the Canadian government with observing
different orca populations. Back in the 1970s, scientists
here began documenting individual animals as well
as their group structures. A108 is right underneath
the boat here, here she is, she
is gone off ahead. Jared Towers has no problem
telling the orcas apart. This scratch here on
A108 has persisted for at least a
couple of years now. So the way that we identify the
individuals no matter what population they belong
to is by appearance and you get used to looking
for certain features on an individual
killer whale. The dorsal fins and
the patches around the fins have
different shapes — enabling scientists to catalog
each of the Northern Residents. Every family forms
a lifelong bond. What we are looking at
with all these families is an adult female
leading the group, and the fathers of their
offspring don’t play much of a role in
that family group. Jared Towers works for
a federal institution that safeguards
Canada’s waters. His former boss,
John Ford, was the first to distinguish between
the disparate calls of the whales. Ford’s research revolutionized
our understanding of the communication system used
by the resident orcas. We are listening to
calls of A-Clan whales. That is the exciting part
of underwater listening. You are getting a window into their
life that you would never see. John Ford discovered that the whales
use about 50 different calls. Different families prefer
different types of calls. These were named
“dialects” and used to help identify
individual families. When they are making the
various stereotyped signals, it’s simply to keep in touch
with everybody in the group — they exchange them,
they are constantly monitoring each
other’s location. They know where they are because
of their very directional hearing, and they can monitor the behavioral
state, the excitement level, the arousal state of all the
other animals in the kin group. Northern Resident orcas
visit pebble beaches daily. You can hear them
rubbing right now. They’re making socializing
sounds and you can hear the pebbles getting
pushed around. And they just rub all
sides of their body. Just in the shallow
part of the beach. Only a few orca groups worldwide
engage in this sort of body rub. This behavior is not genetic,
rather it’s a tradition passed on within families of the Northern
Residents, just like their language. Back on the German-Canadian
research boat, it’s a challenge to locate each
family among the 300 individuals that make up the Northern
Resident population. The orcas are constantly
on the move in an expanse of water the
size of Belgium. The expedition
covers an area from Vancouver Island to the
southern tip of Alaska. Seagulls indicate where
schools of salmon might be. And this is where
orcas often hunt too. Calls from the “A-Clan” can be heard
on the underwater microphones. The team tries to determine
the position of the whales. They compare the calls with the
catalog compiled by John Ford, but they encounter
discrepancies. The calls of class "N9" are used
by different whale families. Calls from the same class
should be almost identical, but these differ in length,
melody and harmonics. Human analysis so far has amounted
to only a rough classification. Looking at the spectrogram
I am very sure that we can achieve something that rivals
the human performance. Two whale families approach.
What calls are they exchanging? Here is that call. OK,
let’s just record this. After I hear the call the
back group surfaced. Then I spotted the front
group turning around. The researchers are interested
in which group is calling, which one answers, and which
sounds they’re using. Look here, I4 is approaching the
boat. You can hear lots of calls. That's N23 from
the GI clan. There is a lot of
variation also — and interestingly, here
is the A23 family. The calls look
very different. That’s how people differentiate
different matrilines. If we have a lot of calls, we
can try to train a classifier. The programmers use
algorithms or “classifiers” to automatically analyze millions
of whale calls in order to compare recurring sound patterns
with recurring behavioral patterns. This same method is used
to decipher the meaning of individual words in
foreign languages. For Elmar Nöth and his team, it’s no
easy feat to automatically filter out the weak orca calls
from the constantly fluctuating noise
of the ocean. By matching thousands of orca
calls on the input side with a target value on the output side,
they train “neural networks.” These layers of neural
networks then learn bit by bit which sounds are
typical for orcas. The purple peaks
indicate a high probability of having
found a whale call. It’s the first time a
machine has been trained to automatically
detect orca sounds. Still, clustering
calls into similar sound groups remains
a big challenge. When comparing tens of
thousands of recordings, all whale calls are sorted
by what they have in common. But the cluster is still too
crude to form reliable classes. Further programming
is required. The team suspects that the
Northern Resident orcas have a complex language system
and may even process more intricate sound
patterns than humans. But how much is even known about the
acoustic capacity of their brains? This is Telegraph Cove
on Vancouver Island, where Elmar Nöth and Rachael
Cheng have an appointment. Only a handful of
researchers worldwide have expertise on whale
and dolphin brains. Neuroscientist Lori
Marino is one of them. Is this an
orca brain? This is an orca skull, a cranium and
in the front you find the melon, where they do the
echolocation. And this part right here, this large
part, is where the brain would sit. Human and whale brains
generally share the same mammalian
architecture. Both cerebral cortices are
wrinkled and complex. But the orca brain
is 5 times bigger, among the largest
of any animal. And unlike the human brain, the
orca also has a paralimbic lobe. The part of the orca
brain that mostly fascinates me is this
part right here. We have this special extra
lobe, called paralimbic lobe. It connects feelings,
emotions and thoughts. And it’s so complex in the orca
brain, it has become its own lobe. That means a lot more of
their communication has to do with their
feelings and emotion? Communicating feelings,
communicating emotions, is very important
to an orca. Whales have a far larger
acoustic area than humans. Lori Marino’s team also discovered
a second acoustic cortex. And now we know that they
have also a sound processing area in the same area where
we process language. Based on what you said
how they process sound, can we say that we need a much
finer window when we analyze? It‘s not just fine
resolution that we need. We need a way to do the
analysis at a much more complex level than our
brains can do it. New brain scan methods have
revealed that orcas have a far more complex acoustic system
than previously believed. Could the same be true
for other marine mammals that use sound for their
communication and orientation? In contrast to
other orca groups, the Northern Residents at
some point stopped hunting seals and other
marine mammals. Today they feed almost exclusively
on the largest salmon in Canadian waters,
the Chinook. Stocks of the
fish, however, have been declining rapidly
since the late 1990s. The Northern Residents have
to travel increasingly long distances to find the
salmon that still remains. And despite stringent fishing
quotas for the species, their numbers
continue to decrease. Jared Towers from Fisheries
and Oceans Canada monitors if people are complying with
the fishing regulations. It seemed to be that
if Chinook salmon aren't quite so
much abundant, the killer whales aren't
getting enough to eat and that causes populations to
go down a little bit. The Northern
Resident Orcas are classified as a
"threatened species.” When Chinook
abundance is low, they don’t simply go and
try catching other fish. They have not found an
alterior prey resource. It is not like they start
and go kill harbor seals, just when there is not
enough salmon around. The substantial human
appetite for salmon has decimated stocks to
critically low levels. The ruins of the salmon fishing
industry serve as a reminder of the once abundant
fish stocks. Wild Canadian salmon were in
great demand around the world. The survival of the Northern
Resident orcas is tightly linked to the fate of the
Chinook salmon. Lance Barrett-Lennard
is a geneticist and behavioral ecologist at
the Vancouver Aquarium. He visits the resident orcas
every summer to document their long-term health and measure
how their body size and shape compares to
previous years. To ensure the images are
comparable from year to year, the drone hovers exactly 30
meters above the whales. The group's calf is
almost a year old — and is still suckling
from its mother. The social skills of
orcas are illustrated by how they deal
with their prey. You’ve got a species that is highly
social, that shares all of its food. So they’re compulsive
sharers, killer whales. When they catch a fish, one will
eat half and head it off to somebody else and have a bite, and
then another one takes a bite. The adults sometimes leave living
salmon to the younger whales. In this way, the juveniles learn to
hunt their prey in a playful way. At the Vancouver
Aquarium, researchers compare the
images from different years. One indicator of the whales’
state of health are the white patches
above their eyes. If there are not enough fat
reserves in the neck area, the angle along the
eye-spots will change. Comparing the images
side by side, researchers observe a
life-threatening weight loss. In response to the decline
of natural stocks, the fishing industry installed
aqua-farms in the regions the remaining wild salmon
migrate through. Nets protect the farmed salmon
from orcas and other predators. The predominantly Norwegian
operators introduced farmed Atlantic
salmon species — and with them, new
viral diseases not previously found in
Canada’s Pacific Northwest. These fish farms
raise as much salmon as possible in a
confined space. It’s all about
maximizing profit. The by-product? Feces and germs
that escape into open waters. There is growing
concern of viral outbreaks among the
wild Chinook salmon. If the primary food
source of the resident orca population
continues to decline, the whales’ own survival
will also be endangered. Whale researcher Alexandra
Morton is studying the new diseases affecting
the salmon. She visits the research trimaran to
share the results of her analysis. I brought some results
on my laptop, if you’d like to see them,
the virus research — where we found it and why
are the salmon farms such a big problem for the
salmon and the whales. Mostly because of where
they are located. So we do a molecular test that
basically just looks for the genetic sequence or part of
the sequence of the virus. Salmon take in viruses
through their gills. The pathogens then infect their
bloodstream, causing organ failure. When the salmon farms moved
in, the orca stayed away. That made Alexandra
Morton suspicious. She took samples
near the farms, which were then analyzed
in special laboratories. She examined the organs of
both wild and farmed salmon. 95 percent of the farmed fish
she dissected were sick. A large proportion of the
wild salmon also became infected through
contaminated water. The pathogens escape from
the fish farms with the tidal currents and spread
to the regions the young wild salmon have to pass
through as they migrate. 200 kilometers further
north, the “A42” family searches for prey
along the shorelines. The research expedition has reached
the fjords of Fisher Channel, once famous for its
ample fishing grounds. The whales hunt by
day and by night. They don't need light
for orientation. Biologist Florence Sullivan
compares her recordings to known call types and
notes down behavior. Do the orcas tend to
use certain calls in a social context, or
more when hunting? All recordings are later
re-analyzed in Germany with the help of "deep
learning" programs. In their search for prey, the
Northern Resident orcas now often cover 80 to 120
kilometers a day — along hunting routes that have
been passed on for generations. To identify individual whales and
examine changes in group structures, programmer Manuel Schmitt and
biologist James Field are working on automatic "fin
recognition software.” It works similarly to facial
recognition programs. The process of identifying whales
by their fins and dialect was used years ago in the
famous case of an orphaned calf known as
"A73” or “Springer.” When her mother died, the
then 2-year-old Springer was separated from the
rest of her family. Eventually, 300 miles
further south, the lone calf was found
off the coast of Seattle. Scientists could tell from her calls
that she was a “Northern Resident.” Springer was brought back and
reunited with her family. So while researchers
have been able to identify whale dialects
for 20 years now, understanding the
“language of whales” represents a whole
new challenge. The call systems used by
different whale families change little from one
generation to the next. Springer now has two
calves of her own, "Spirit" and "Storm" — roughly
eight and four years old. Orcas typically spend their
entire lives with their mothers. For the first two years
they are suckled and over the following
eight years, they learn what they
need to do to survive. Another orca group has
joined Springer's family. Together they search for
salmon close to the shore. Their hunting strategies vary
depending on the local environment. This gives them a wealth
of experience that, like their communication system, is
passed down through generations. Their group behavior during the
hunt seems to be coordinated. The whales exchange
a stream of calls. The orcas are still able
to find enough food by spending more and
more time hunting. For a calf, it’s one of many
lessons in salmon hunting. For the scientists, it’s
a rich source of data. Human activity greatly
impacts the whales’ habitat. This continuous
loud sound... and now the orca calls
become the background. The sound of passing ships
doesn't frighten the whales... but the noise does interfere
with their communication — and compromises their
hunting ability. The Canadian government is
investigating the impact of ship noise on
marine mammals. We listen for their
vocalizations, which can cover a
range of 10 km or so. Man-made noise in the world's
oceans doubles every ten years. One of the concerns we have
about vessel noise is its potential
impact on the echo-location abilities
of the whales. The whales’ vision is
very limited underwater, because of murkiness in the
water, especially at night. Typically in these waters they can’t
see more than a whale's length, perhaps 10
meters max. In quiet conditions the
whales can probably detect a Chinook salmon at a range
of perhaps 200 meters, but that might be significantly
reduced by masking, by boat noise. Similar to bats, orcas scan
their surroundings by emitting clicking sounds and
listening to their echoes. The noise from ships
masks those echoes. To reduce its own
engine noise, the expedition boat runs on an
ultra-quiet electric drive. It sounds like
a N4 call. But it‘s masked
in the noise. Here we have the
boat‘s engine noise. And there is an
orca call here. I have another recording
with our engine You can hear the harmonics.
It‘s much clearer. Still we have this
electrical noise at 10 kHz. Let me have a look
at the engine. Although electric engines
are far quieter, there’s nevertheless
some interference. Rachael Cheng inspects
the static noise and examines the
control cables. Is it
better now? Much better
now. The sound of electric motors
is still rare in the ocean — which is perhaps why the ship
triggers the whales' interest. The boat stops in
order to maintain the requisite distance
of 100 meters. The curious orcas are
not quite as cautious — and inspect the
boat and its crew. The scientists wonder
whether the whales might use distinct calls
for different boats. In any case, there is immense
curiosity on both sides. Orcas are extremely social creatures
— and love to touch one another. I think this is the A23 and
A25 group. Floating sideways. Putting its fin on
the water surface — where others just
mingle around. Are their jumps also part
of their communication? Or just a
bit of fun? Many of their
behavioral patterns are still only partially
understood. This is also due to
the fact that orcas rarely roam close
to the surface, where humans can
observe them. Some distance away, another group
of orcas suddenly appears. Marine biologist Elysanne Durand
wants to get an overview. Which direction
should I go now? If we can have the whole group
on our starboard side ... This is a completely
different type of orca. There are maybe 4
transients off the island. Unlike the
resident whales, these transient orcas
don’t feed on salmon. They hunt other
marine mammals. Do you hear anything from
these guys, Rachael? No, I don’t hear
any vocalization. But it seems they’re
hunting there, foraging. In another contrast to
Northern Residents, transient whales hunt
almost silently. This is because dolphins,
seals and other whales — their natural prey —
have excellent hearing. That means, they know exactly
when they’re in danger. The transient orcas also inspect
a rocky plateau for prey. ? a tactic that appears
to be successful. We hear them
vocalize now. The blood at the surface
reveals a kill. The transients begin to
celebrate their catch — and now exchange many
calls back and forth. Acoustically they are very
different from the residents. I’m curious whether the classifier
can pick up the difference. The homes of the
roughly 500 transients and 300 Northern
Residents overlap. But their calls differ
so much, it’s as if they’re speaking
different languages. Dolphins and other marine mammals
can tell the difference. While they immediately flee
from transient whales, they rush intentionally
toward the residents as soon as they
hear their calls, to hunt and
play with them. Below deck, Elmar Noeth and
Christian Bergler work on the automatic call
classification. Here, hear the orcas? It shows here
the detection of those signals. We always have 2 seconds,
2 seconds, 2 seconds. You can clearly see
the incoming sound. If there is an orca,
they will be detected. What I like is that the
confidence is so high. The algorithm is quite
sure that it did find. The automatic call
detection works? But the challenge is still to match
one groups of calls with another. In human language, the equivalent
of these sound sequences might be simple
statements. The Northern Residents
consist of 32 families. They typically use
different call systems — but are still able to communicate
with each other when they meet up. Looks like the other
group is joining them. The calls between the
groups go back and forth — announcing their arrival in a
kind of greeting ceremony. Do you still think they are saying,
I am here, if they are so close by? No, it’s not
the same call. The last time it was
really one call the individual was repeating
all the time. Here they are doing
combinations. Meetings between family groups
used to be more frequent — when salmon stocks were more
abundant, and they had to hunt less. It seems to be
more melodic. Their sounds or vocalizations
were a lot longer. I would say they
are more complex. They exchange another
series of calls before the meeting
slowly breaks up. These sounds might
signify "goodbye.” But the researchers will need to
compare many other similar exchanges before they can interpret
it more decisively. There are thick banks of fog
between Vancouver Island and the mainland as the end
of summer approaches. Tracking down the whales
is more of a challenge. The researchers’ only
option is to use acoustic localization
with the hydrophones. Under calm sea conditions
the sound of the orca calls can range up to
10 kilometers. Elmar Nöth checks the
location forecasts. At 12 o'clock,
200-300 meters? Over there at 1
o'clock — 1, 2, 3, 4. Localization is also important
in order to interpret the calls. It’s like 2 or 3
here, 2 or 3 there — and then it’s always the
same call we hear right now. So it’s probably,
"I’m here.” And the other one recognizes,
"oh, Fred is over there.” What I would do is to cluster
exactly those calls where I’m convinced that
I heard the same. And then just from the sequence
one could make the assumption that's a signal-and-response
signal, something like that. The last of the three expeditions
is coming to an end — with data from a total of 20 weeks
of field research on board. A few months later, the
researchers meet at the Pattern Recognition Lab at
the University of Erlangen in Nuremberg,
Germany. Rachael Cheng has an appointment
with the programming team. Manuel Schmitt has run an automatic
comparison of one million calls. Hi Rachael, good
that you’re here. Take a look at the clustering.
I made some changes. It’s a little bit
better than before. N3 is here, it looks very similar,
but its mixed with N9s over here. This one looks very different
but it still needs training. Background noises hinder the
automatic sorting of various calls. A newly programmed noise filter
increases the precision. Is that the new clustering?
That’s a lot better, right? It makes tiny
differences. There are tiny
artefacts. There are still small errors.
But on such a large scale — where millions of calls are
being automatically compared — these become less
significant. I think we’re now at the
level where we can run through a couple
of tapes now. But you can use those cluster
sequences in order to find those language patterns -
like a semantic structure? And those can be
interpreted as a call, followed by an answer-call,
followed by another call. The new clustering results
are better than expected. Rachael Cheng's task
now is to assign recurring call patterns
to certain behaviors. She compares the new clusters
with the existing call-catalogs — and discovers reappearing matches
that might be meaningful. The algorithms
have learned to differentiate the calls of
different orca families. That would normally take human
researchers years of training. In the future, this
will allow automatic detection of which
family is calling, and the ability
to follow the subsequent dialogues
between families. The next step is comparing
millions of call sequences. Which calls appear
together frequently, accompanied by which
similar behavior? When decoding
human languages, such connections might reveal
meaningful possibilities such as “sit” and “chair” or
“table” and plate.” The deep-learning programs cannot
yet work completely independently. Without human control,
there could be a huge increase in
assignment errors. The first
matches appear The same call sequences were
used in a similar context. But there are
exceptions. The comparisons made so
far are not conclusive. The re-occurring
call sequences point to the scene — they
are socializing. And we also find it
the second time in a socializing context, the
same call patterns. We expect to find it
in the third sample also, but we did not
find it in this. That does not disprove it. There
could be different explanations. We just don’t have
enough data to say that. The German-Canadian research
team is not yet able to create a kind of dictionary
of the orca language. But they do now have tools to
compare whale calls in a more detailed and systematic way
than was previously possible. And as more recordings of
whale calls are shared and made available for training
deep-learning machines, the faster it will
be to recognize the subtleties in
communication patterns. Humans have long faced
limitations when it comes to understanding what
animals might be talking about. But with the help of AI, a new era
of research might allow us to decipher the secrets behind
their communication.
So long, and thanks for all the fish.
I'm not worried about whales, we saved them back in 86
Mr Whale: Where should we eat?
Mrs Whale: I dont care.
Mr Whale: Feel like krill?
Mrs Whale: No.
Mr Whale: Plankton?
Mrs Whale: Ugh.
Mr Whale: Squid?
Mrs Whale: Not again...
Mr Whale: ............
Mrs Whale: I'm hungry!