As OpenAI’s ChatGPT continues to change the game for automated text generation, researchers warn that more measures are needed to avoid dangerous responses.
While advanced language models such as ChatGPT could quickly write a computer program with complex code or summarize studies with cogent synopsis, these text generators are also able to provide toxic information, such as how to build a bomb.
In order to prevent these potential safety issues, companies using large language models deploy safeguard measures called “red-teaming,” where teams of human testers write prompts aimed at provoking unsafe responses, in order to trace risks and train chatbots to avoid providing those types of answers.
However, according to researchers with Massachusetts Institute of Technology (MIT), “red teaming” is only effective if engineers know which provocative responses to test.
In other words, a technology that does not rely on human cognition to function still relies on human cognition to remain safe.
Researchers from Improbable AI Lab at MIT and the MIT-IBM Watson AI Lab are deploying machine learning to fix this problem, developing a “red-team language model” specifically designed to generate problematic prompts that trigger undesirable responses from tested chatbots.
"Right now, every large language model has to undergo a very lengthy period of red-teaming to ensure its safety,” said Zhang-Wei Hong, a researcher with the Improbable AI lab and lead author of a paper on this red-teaming approach, in a press release.
“That is not going to be sustainable if we want to update these models in rapidly changing environments. Our method provides a faster and more effective way to do this quality assurance.”
According to the research, the machine-learning technique outperformed human testers by generating prompts that triggered increasingly toxic responses from advanced language models, even drawing out dangerous answers from chatbots that have built-in safeguards.
The automated process of red-teaming a language model depends on a trial-and-error process which rewards the model for triggering toxic responses, says MIT researchers.
This reward system is based on what’s called “curiosity-driven exploration,” where the red-team model tries to push to boundaries of toxicity, deploying sensitive prompts with different words, sentence patterns or content.
"If the red-team model has already seen a specific prompt, then reproducing it will not generate any curiosity in the red-team model, so it will be pushed to create new prompts," Hong explained in the release.
The technique outperformed human testers and other machine-learning approaches by generating more distinct prompts that elicited increasingly toxic responses. Not only does their method significantly improve the coverage of inputs being tested compared to other automated methods, but it can also draw out toxic responses from a chatbot that had safeguards built into it by human experts.
The model is equipped with a “safety classifier” that provides a ranking for the level of toxicity provoked.
MIT researchers hope to train red-team models to generate prompts on a wider range of illicit content, and to eventually train chatbots to abide by specific standards, such as a company policy document, in order to test for company policy violations amidst increasingly automated output.
“These models are going to be an integral part of our lives and it's important that they are verified before released for public consumption,” said Pulkit Agrawal, senior author and director of Improbable AI, in the release.
“Manual verification of models is simply not scalable, and our work is an attempt to reduce the human effort to ensure a safer and trustworthy AI future," Agrawal said.
The federal firearm buyback program has cost taxpayers nearly $67.2 million since it was announced in 2020, but it still hasn't collected a single gun.
If something looks too good to be true, it might be. That's the message from Saskatchewan horticulturists after customers have come into their stores hoping to buy purple apple trees this month.
The province's public security minister said he was "shocked" Thursday amid reports that a body believed to be that of a 14-year-old boy was found this week near a Hells Angels hideout near Quebec City.
Chris Knayzeh was in a town overlooking Lebanon's capital when he heard the rumbling aftershock of the 2020 Beirut port blast. Hundreds of tons of haphazardly stored ammonium nitrates had exploded, killing and injuring thousands of people.
The Royal Canadian Mounted Police has lost 205 firearms since 2020, including more than 120 handguns and at least five fully automatic weapons like machine-guns.
For most people, dinner on a cruise ship is a time to relax. But when influencer couple Abby and Matt Howard decided to kick back with a dinner à deux, they ended up kicking up a storm.
Prime Minister Justin Trudeau tapped Treasury Board President Anita Anand to take on additional duties as Canada's minister of transport on Thursday.
B.C.'s police watchdog is investigating the death of a woman who was shot by the RCMP after allegedly barricading herself in a room with a toddler early Thursday morning.
Conservative Leader Pierre Poilievre and NDP Leader Jagmeet Singh got into a heated exchange in the House of Commons on Thursday, just minutes after Singh announced his party would not be supporting the Conservatives' first non-confidence motion against Prime Minister Justin Trudeau's government.
B.C.'s police watchdog is investigating the death of a woman who was shot by the RCMP after allegedly barricading herself in a room with a toddler early Thursday morning.
Shamattawa RCMP are searching for a missing six-year-old boy who hasn’t been seen since Wednesday morning.
Friends and strangers have set up a makeshift memorial outside the home of a five-year-old boy who was found dead yesterday in Coteau-du-lac.
The Royal Canadian Mounted Police has lost 205 firearms since 2020, including more than 120 handguns and at least five fully automatic weapons like machine-guns.
The lawyer for a B.C. RCMP officer convicted of obstruction for telling a witness to delete cellphone video following the violent 2017 arrest of Dale Culver has requested a stay of proceedings.
The province's public security minister said he was "shocked" Thursday amid reports that a body believed to be that of a 14-year-old boy was found this week near a Hells Angels hideout near Quebec City.
Israel warned U.S. Defence Secretary Lloyd Austin in a call Tuesday that a military operation was going to take place in Lebanon but gave no details, U.S. officials said Thursday. The same day of the call, in an attack widely blamed on Israel, thousands of pagers belonging to Hezbollah militants exploded.
A federal law enforcement agency confirmed it's opened an investigation into Robert F. Kennedy Jr. after he allegedly cut off the head of a dead whale and took it home two decades ago.
A judge in a rural Kentucky county was shot and killed in his courthouse chambers Thursday, and the local sheriff was charged with murder, police said.
A 22-year-old woman who became an abortion rights advocate after she was raped by her stepfather as a child tells her story in a new campaign ad for Democratic presidential nominee Kamala Harris.
South Korea’s government, western countries and adoption agencies worked in tandem to supply some 200,000 Korean children to parents overseas, despite years of evidence they were being procured through questionable or downright unscrupulous means, an investigation led by The Associated Press found.
Chris Knayzeh was in a town overlooking Lebanon's capital when he heard the rumbling aftershock of the 2020 Beirut port blast. Hundreds of tons of haphazardly stored ammonium nitrates had exploded, killing and injuring thousands of people.
Prime Minister Justin Trudeau tapped Treasury Board President Anita Anand to take on additional duties as Canada's minister of transport on Thursday.
Conservative Leader Pierre Poilievre and NDP Leader Jagmeet Singh got into a heated exchange in the House of Commons on Thursday, just minutes after Singh announced his party would not be supporting the Conservatives' first non-confidence motion against Prime Minister Justin Trudeau's government.
The head of Via Rail repeatedly told MPs a train delay over the Labour Day long weekend was an isolated incident, despite a similar event two years ago.
Ontario’s Minister of Sport, Neil Lumsden, will donate his brain to research.
Manitobans are getting another reminder of how long wait times are for diagnostic images.
A new strain of the virus is spreading, largely through skin-to-skin contact, including but not limited to sex. A lack of funds, vaccines and information is making it difficult to stem the spread, according to alarmed disease experts.
Lebanese authorities on Thursday banned walkie-talkies and pagers from being taken on flights from Beirut airport, the National News Agency reported, after thousands of such devices exploded during a deadly attack on Hezbollah this week.
If something looks too good to be true, it might be. That's the message from Saskatchewan horticulturists after customers have come into their stores hoping to buy purple apple trees this month.
NASA scientists have successfully replicated spider-like shapes found on the surface of Mars in a laboratory setting for the first time.
Lawyers representing the producers of 'Russians at War' say they may pursue legal action against Ontario's public broadcaster for pulling support for the controversial documentary amid outcry from the Ukrainian community and some Canadian politicians.
Viewers of Jeopardy got a chance to test their knowledge of trivia about B.C.'s biggest city Wednesday night.
British actress Florence Pugh has revealed that shaving her head for her latest movie role was a 'really bizarre' experience that sent her body 'into a bit of trauma.'
A case involving stolen funds from a Saskatchewan business being used to purchase cryptocurrency will be heading back to the courts, thanks to a new decision by Saskatchewan's Court of Appeal.
Nike Inc. said Thursday it has named Elliott Hill as its president and CEO, replacing John Donahoe, who will retire next month.
Maple Leaf Foods asserted its innocence in an alleged bread price-fixing scheme ahead of a hearing to determine whether it will be added to an ongoing class-action lawsuit.
Only one Toronto restaurant earned a Michelin star this year, but it is not a first for the chef who brought the establishment to stardom.
They say a dog is a man’s best friend. In the case of Darren Cropper, from Bonfield, Ont., his three-year-old Siberian husky and golden retriever mix named Bear literally saved his life.
For most people, dinner on a cruise ship is a time to relax. But when influencer couple Abby and Matt Howard decided to kick back with a dinner à deux, they ended up kicking up a storm.
Shohei Ohtani became the first major league player to hit 50 home runs and steal 50 bases in a season, with the Los Angeles Dodgers star going deep twice to reach the half-century mark and swiping two bags to get to 51 against the Miami Marlins on Thursday.
Vancouver Canucks goalie Thatcher Demko says he's been working his way back from a rare lower-body muscle injury since being sidelined in last season's playoffs.
Toronto Blue Jays shortstop Bo Bichette returned to the 10-day injured list Thursday due to a right middle finger fracture.
Video of a brazen daylight auto theft which shows a suspect running over a victim in a stolen luxury SUV has been released by police west of Toronto.
The plant was expected to produce batteries for a million electric vehicles a year. Once up and running, it was supposed to create hundreds of permanent jobs in a small southeastern Ontario municipality. But two years later, spending on the construction of the Umicore plant has been delayed in what the company calls a "significant worsening of the EV market context."
A Winnipeg man is asking for help after a classic car that has been a part of his family since the 1950s was stolen from his garage.
They say a dog is a man’s best friend. In the case of Darren Cropper, from Bonfield, Ont., his three-year-old Siberian husky and golden retriever mix named Bear literally saved his life.
Paleontologists from the Royal B.C. Museum have uncovered "a trove of extraordinary fossils" high in the mountains of northern B.C., the museum announced Thursday.
The search for a missing ancient 28-year-old chocolate donkey ended with a tragic discovery Wednesday.
The Royal Canadian Mounted Police is celebrating an important milestone in the organization's history: 50 years since the first women joined the force.
It's been a whirlwind of joyful events for a northern Ontario couple who just welcomed a baby into their family and won the $70 million Lotto Max jackpot last month.
A Good Samaritan in New Brunswick has replaced a man's stolen bottle cart so he can continue to collect cans and bottles in his Moncton neighbourhood.
David Krumholtz, known for roles like Bernard the Elf in The Santa Clause and physicist Isidor Rabi in Oppenheimer, has spent the latter part of his summer filming horror flick Altar in Winnipeg. He says Winnipeg is the most movie-savvy town he's ever been in.
Edmontonians can count themselves lucky to ever see one tiger salamander, let alone the thousands one local woman says recently descended on her childhood home.
A daytrip to the backcountry turned into a frightening experience for a Vancouver couple this weekend.
B.C.'s police watchdog is investigating the death of a woman who was shot by the RCMP after allegedly barricading herself in a room with a toddler early Thursday morning.
Since she was a young girl growing up in Vancouver, Ginny Lam says her mom Yat Hei Law made it very clear she favoured her son William, because he was her male heir.
The 'Namgis First Nation says a fish farm it owns near Port McNeill shows the potential of land-based aquaculture in B.C.
An Ontario man says it is 'unfair' to pay a $1,500 insurance surcharge because his four-year-old SUV is at a higher risk of being stolen.
Toronto police say a driver who crashed into a house in the city’s west end early Friday morning has been arrested on suspicion of impaired driving.
The federal firearm buyback program has cost taxpayers nearly $67.2 million since it was announced in 2020, but it still hasn't collected a single gun.
We've produced a special webcast edition of CTV News Calgary for Thursday, Sept. 19, 2024.
The Parole Board of Canada says a man who assaulted a young pregnant woman and left her for dead remains too dangerous to be released into the community.
A Vulcan, Alta., man has been charged with a Lethbridge woman's murder after her body was found in the Oyen area.
Police say a man who was taken to hospital in critical condition after being shot Thursday night has died.
A byelection was called for the Bay of Quinte riding less than a week after Progressive Conservative MPP Todd Smith, Ontario's education minister at the time, resigned his seat in August.
CTVNewsOttawa.ca looks at things to do in Ottawa and eastern Ontario this weekend.
Quebec Premier Francois Legault is calling on the Bloc Quebecois to topple the Trudeau government next Wednesday and trigger a federal election.
The head of Via Rail repeatedly told MPs a train delay over the Labour Day long weekend was an isolated incident, despite a similar event two years ago.
A harrowing video has been presented to a jury at the trial of three men facing first-degree murder charges.
Five bison are dead at Elk Island National Park east of Edmonton following an early Thursday morning collision.
An Edmonton man who is charged with sexually assaulting a teen boy is a member of the Alberta sheriffs, CTV News Edmonton has confirmed.
The mayor of Fort Saskatchewan apologized on Thursday for comments she made earlier this week about killing feral cats.
Gas prices increased in all three Maritime provinces for the first time in more than a month.
A section of Herring Cove Road in the Spryfield area of Halifax is closed to traffic in both directions following a single-vehicle collision Friday morning.
Police are asking people to avoid the Dartmouth Commons area in Dartmouth, N.S., as they assist with a wildlife call for a bear.
Manitobans are continuing to mop up after a deluge of rain hit southern Manitoba earlier in the week.
Shamattawa RCMP are searching for a missing six-year-old boy who hasn’t been seen since Wednesday morning.
A Winnipeg pet rescue is putting out a warning to dog owners across the city about a possible parvovirus in the province.
A social media post of purple apples “growing” in Saskatchewan has sparked a lot of attention. However, garden experts say there's no such thing.
A case involving stolen funds from a Saskatchewan business being used to purchase cryptocurrency will be heading back to the courts, thanks to a new decision by Saskatchewan's Court of Appeal.
Environment and Climate Change Canada (ECCC) has confirmed a tornado touched down Wednesday evening near Langbank, Sask.
Amber and Adam Brueckner have a pool-shaped hole in their backyard – and their wallets.
New details have been shared about a missing family that was last seen on Sept. 1 in Kitchener, Ont.
Police are trying to identify a man who approached a woman and child in Wellesley.
A gym teacher at a private Christian school in Saskatoon has been charged. Terra MacEwan, 44, is charged with assault with a weapon. A Saskatoon mother who spoke with CTV News says her autistic son was MacEwan's victim.
Thomas Hamp says he believed secret police were out to kidnap, torture, and kill him when he fatally stabbed his girlfriend Emily Sanche in February of 2022.
A social media post of purple apples “growing” in Saskatchewan has sparked a lot of attention. However, garden experts say there's no such thing.
They say a dog is a man’s best friend. In the case of Darren Cropper, from Bonfield, Ont., his three-year-old Siberian husky and golden retriever mix named Bear literally saved his life.
As northern Ontario gets ready to welcome autumn this weekend, it’s still feeling a lot like summer as provincial forest fire crews continue to battle blazes.
James Bay Ontario Provincial Police have charged four commercial vehicle drivers with impaired, including one who was hauling 36,000 kilograms of ammonia hydrate.
Police say just before 7:00 p.m. four teens entered a business in the 1000 block of Adelaide Street North.
Police were focusing their attention on the northeast corner of Richmond and Horton, with members of the emergency response unit taking up positions at the scene.
Shedden, southwest of London, hasn’t changed much in decades – but word is spreading that hundreds of new residents could soon be calling the town home.
Two men from Barrie have been charged after a deadly shooting at a park in Keswick on Wednesday.
A growing group of brides and wedding photographers from across the province say they have been taken for tens of thousands of dollars by a Barrie, Ont. wedding photographer.
Ontario Provincial Police executed a search warrant in Walkerton.
The Windsor Police Service (WPS) has seized over $56,000 in drugs and have arrested one suspect.
SERVEONE Canada, a global procurement solution company, has begun Canadian operations in Tecumseh.
Enrolment may be down at St. Clair College, but President Mike Silvaggi isn’t bothered.
In a rare move, the Greater Victoria School District Board of Education has been slapped with a ministerial order from the province requiring it to update a student safety plan – drawing concern around political posturing leading up to an election.
The Union of B.C. Municipalities is asking the provincial government to make transit free for teenagers.
To appreciate why Mimi Vandermolen is so pleased to notice one particular vehicle she’s walking past, you need to know that seeing this many cars in one place would have been unimaginable when she was growing-up in the post-war Netherlands.
A pair of runaway pigs are in the custody of an animal sanctuary in the Okanagan after evading police and volunteers for hours earlier this week.
The Red Bridge, a historic landmark in Kamloops, B.C., was completely destroyed by fire early Thursday morning.
Animal protection officers in British Columbia have rescued three pit bulls – including one that gave birth to 10 puppies – from a rat-infested home in Kelowna.
A Vulcan, Alta., man has been charged with a Lethbridge woman's murder after her body was found in the Oyen area.
A Lethbridge couple got a good reminder as to why you should keep your vehicle doors locked at all times.
Lethbridge residents who live near the police range can expect to hear plenty of shots fired Wednesday and Thursday.
A new community health profile in the Algoma District shows the area is significantly below provincial averages in a number of health metrics.
North Bay Police Service says one person has died following an industrial accident at the Ontario Northland Transportation Commission Rail Yard on Tuesday.
Three northern Ontario residents are charged with drug trafficking after the vehicle they were in got stuck along a bush road off Highway 17 on Monday.
Newfoundland and Labrador's chief medical officer is monitoring the rise of whooping cough infections across the province as cases of the highly contagious disease continue to grow across Canada.
A 16-year-old biennial event aimed at fostering business in the country's eastern Arctic and northern regions has been cancelled indefinitely as a dispute unfolds between Inuit in Canada and a Labrador group claiming to share their heritage.
Jim Lester’s farm hopped on the cow-cuddling trend in early August, and his time slots have been pretty well sold out ever since.
The Shopping Trends team is independent of the journalists at CTV News. We may earn a commission when you use our links to shop. Read about us.
