The process of updating deep learning/AI models when they face new tasks or must accommodate changes in data can have significant costs in terms of computational resources and energy consumption.
The process of updating deep learning/AI models when they face new tasks or must accommodate changes in data can have significant costs in terms of computational resources and energy consumption. Researchers have developed a novel method that predicts those costs, allowing users to make informed decisions about when to update AI models to improve AI sustainability.
"There have been studies that focused on making deep learning model training more efficient," says Jung-Eun Kim, corresponding author of a paper on the work and an assistant professor of computer science at North Carolina State University."However, over a model's life cycle, it will likely need to be updated many times. One reason is that, as our work here shows, retraining an existing model is much more cost effective than training a new model from scratch. "If we want to address sustainability issues related to deep learning AI, we must look at computational and energy costs across a model's entire life cycle -- including the costs associated with updates. If you cannot predict what the costs will be ahead of time, it is impossible to engage in the type of planning that makes sustainability efforts possible. That makes our work here particularly valuable." Training a deep learning model is a computationally intensive process, and users want to go as long as possible without having to update the AI. However, two types of shifts can happen that make these updates inevitable. First, the task that the AI is performing may need to be modified. For example, if a model was initially tasked with only classifying digits and traffic symbols, you may need to modify the task to identify vehicles and humans as well. This is called a task shift. Second, the data users provide to the model may change. For example, you may need to make use of a new kind of data, or perhaps the data you are working with is being coded in a different way. Either way, the AI needs to be updated to accommodate the change. This is called a distribution shift. "Regardless of what is driving the need for an update, it is extremely useful for AI practitioners to have a realistic estimate of the computational demand that will be required for the update," Kim says."This can help them make informed decisions about when to conduct the update, as well as how much computational demand they will need to budget for the update." To forecast what the computational and energy costs will be, the researchers developed a new technique they call the REpresentation Shift QUantifying Estimator . Essentially, RESQUE allows users to compare the dataset that a deep learning model was initially trained on to the new dataset that will be used to update the model. This comparison is done in a way that estimates the computational and energy costs associated with conducting the update. Those costs are presented as a single index value, which can then be compared with five metrics: epochs, parameter change, gradient norm, carbon and energy. Epochs, parameter change and gradient norm are all ways of measuring the amount of computational effort necessary to retrain the model. "However, to provide insight regarding what this means in a broader sustainability context, we also tell users how much energy, in kilowatt hours, will be needed to retrain the model," Kim says."And we predict how much carbon, in kilograms, will be released into the atmosphere in order to provide that energy." The researchers conducted extensive experiments involving multiple data sets, many different distribution shifts, and many different task shifts to validate RESQUE's performance. "We found that the RESQUE predictions aligned very closely with the real-world costs of conducting deep learning model updates," Kim says."Also, as I noted earlier, all of our experimental findings tell us that training a new model from scratch demands far more computational power and energy than retraining an existing model." In the short term, RESQUE is a useful methodology for anyone who needs to update a deep learning model. "RESQUE can be used to help users budget computational resources for updates, allow them to predict how long the update will take, and so on," Kim says. "In the bigger picture, this work offers a deeper understanding of the costs associated with deep learning models across their entire life cycle, which can help us make informed decisions related to the sustainability of the models and how they are used. Because if we want AI to be viable and useful, these models must be not only dynamic but sustainable."Researchers have developed a new framework to make complex AI tasks more accessible and transparent to users. The framework specializes in providing solutions to deep reinforcement learning ... The Data Provenance Explorer can help machine-learning practitioners make more informed choices about the data they train their models on, which could improve the accuracy of models deployed in the ... Researchers have explained how large language models like GPT-3 are able to learn new tasks without updating their parameters, despite not being trained to perform those tasks. They found that these ... Researchers developed a system that streamlines the process of federated learning, a technique where users collaborate to train a machine-learning model in a way that safeguards each user's data. The ...COMPUTERS & MATH
Energy Technology Energy Policy Petroleum Computer Modeling Mathematical Modeling Computational Biology Mathematics
Trending
A gorgeous April afternoon in store across the Denver metro area 
‘Artemis Mission Cannot Lead To Interplanetary Wild West,’ Astronomer Warns 
Trump says US forces will ‘finish the job’ soon in first prime-time speech since starting Iran war 
Former Wisconsin football player, who left the sport amid mental health struggles, dead at 24 
Drew McIntyre Gives Honest Take About His Recent WWE Title Reign 
U.S. Sen. Bernie Sanders introduces bill that could keep the Padres in San Diego United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
New York Times Launches 'Strands': A Tricky New Word Search PuzzleThe New York Times introduces 'Strands', a challenging daily word search puzzle that tests your vocabulary and observation skills. Learn how to play, find hints for today's puzzle, and explore the solved puzzle to sharpen your word-finding abilities.
Read more »
New Jersey Minimum Wage Hikes and New Laws Taking Effect in 2025Several new laws will come into effect in New Jersey starting in 2025, including a minimum wage increase, transparency in promotional opportunities, and data protection regulations.
Read more »
New York company chosen to make new crystals for NYC New Year's Eve ballThis year, one local New York company will have some bragging rights as we count down to 2025 in Times Square.
Read more »
Florida condo owners look at higher costs as new regulations take effect in the new yearFlorida condominium owners are looking at higher costs from condo associations in the new year. That's a consequence of a Florida condo safety law passed by state lawmakers in 2022. It requires condo associations to have sufficient reserves to cover major repairs and to conduct a survey of reserves every decade.
Read more »
Pop-Tarts Bowl: New Name, New Popularity, New Viral MomentsThe Pop-Tarts Bowl has become one of the most popular postseason games thanks to its catchy name and creative marketing strategies. This year's game saw Iowa State quarterback Rocco Becht become an internet sensation, winning MVP and receiving a massive Pop-Tart trophy.
Read more »
New Year, New Laws in New York: Minimum Wage Hike, Paid Pregnancy Leave and MoreNew York kicks off the new year with impactful legislation aimed at improving the lives of residents. These changes include a minimum wage increase in Western New York, paid time off for pregnancy-related appointments, lower insulin costs, and heat mitigation plans in schools.
Read more »