Google is continuing its aggressive Gemini updates as it races towards its 2.0 model.
The company today announced a smaller variant of Gemini 1.5, Gemini 1.5 Flash-8B, alongside a “significantly improved” Gemini 1.5 Flash and a “stronger” Gemini 1.5 Pro. These show increased performance against many internal benchmarks, the company says, with “huge gains” with 1.5 Flash across the board and a 1.5 Pro that is much better at math, coding, and complex prompts.
“Gemini 1.5 Flash is the best… in the world for developers right now,” Logan Kilpatrick, product lead for Google AI Studio, boasted in a post on X.
‘Newest experimental iteration’ of ‘unprecedented’ Gemini models
Google introduced Gemini 1.5 Flash — the lightweight version of Gemini 1.5 — in May. The Gemini 1.5 family of models was built to handle long contexts and can reason over fine-grained information from 10M and more tokens. This allows the models to process high-volume multimodal inputs including documents, video and audio.
Today, Google is making available an “improved version” of a smaller 8 billion parameter variant of Gemini 1.5 Flash. Meanwhile, the new Gemini 1.5 Pro shows performance gains on coding and complex prompts and serves as a “drop-in replacement” to its previous model released in early August.
Kilpatrick was light on additional details, saying that Google will make a future version available for production use in the coming weeks that “hopefully will come with evals!”
He explained in an X thread that the experimental models are a means to gather feedback and get the latest, ongoing updates into the hands of developers as quickly as possible. “What we learn from experimental launches informs how we release models more widely,” he posted.
The “newest experimental iteration” of both Gemini 1.5 Flash and Pro feature 1 million token limits and are available to test for free via Google AI Studio and Gemini API, and also soon through the Vertex AI experimental endpoint. There is a free tier for both and the company will make available a future version for production use in coming weeks, according to Kilpatrick.
Beginning Sept. 3, Google will automatically reroute requests to the new model and will remove the older model from Google AI Studio and the API to “avoid confusion with keeping too many versions live at the same time,” said Kilpatrick.
“We are excited to see what you think and to hear how this model might unlock even more new multimodal use cases,” he posted on X.
Google DeepMind researchers call Gemini 1.5’s scale “unprecedented” among contemporary LLMs.
“We have been blown away by the excitement for our initial experimental model we released earlier this month,” Kilpatrick posted on X. “There has been lots of hard work behind the scenes at Google to bring these models to the world, we can’t wait to see what you build!”
‘Solid improvements,’ still suffers from ‘lazy coding disease’
Just a few hours after the release today, the Large Model Systems Organization (LMSO) posted a leaderboard update to its chatbot arena based on 20,000 community votes. Gemini 1.5-Flash made a “huge leap,” climbing from 23rd to sixth place, matching Llama levels and outperforming Google’s Gemma open models.
Gemini 1.5-Pro also showed “strong gains” in coding and math and “improved [d] significantly.”
The LMSO lauded the models, posting: “Big congrats to Google DeepMind Gemini team on the incredible launch!”
As per usual with iterative model releases, early feedback has been all over the place — from sycophantic praise to mockery and confusion.
Some X users questioned why so many back-to-back updates versus a 2.0 version. One posted: “Dude this isn’t going to cut it anymore :| we need Gemini 2.0, a real upgrade.”
On the other hand, many self-described fanboys lauded the fast upgrades and quick shipping, reporting “solid improvements” in image analysis. “The speed is fire,” one posted, and another pointed out that Google continues to ship while OpenAI has effectively been quiet. One went so far as to say that “the Google team is silently, diligently, and constantly delivering.”
Some critics, though, call it “terrible,” and “lazy” with tasks requiring longer outputs, saying Google is “far behind” Claude, OpenAI, and Anthropic.
The update “sadly suffers from the lazy coding disease” is similar to GPT-4 Turbo, one X user lamented.
Another called the updated version “definitely not that good” and said it “often goes crazy and starts repeating stuff non-stop like small models tend to do.” Another agreed that they were excited to try it but that Gemini has “been by far the worst at coding.”
Some also poked fun at Google’s uninspired naming capabilities and called back to its huge woke blunder earlier this year.
“You guys have completely lost the ability to name things,” one user joked, and another agreed, “You guys seriously need someone to help you with nomenclature.”
And, one dryly asked: “Does Gemini 1.5 still hate white people?”
Earlier this summer, the San Francisco-based AI startup Anthropic — a leading rival of OpenAI when it comes to developing useful new large language models (LLMs) — unveiled a surprise new feature it called “Artifacts.”
The feature allowed users of Anthropic’s Claude family of LLMs and chatbots on the web to enable a new window that would appear alongside their chat interface and run code snippets and even full programs generated by the LLM at the user’s request.
For example, a user could ask Claude to generate a simple interactive visualization, chart, or a playable game and run it alongside the chatbot right in their browser. The feature was impressive enough that VentureBeat editorial director Michael Nuñez called it “this year’s most important AI feature” and users have generated tens of millions of Artifacts since its release, according to Anthropic.
However, users previously had to turn on Claude Artifacts manually by clicking their username initials in the lower left corner of the Claude chatbot screen, selecting “Feature Preview” and toggling on Artifacts. But no more: Anthropic today announced the general availability of Artifacts across its Free, Pro, and Team tiers, as well as its availability on the official Claude iOS and Android mobile apps, making it easier to create and interact with interactive code on the go.
Anthropic’s Head of Developer Relations Alex Albert posted on the social network X that he spent “all morning replicating simple games with Claude. We’re nearing the era of mobile apps created in real-time by LLMs.”
Only Free and Pro plans will have the ability to publish and remix Artifacts with the broader Claude community. This feature allows users to build upon and iterate on content created by others worldwide, facilitating a dynamic exchange of ideas and resources.
For users on the Team plan, Artifacts can be shared within Projects, enabling secure and efficient collaboration among team members.
Anthropic expects Artifacts to streamline workflows and enhance productivity by allowing teams to use it collaboratively and iterate on one another’s creations securely over the web.
The strategy behind Claude Artifacts: user experience over raw power?
While much of the AI development world has focused on enhancing raw processing power and expanding model capabilities, Artifacts represents a focus on user experience and redesigning AI interfaces away from the simple chatbot model.
In a way, I believe it is analogous to Nintendo in gaming — the company often comes out with game consoles that have much less processing and graphics power than rivals at Microsoft and Sony but seeks to gain users with novel user interfaces, often to great success.
Anthropic envisions Artifacts as a versatile tool that can be utilized by teams across various industries to accelerate the creation of high-quality work products.
The platform supports a wide range of outputs, including code snippets, flowcharts, SVG graphics, websites, and interactive dashboards.
For instance, developers can now create architecture diagrams directly from their codebases, product managers can develop interactive prototypes for rapid feature testing, and designers can produce visualizations for quick prototyping. Similarly, marketers can design campaign dashboards rich with performance metrics, while sales teams can visualize their pipelines and forecast insights more effectively.
As Artifacts become a standard part of the Claude experience, Anthropic anticipates seeing a wide array of creative and practical uses emerge from its global user base.
On a hot summer day in Oak Ridge, Tennessee, dozens of men removed pipes, asbestos, and hazardous waste while working to decontaminate a nuclear facility and prepare it for demolition.
Dressed in head-to-toe coveralls and fitted with respirators, the crew members toiling in a building without power had no obvious respite from the heat. Instead, they wore armbands that recorded their heart rates, movements, and exertion levels for signs of heat stress.
Stephanie Miller, a safety and health manager for a U.S. government contractor doing cleanup work at the Oak Ridge National Laboratory, watched a computer screen nearby. A color-coding system with little bubbles showing each worker’s physiological data alerted her if anyone was in danger of overheating.
“Heat is one of the greatest risks that we have in this work, even though we deal with high radiation, hazardous chemicals, and heavy metals,” Miller said.
As the world experiences record-high temperatures, employers are exploring wearable technologies to keep workers safe. New devices collect biometric data to estimate core body temperature - an elevated one is a symptom of heat exhaustion - and prompt workers to take cool-down breaks.
The devices, which were originally developed for athletes, firefighters, and military personnel, are getting adopted at a time when the Atlantic Council estimates heat-induced losses in labor productivity could cost the U.S. approximately $100 billion annually.
However, there are concerns about how the medical information collected on employees will be safeguarded. Some labor groups worry managers could use it to penalize people for taking needed breaks.
“Any time you put any device on a worker, they’re very concerned about tracking, privacy, and how are you going to use this against me,” said Travis Parsons, director of occupational safety and health at the Laborers’ Health and Safety Fund of North America. “There’s a lot of exciting stuff out there, but there’s no guardrails around it.”
VULNERABLE TO HEAT
At the Tennessee cleanup site, the workers wearing heat stress monitors made by Atlanta company SlateSafety are employed by United Cleanup Oak Ridge. The company is a contractor of the U.S. Department of Energy, which has rules to prevent on-the-job overheating.
But most U.S. workers lack protections from extreme heat because there are no federal regulations requiring them, and many vulnerable workers don’t speak up or seek medical attention. In July, the Biden administration proposed a rule to protect 36 million workers from heat-related illnesses.
From 1992 to 2022, 986 workers died from heat exposure in the U.S., according to the Environmental Protection Agency. Experts suspect the number is higher because a coroner might not list heat as the cause of death if a sweltering roofer takes a fatal fall.
Setting occupational safety standards can be tricky because individuals respond differently to heat. That’s where the makers of wearable devices hope to come in.
HOW WEARABLE HEAT TECH WORKS
Employers have observed workers for heat-related distress by checking their temperatures with thermometers, sometimes rectally. More recently, firefighters and military personnel swallowed thermometer capsules.
“That just was not going to work in our work environment,” Rob Somers, global environment, health and safety director at consumer product company Perrigo, said.
Instead, more than 100 employees at the company’s infant formula plants were outfitted with SlateSafety armbands. The devices estimate a wearer’s core body temperature and a reading of 101.3 degrees triggers an alert.
Another SlateSafety customer is a Cardinal Glass factory in Wisconsin, where four masons maintain a furnace that reaches 3000 degrees Fahrenheit.
“They’re right up against the face of the wall. So it’s them and fire,” Jeff Bechel, the company’s safety manager, said.
Cardinal Glass paid $5,000 for five armbands, software and air-monitoring hardware. Bechel thinks the investment will pay off; an employee’s two heat-related emergency room visits cost the company $15,000.
Another wearable, made by Massachusetts company Epicore Biosystems, analyzes sweat to determine when workers are at risk of dehydration and overheating.
“Until a few years ago, you just sort of wiped (sweat) off with a towel,” CEO Rooz Ghaffari said. “Turns out there’s all this information packed away that we’ve been missing.”
Research has shown some devices successfully predict core body temperature in controlled environments, but their accuracy remains unproven in dynamic workplaces, according to experts. A 2022 research review said factors such as age, gender, and ambient humidity make it challenging to reliably gauge body temperature with the technology.
The United Cleanup Oak Ridge workers swathed in protective gear can get sweaty even before they begin demolition. Managers see dozens of sensor alerts daily.
Laborer Xavier Allison, 33, was removing heavy pieces of ductwork during a recent heat wave when his device vibrated. Since he was working with radioactive materials and asbestos, he couldn’t walk outside to rest without going through a decontamination process, so he spent about 15 minutes in a nearby room which was just as hot.
“You just sit by yourself and do your best to cool off,” Allison said.
The armband notifies workers when they’ve cooled down enough to resume work.
“Ever since we implemented it, we have seen a significant decrease in the number of people who need to get medical attention,” Miller said.
COLLECTING PERSONAL DATA
United Cleanup Oak Ridge uses the sensor data and an annual medical exam to determine work assignments, Miller said. After noticing patterns, the company sent a few employees to see their personal physicians, who found heart issues the employees hadn’t known about, she said.
At Perrigo, managers analyze the data to find people with multiple alerts and speak to them to see if there’s “a reason why they’re not able to work in the environment,” Somers said. The information is organized by identification numbers, not names, when it goes into the company’s software system, he said.
Companies keeping years of medical data raises privacy concerns and whether bosses may use the information to kick an employee off a health plan or fire them, said Adam Schwartz, privacy litigation director at the Electronic Frontier Foundation.
“The device could hurt, frankly, because you could raise your hand and say ‘I need a break,’ and the boss could say, ‘No, your heart rate is not elevated, go back to work,’” Schwartz said.
To minimize such risks, employers should allow workers to opt in or out of wearing monitoring devices, only process strictly necessary data and delete the information within 24 hours, he said.
Wearing such devices also may expose workers to unwanted marketing, Ikusei Misaka, a professor at Tokyo’s Musashino University, said.
A PARTIAL SOLUTION
The National Institute for Occupational Safety and Health advises employers to institute a plan to help workers adjust to hot conditions and to train them to recognize signs of heat-related illness and administer first aid. Wearable devices can be part of efforts to reduce heat stress, but more work needs to be done to determine their accuracy, said Doug Trout, the agency’s medical officer.
The technology also needs to be paired with access to breaks, shade, and cool water, since many workers, especially in agriculture, fear retaliation for pausing to cool off or hydrate.
“If they don’t have water to drink, and the time to do it, it doesn’t mean much,” Juanita Constible, senior advocate at the Natural Resources Defense Council, said. “It’s just something extra they have to carry when they’re in the hot fields.”