Go back

Beating GPT-4 on a specialized task

Crowd.dev's goal was to enhance their product by adding sentiment analysis to their existing community analytics. As their customer base are developer tools with billions of community activities, they had to find a cost-efficient, specialized alternative to GPT.

Beating GPT-4 on a specialized task
https://eu-central-1-shared-euc1-02.graphassets.com/A1wsiNXmLQieYs9PaKQmlz/Gym9yet6R8ONR45aMTcf

Crowd Technologies

Crowd.dev is an open-source community-led growth platform designed to assist companies in fostering their developer communities. It serves as a suite of tools that help businesses analyze, grow, and leverage their online communities to drive business outcomes.

Go to website

The Problem - analyzing billions of community activities

Crowd.dev helps customers to monitor community activities across platforms like X (formerly Twitter), LinkedIn, Dev.to, GitHub and many more. Their goal was to add sentiment analysis - i.e. scoring whether a community activity was seen as "positive" or "negative" (e.g. a positive feedback or a negative review) - at large scale.
positive.png

Impossible to run GPT at scale.

Even though GPT is great at differentiating positive and negative reviews, it was - for cost and speed reasons - not feasible to apply GPT to the overall corpus of community activities.

Exploding costs and never-ending analytics.

Running GPT on the scale of crowd.dev's analytics would cost hundreds of millions and would take several years. GPT is not an option.

The Solution - distilling knowledge from GPT into specialized language models

GPT was used to label a medium-sized corpus (~20,000 community activities) within hours and a small bill, resulting in training data for a specialized language model.

Quality assessment on auto pilot.

With our Kern AI refinery, the right data from the community acitivites was selected to identify which data + prompt results in the best training for a specialized language model.
sentiment-screenshot.png

Domain-finetuned sentiment analysis, now on Hugging Face

Give it a try!
screenshot-crowd.png

The Results - building a fast, reliable model on a small budget

  1. Better output that GPT-4 on sentiment analysis
  2. Trained within hours
  3. Few hundred dollars to build the model in total

"The team of Kern AI not only provided software, but also expertise."

We've been in contact with Kern AI months before our plan already, and reached out to them when we thought about improving our current sentiment analysis. The team helped us understand our options and how to build a distilled, specialized language model.

Joan Reyero
Co-Founder & CTO