DeepSeek unveils new technique for smarter, scalable AI reward models

Deepseek aiA Chinese Research Laboratory for powerful open source language models such as DeepSEEK-R1, presented an important progress in award modeling for a large language models (LLS).

Their new technique aims to create a unique critical regulation (SPCT), generally and expanding premium models (RMS). This can lead to more skilled AI applications for open workers and domains where the current models cannot seize the nuance and complexity of the environment and users.

The decisive role and current limits of premium models

The learning of strengthening (RL) has become a foundation stone in the development of the most modern LLMS. In RL, models are beautifully regulated based on feedback signals showing the quality of answers.

Reward models are a critical component that provides these signals. In fact, an RM acts as a judge, evaluates LLM outputs and calls the RL process or “reward” or “prize” and teaches the LLM to make more useful answers.

However, the current RMS often faces restrictions. Usually, they are clearly cut in narrow domains with cut rules or easily checked answers. For example, the most modern main reasoning models DeepSEek-R1 up to a RL stagewhere the truth is trained in accurate mathematics and coding problems.

However, creating a premium model for complex, open or subjective questions in general domains remains a great obstacle. In paper Explaining new techniques, explaining the researchers, “Generally, the awards require high quality awards outside of special domains where the premises are not more different and complexity, and there is no open reference or truth.”

The generalist, which can handle extensive assignments, emphasizes four main problems in creating RMS:

1. Ease of entry: RM must handle various types of access and can be able to assess one or more answers at the same time.
2. Accuracy: The criteria should create accurate prize signals in various fields where the truth of the soil is often available.
3. Next time measurement: RM must give more high quality rewards when more calculation sources are separated during the inferences.
4. Learn sized behaviors: In an effective time for RMS, it must learn the behavior that allows you to perform improved due to more computing, as more calculation is used.

Reward models can be widely classified with “award-winning part of the award-winning” (for example, a generation of RMS, generation regulating generation, generation) (eg PointWicWice goal selects two answers to each other). These design options affect the compliance of the model for general, especially its common tasks Ease of entry and potential scalting time.

For example, the simple scalar RMS will be able to repeat the same account, because the RMS of the Charin is comparing with recyclubes because they cannot easily appreciate the unified answers.

Researchers offer the model’s text criticism and the PointWise Neadwe Neadwe Neadwise Reward Modeling “(GRM) acquired by them.

DeepSeek team conducted primary experience on models such as GPT-4O and GEMMA-2-27B and said “improving the quality of certain principles, improving the results of high quality principles and accurate criticism.”

RMS training to create its own principles

According to these findings, researchers have developed a self-principled criticism Tuning (SPCT) preparing the GRM to create principles and criticism based on surveys and answers.

Researchers suggest that the principles “must be part of the premium generation instead of performing pre-processing.” In this way, the GRMS can create principles in the flies for their assessment, and then creates criticism based on principles.

“This change allows [the] Principles to be created on the basis of input request and answers, adaptation [the] The fact that the premium is the process of creating and the quality and future of principles and relevant criticisms in the Grm, “researchers write.

SPCT covers two main stages:

1. Dear subtle adjustment: This phase is preparing for the principles and criticism for various entry types using the correct format. The model creates principles, criticisms and prizes for the requested requests / answers. Trajectories (generation attempts) are accepted only if the predicted premium is compatible with the truths (for example, it is correctly determined) and otherwise rejected. This process is repeated and the model is thin to improve the principle / critical generation capabilities over filtered samples.
2. Rule Based RL: At this stage, the model is even more beautiful by learning the resulting strengthening. Grm creates principles and criticism for each query and the premium signals are calculated according to simple accuracy rules (eg, for example, the known best answer?). Then the model is updated. This encourages you to learn to create effective principles and accurate criticism in GRM in a dynamic and expandable way.

“Using the rules-based online RL, the SPCT allows the principles of GRM based on the principles and criticism based on prompts and answers, the common domains,” researchers are writing.

Conclusive-solving the scale problem (to get better results with calculation), researchers work many times more than Grm to create a set of various entrances and criticism. The latest premium is determined by voting (collecting sample points). This allows you to consider more prospects for more accurate and nuanced recent decisions because the model is more resources.

However, some created principles / criticism may be poor quality or bias due to model restrictions or random. To solve this, researchers presented “Meta” RM “- A separate, light-scald scalar is specially designed to predict a principle created by the RM, the main Grm, probably the correct final premium.

During the presentation, the Meta RM appreciates the samples created and increases scale performance before low-quality judgments before the last vote.

To practice with DeepSseek-Grm

Researchers have applied SPCT Gemma-2-27BGoogle’s open weight model creates DeepSeek-GRM-27B. Bunu bir neçə meyildə (llm-aaaa-hak, skalar rms və yarı skalar rms) və ictimai modellərdə (GPT-4O və Nemotron-4-340B-4-340B-4-340B-4-340B-4-340B kimi) qarşı bir neçə güclü əsas rm (Scalar RMS) və ictimai modelləri (GPT-4O və Nemotron-4-340B) qarşı qiymətləndirdilər.

DeepSEEK-GRM-27B found that the initial methods trained in the same information. SPCT, compared to standard fine regulation, has improved significantly with the end of the quality and at the highest level.

The performance of DeepSeek-GRM-27B, which creates more samples, increased by exceeding larger models Nemotron-4-340B-award and GPT-4O. Meta RM has improved the expansion by filtering judgments and achieving the best results.

“With more widespread selection, DeepSeek-Grm can judge principles more accurately with higher diversity, and the awards, which are more subtle granulicity,” researchers write.

Interestingly, the SPCT has shown a good job in frequent check-in positions, but better worked in other places in different areas compared to RMS.

Effects for the enterprise

The development facility of more generalists and expandable premiums can be promised for AI applications. Potential areas that can benefit from the generalized RMS include creative tasks and applications that are adapted to dynamic environments such as the model developing customer options.

Despite the strongest results, DeepSeek-GRM is behind RMS of specialized skads, which can be less effective than a generation of open meditation directly. Efficiency remains a problem compared to RMS that is not a generational RMS.

The DeepSseek team offers to improve the efficiency of future work and pay attention to deeper integration. As a result, future directions can integrate the versatile interfaces of Revel systems, which adapt to GRMS online RL pipelines, including policy models, or serve as solid offline evaluators for basic models. ”