Evaluasi Azure OpenAI (Pratinjau)

Evaluasi model bahasa besar adalah langkah penting dalam mengukur performanya di berbagai tugas dan dimensi. Ini sangat penting untuk model yang disempurnakan, di mana menilai perolehan performa (atau kerugian) dari pelatihan sangat penting. Evaluasi menyeluruh dapat membantu pemahaman Anda tentang bagaimana versi model yang berbeda dapat memengaruhi aplikasi atau skenario Anda.

Evaluasi Azure OpenAI memungkinkan pengembang membuat eksekusi evaluasi untuk menguji pasangan input/output yang diharapkan, menilai performa model di seluruh metrik utama seperti akurasi, keandalan, dan performa keseluruhan.

Dukungan evaluasi

Ketersediaan regional

• US Timur2
• AS Tengah Bagian Utara
• Swedia Tengah
• Swiss Barat

Jenis penyebaran yang didukung

• Standard
• Standar global
• Standar zona data
• Dikelola yang disediakan
• Terkelola yang disediakan global
• Zona data yang dikelola yang disediakan

Alur evaluasi

Menguji data

Anda perlu merakit himpunan data kebenaran dasar yang ingin Anda uji. Pembuatan himpunan data biasanya merupakan proses berulang yang memastikan evaluasi Anda tetap relevan dengan skenario Anda dari waktu ke waktu. Himpunan data kebenaran dasar ini biasanya dibuat dan mewakili perilaku yang diharapkan dari model Anda. Himpunan data juga diberi label dan menyertakan jawaban yang diharapkan.

Sumber data Anda harus dalam format JSONL. Di bawah ini adalah dua contoh himpunan data evaluasi JSONL:

Format evaluasi

{"question": "Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q.", "subject": "abstract_algebra", "A": "0", "B": "4", "C": "2", "D": "6", "answer": "B", "completion": "B"}
{"question": "Let p = (1, 2, 5, 4)(2, 3) in S_5 . Find the index of <p> in S_5.", "subject": "abstract_algebra", "A": "8", "B": "2", "C": "24", "D": "120", "answer": "C", "completion": "C"}
{"question": "Find all zeros in the indicated finite field of the given polynomial with coefficients in that field. x^5 + 3x^3 + x^2 + 2x in Z_5", "subject": "abstract_algebra", "A": "0", "B": "1", "C": "0,1", "D": "0,4", "answer": "D", "completion": "D"}

{"input": [{"role": "system", "content": "Provide a clear and concise summary of the technical content, highlighting key concepts and their relationships. Focus on the main ideas and practical implications."}, {"role": "user", "content": "Tokenization is a key step in preprocessing for natural language processing, involving the division of text into smaller components called tokens. These can be words, subwords, or characters, depending on the method chosen. Word tokenization divides text at word boundaries, while subword techniques like Byte Pair Encoding (BPE) or WordPiece can manage unknown words by breaking them into subunits. Character tokenization splits text into individual characters, useful for multiple languages and misspellings. The tokenization method chosen greatly affects model performance and its capacity to handle various languages and vocabularies."}], "output": "Tokenization divides text into smaller units (tokens) for NLP applications, using word, subword (e.g., BPE), or character methods. Each has unique benefits, impacting model performance and language processing capabilities."}      
{"input": [{"role": "system", "content": "Create a comprehensive yet concise summary that captures the essential technical details and significance of the topic."}, {"role": "user", "content": "Self-attention mechanisms are vital in modern transformer models, allowing them to evaluate the relevance of different parts of an input sequence when processing each element. In self-attention, each position in a sequence learns to focus on all other positions, forming a weighted sum of their values. Attention weights are derived using queries, keys, and values, with compatibility between a query and key determining attention to each value. This mechanism enables capturing long-range dependencies and relationships within a sequence, making it effective for tasks needing context understanding and inter-part relationships."}], "output": "Self-attention is crucial in transformers, enabling each sequence position to evaluate and collect data from all others. It uses queries, keys, and values for attention weights, effectively capturing long-range dependencies and contextual links."}      
{"input": [{"role": "system", "content": "Synthesize the technical information into a brief, accurate summary that maintains the core concepts and their significance."}, {"role": "user", "content": "The transformer architecture significantly changed natural language processing by introducing a new method for sequence processing. Unlike recurrent neural networks, transformers process whole sequences at once using self-attention. The architecture consists of an encoder and decoder, each with multiple layers of multi-head attention, feed-forward networks, and normalization layers. The encoder processes the input sequence, and the decoder produces the output sequence. Positional encodings are added to retain sequence order information since the model lacks inherent sequence position awareness. This architecture underlies many leading language models like BERT and GPT."}], "output": "Transformers are a transformative neural architecture processing sequences with self-attention instead of recurrence. Their encoder-decoder design, with multi-head attention and position encodings, is fundamental to modern language models like BERT and GPT."}      

Anda dapat memilih himpunan data yang sudah ada yang diunggah sebelumnya, atau mengunggah himpunan data baru.

Membuat respons (opsional)

Perintah yang Anda gunakan dalam evaluasi harus sesuai dengan permintaan yang Anda rencanakan untuk digunakan dalam produksi. Perintah ini memberikan instruksi untuk diikuti model. Mirip dengan pengalaman taman bermain, Anda dapat membuat beberapa input untuk menyertakan contoh beberapa bidikan dalam perintah Anda. Untuk informasi selengkapnya, lihat teknik rekayasa yang tepat untuk detail tentang beberapa teknik canggih dalam desain cepat dan rekayasa yang cepat.

Anda dapat mereferensikan data input Anda dalam perintah dengan menggunakan {{input.column_name}} format , di mana column_name sesuai dengan nama kolom dalam file input Anda.

Output yang dihasilkan selama evaluasi akan direferensikan dalam langkah-langkah berikutnya menggunakan {{sample.output_text}} format .

Penyebaran model

Sebagai bagian dari membuat evaluasi, Anda akan memilih model mana yang akan digunakan saat menghasilkan respons (opsional) serta model mana yang akan digunakan saat menilai model dengan kriteria pengujian tertentu.

Di Azure OpenAI, Anda akan menetapkan penyebaran model tertentu untuk digunakan sebagai bagian dari evaluasi Anda. Anda dapat membandingkan beberapa penyebaran model dalam eksekusi evaluasi tunggal.

Anda dapat mengevaluasi penyebaran model dasar atau yang disempurnakan. Penyebaran yang tersedia dalam daftar Anda bergantung pada penyebaran yang Anda buat dalam sumber daya Azure OpenAI Anda. Jika Anda tidak dapat menemukan penyebaran yang diinginkan, Anda dapat membuat penyebaran baru dari halaman Evaluasi Azure OpenAI.

Kriteria pengujian

Kriteria pengujian digunakan untuk menilai efektivitas setiap output yang dihasilkan oleh model target. Pengujian ini membandingkan data input dengan data output untuk memastikan konsistensi. Anda memiliki fleksibilitas untuk mengonfigurasi kriteria yang berbeda untuk menguji dan mengukur kualitas dan relevansi output pada tingkat yang berbeda.

Memulai

1. Pilih Evaluasi Azure OpenAI (PRATINJAU) dalam portal Azure AI Foundry. Untuk melihat tampilan ini sebagai opsi mungkin perlu terlebih dahulu memilih sumber daya Azure OpenAI yang ada di wilayah yang didukung.

2. Pilih Evaluasi baru

   

3. Masukkan nama evaluasi Anda. Secara default, nama acak dibuat secara otomatis kecuali Anda mengedit dan menggantinya. Pilih Unggah himpunan data baru.

   

4. Pilih evaluasi Anda yang akan dalam .jsonl format. Jika Anda memerlukan sampel file pengujian, Anda dapat menyimpan 10 baris ini ke file yang disebut eval-test.jsonl:

   {"input": [{"role": "system", "content": "Provide a clear and concise summary of the technical content, highlighting key concepts and their relationships. Focus on the main ideas and practical implications."}, {"role": "user", "content": "Tokenization is a key step in preprocessing for natural language processing, involving the division of text into smaller components called tokens. These can be words, subwords, or characters, depending on the method chosen. Word tokenization divides text at word boundaries, while subword techniques like Byte Pair Encoding (BPE) or WordPiece can manage unknown words by breaking them into subunits. Character tokenization splits text into individual characters, useful for multiple languages and misspellings. The tokenization method chosen greatly affects model performance and its capacity to handle various languages and vocabularies."}], "output": "Tokenization divides text into smaller units (tokens) for NLP applications, using word, subword (e.g., BPE), or character methods. Each has unique benefits, impacting model performance and language processing capabilities."}      
   {"input": [{"role": "system", "content": "Create a comprehensive yet concise summary that captures the essential technical details and significance of the topic."}, {"role": "user", "content": "Self-attention mechanisms are vital in modern transformer models, allowing them to evaluate the relevance of different parts of an input sequence when processing each element. In self-attention, each position in a sequence learns to focus on all other positions, forming a weighted sum of their values. Attention weights are derived using queries, keys, and values, with compatibility between a query and key determining attention to each value. This mechanism enables capturing long-range dependencies and relationships within a sequence, making it effective for tasks needing context understanding and inter-part relationships."}], "output": "Self-attention is crucial in transformers, enabling each sequence position to evaluate and collect data from all others. It uses queries, keys, and values for attention weights, effectively capturing long-range dependencies and contextual links."}      
   {"input": [{"role": "system", "content": "Synthesize the technical information into a brief, accurate summary that maintains the core concepts and their significance."}, {"role": "user", "content": "The transformer architecture significantly changed natural language processing by introducing a new method for sequence processing. Unlike recurrent neural networks, transformers process whole sequences at once using self-attention. The architecture consists of an encoder and decoder, each with multiple layers of multi-head attention, feed-forward networks, and normalization layers. The encoder processes the input sequence, and the decoder produces the output sequence. Positional encodings are added to retain sequence order information since the model lacks inherent sequence position awareness. This architecture underlies many leading language models like BERT and GPT."}], "output": "Transformers are a transformative neural architecture processing sequences with self-attention instead of recurrence. Their encoder-decoder design, with multi-head attention and position encodings, is fundamental to modern language models like BERT and GPT."}      
   {"input": [{"role": "system", "content": "Distill the technical content into a clear, concise summary that preserves the most important concepts and their practical applications."}, {"role": "user", "content": "Cosine similarity is a metric for measuring similarity between vectors by calculating the cosine of the angle between them. The outcome ranges from -1 to 1, with 1 indicating identical orientation, 0 orthogonality, and -1 opposite orientation. In machine learning and NLP, cosine similarity is useful for comparing document vectors, word embeddings, or any high-dimensional representations. It's often preferred over Euclidean distance because it's invariant to vector magnitude, focusing instead on orientation. This makes it particularly useful for comparing documents or embeddings of various lengths or scales."}], "output": "Cosine similarity evaluates vector similarity by computing their angle's cosine, with values from -1 to 1. It's widely used in ML and NLP for comparing document vectors and embeddings due to its magnitude-invariant nature."}      
   {"input": [{"role": "system", "content": "Provide a succinct summary that captures the key technical concepts and their relationships while maintaining clarity and accuracy."}, {"role": "user", "content": "Neural network activation functions are crucial for introducing non-linearity to deep learning models. Commonly used activation functions include ReLU (Rectified Linear Unit), which outputs the input directly if positive and zero otherwise, sigmoid, which compresses values between 0 and 1, and tanh, which normalizes values between -1 and 1. Each function has specific benefits and applications: ReLU avoids vanishing gradients and enables sparse activation, sigmoid is useful for binary classification, and tanh is often used in hidden layers for its zero-centered output. The activation function choice greatly affects model training dynamics and ultimate performance."}], "output": "Activation functions introduce non-linearity to neural networks, with ReLU, sigmoid, and tanh being the most common. Each serves distinct purposes: ReLU prevents vanishing gradients, sigmoid is suitable for binary classification, and tanh offers zero-centered output, all affecting model performance differently."}      
   {"input": [{"role": "system", "content": "Create a focused summary that emphasizes the main technical concepts and their practical implications while maintaining accuracy."}, {"role": "user", "content": "Batch normalization is a technique that stabilizes and accelerates deep neural network training by normalizing each layer's inputs. It normalizes activations using the mean and variance computed over the current data batch. This normalization reduces internal covariate shift, where the input distribution changes during training. Batch normalization includes learnable parameters (gamma and beta) allowing the network to reverse normalization if necessary. Benefits include enabling higher learning rates, lessening the importance of precise parameter initialization, and serving as a regularizer, potentially removing the need for dropout."}], "output": "Batch normalization stabilizes neural network training by normalizing layer inputs using batch statistics. It addresses internal covariate shift with learnable parameters, allowing higher learning rates and improved training dynamics, while offering regularization benefits."}      
   {"input": [{"role": "system", "content": "Summarize the technical content while preserving the essential concepts and their interconnections, ensuring clarity and precision."}, {"role": "user", "content": "Gradient descent optimization algorithms are foundational to training machine learning models. The basic version updates parameters in the steepest descent direction of the loss function. Variants like Stochastic Gradient Descent (SGD) use random samples to estimate gradients, enhancing computational efficiency. Advanced optimizers like Adam combine momentum (helping overcome local minima) and RMSprop (adapting learning rates for each parameter). These optimizers maintain running averages of gradients and their squares, enabling adaptive learning rates and more efficient training. The optimizer choice can significantly influence model convergence speed and final performance."}], "output": "Gradient descent algorithms optimize ML models by updating parameters based on loss gradients. Variants like SGD use sampling for efficiency, while advanced optimizers like Adam combine momentum with adaptive learning rates for improved training performance."}      
   {"input": [{"role": "system", "content": "Deliver a concise yet comprehensive summary that highlights the key technical aspects and their practical significance."}, {"role": "user", "content": "Cross-validation is a resampling method used to evaluate machine learning models on limited data. The most common form, k-fold cross-validation, involves dividing the dataset into k equal parts, training the model on k-1 folds while validating on the remaining fold, and rotating through all combinations. This provides a more robust model performance assessment than a single train-test split, helping detect overfitting and estimate model generalization to unseen data. Variations include stratified k-fold (maintaining class distribution), leave-one-out (k equals dataset size), and time series cross-validation for temporal data."}], "output": "Cross-validation evaluates ML models by training and testing on different data splits, typically using k-fold methodology. This approach offers better performance assessment than single splits, with variations for different data types and requirements."}      
   {"input": [{"role": "system", "content": "Generate a clear and focused summary that captures the essential technical details while maintaining their relationships and significance."}, {"role": "user", "content": "Transfer learning is a machine learning method where a model developed for one task is reused as the starting point for a model on a second task. This approach is powerful in deep learning, where pre-trained models on large datasets (like ImageNet for computer vision or BERT for NLP) are fine-tuned on specific downstream tasks. Transfer learning reduces the need for large amounts of task-specific training data and computational resources, as the model has already learned useful features from the source domain. Common strategies include feature extraction (freezing pre-trained layers) and fine-tuning (updating all or some pre-trained weights)."}], "output": "Transfer learning reuses models trained on one task for different tasks, particularly effective in deep learning. It leverages pre-trained models through feature extraction or fine-tuning, reducing data and computational needs for new tasks."}      
   {"input": [{"role": "system", "content": "Provide a precise and informative summary that distills the key technical concepts while maintaining their relationships and practical importance."}, {"role": "user", "content": "Ensemble methods combine multiple machine learning models to create a more robust and accurate predictor. Common techniques include bagging (training models on random data subsets), boosting (sequentially training models to correct earlier errors), and stacking (using a meta-model to combine base model predictions). Random Forests, a popular bagging method, create multiple decision trees using random feature subsets. Gradient Boosting builds trees sequentially, with each tree correcting the errors of previous ones. These methods often outperform single models by reducing overfitting and variance while capturing different data aspects."}], "output": "Ensemble methods enhance prediction accuracy by combining multiple models through techniques like bagging, boosting, and stacking. Popular implementations include Random Forests (using multiple trees with random features) and Gradient Boosting (sequential error correction), offering better performance than single models."}
   

   Anda akan melihat tiga baris pertama file sebagai pratinjau:

   

5. Di bawah Respons pilih tombol Buat . Pilih {{item.input}} dari menu dropdown Buat dengan templat . Ini akan menyuntikkan bidang input dari file evaluasi kami ke permintaan individual untuk model baru yang ingin kami bandingkan dengan himpunan data evaluasi kami. Model akan mengambil input tersebut dan menghasilkan output uniknya sendiri yang dalam hal ini akan disimpan dalam variabel yang disebut {{sample.output_text}}. Kami kemudian akan menggunakan teks output sampel tersebut nanti sebagai bagian dari kriteria pengujian kami. Atau Anda dapat memberikan pesan sistem kustom Anda sendiri dan contoh pesan individual secara manual.

6. Pilih model mana yang ingin Anda hasilkan responsnya berdasarkan evaluasi Anda. Jika Anda tidak memiliki model, Anda dapat membuatnya. Untuk tujuan contoh ini, kami menggunakan penyebaran gpt-4o-ministandar .

   

   Simbol pengaturan/sproket mengontrol parameter dasar yang diteruskan ke model. Hanya parameter berikut yang didukung saat ini:

• Suhu
• Panjang maksimum
• P Teratas

Panjang maksimum saat ini dibatasi pada 2048 terlepas dari model apa yang Anda pilih.

7. Pilih Tambahkan kriteria pengujian pilih Tambahkan.

8. Pilih Kesamaan> Semantik di bawah Bandingkan tambahkan {{item.output}} di bawah Dengan tambahkan {{sample.output_text}}. Ini akan mengambil output referensi asli dari file evaluasi .jsonl Anda dan membandingkannya dengan output yang akan dihasilkan dengan memberikan permintaan model berdasarkan Anda {{item.input}}.

   

9. Pilih Tambahkan> pada titik ini Anda dapat menambahkan kriteria pengujian tambahan atau Anda memilih Buat untuk memulai eksekusi pekerjaan evaluasi.

10. Setelah memilih Buat , Anda akan dibawa ke halaman status untuk pekerjaan evaluasi Anda.

    

11. Setelah pekerjaan evaluasi dibuat, Anda dapat memilih pekerjaan untuk melihat detail lengkap pekerjaan:

    

12. Untuk detail output Tampilan kesamaan semantik berisi representasi JSON yang dapat Anda salin/tempel pengujian yang lulus.

    

Detail kriteria pengujian

Evaluasi Azure OpenAI menawarkan beberapa opsi kriteria pengujian. Bagian di bawah ini menyediakan detail tambahan pada setiap opsi.

Faktualitas

Menilai akurasi faktual jawaban yang diajukan dengan membandingkannya dengan jawaban ahli.

Faktualitas mengevaluasi akurasi faktual jawaban yang diajukan dengan membandingkannya dengan jawaban ahli. Dengan menggunakan perintah chain-of-thought (CoT) terperinci, grader menentukan apakah jawaban yang dikirimkan konsisten dengan, subset, superset, atau bertentangan dengan jawaban ahli. Ini mengabaikan perbedaan gaya, tata bahasa, atau tanda baca, hanya berfokus pada konten faktual. Faktualitas dapat berguna dalam banyak skenario termasuk tetapi tidak terbatas pada verifikasi konten dan alat pendidikan yang memastikan keakuratan jawaban yang disediakan oleh AI.

Anda dapat melihat teks perintah yang digunakan sebagai bagian dari kriteria pengujian ini dengan memilih menu dropdown di samping perintah. Teks perintah saat ini adalah:

Prompt
You are comparing a submitted answer to an expert answer on a given question.
Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Expert]: {ideal}
************
[Submission]: {completion}
************
[END DATA]
Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.
The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:
(A) The submitted answer is a subset of the expert answer and is fully consistent with it.
(B) The submitted answer is a superset of the expert answer and is fully consistent with it.
(C) The submitted answer contains all the same details as the expert answer.
(D) There is a disagreement between the submitted answer and the expert answer.
(E) The answers differ, but these differences don't matter from the perspective of factuality.

Kesamaan semantik

Mengukur tingkat kesamaan antara respons model dan referensi. Grades: 1 (completely different) - 5 (very similar).

Sentimen

Upaya untuk mengidentifikasi nada emosional output.

Anda dapat melihat teks perintah yang digunakan sebagai bagian dari kriteria pengujian ini dengan memilih menu dropdown di samping perintah. Teks perintah saat ini adalah:

Prompt
You will be presented with a text generated by a large language model. Your job is to rate the sentiment of the text. Your options are:

A) Positive
B) Neutral
C) Negative
D) Unsure

[BEGIN TEXT]
***
[{text}]
***
[END TEXT]

First, write out in a step by step manner your reasoning about the answer to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print only the single character (without quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the letter again by itself on a new line

Pemeriksaan string

Memverifikasi apakah output cocok dengan string yang diharapkan dengan tepat.

Pemeriksaan string melakukan berbagai operasi biner pada dua variabel string yang memungkinkan kriteria evaluasi yang beragam. Ini membantu memverifikasi berbagai hubungan string, termasuk kesetaraan, penahanan, dan pola tertentu. Evaluator ini memungkinkan perbandingan peka huruf besar/kecil atau tidak peka huruf besar/kecil. Ini juga menyediakan nilai yang ditentukan untuk hasil benar atau salah, memungkinkan hasil evaluasi yang disesuaikan berdasarkan hasil perbandingan. Berikut adalah jenis operasi yang didukung:

• equals: Memeriksa apakah string output sama persis dengan string evaluasi.
• contains: Memeriksa apakah string evaluasi adalah substring string output.
• starts-with: Memeriksa apakah string output dimulai dengan string evaluasi.
• ends-with: Memeriksa apakah string output berakhir dengan string evaluasi.

JSON atau XML yang valid

Memverifikasi apakah outputnya adalah JSON atau XML yang valid.

Cocok dengan skema

Memastikan output mengikuti struktur yang ditentukan.

Kecocokan kriteria

Menilai apakah respons model cocok dengan kriteria Anda. Nilai: Lulus atau Gagal.

Anda dapat melihat teks perintah yang digunakan sebagai bagian dari kriteria pengujian ini dengan memilih menu dropdown di samping perintah. Teks perintah saat ini adalah:

Prompt
Your job is to assess the final response of an assistant based on conversation history and provided criteria for what makes a good response from the assistant. Here is the data:

[BEGIN DATA]
***
[Conversation]: {conversation}
***
[Response]: {response}
***
[Criteria]: {criteria}
***
[END DATA]

Does the response meet the criteria? First, write out in a step by step manner your reasoning about the criteria to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print only the single character "Y" or "N" (without quotes or punctuation) on its own line corresponding to the correct answer. "Y" for yes if the response meets the criteria, and "N" for no if it does not. At the end, repeat just the letter again by itself on a new line.
Reasoning:

Kualitas teks

Evaluasi kualitas teks dengan membandingkan dengan teks referensi.

Ringkasan:

• Skor BLEU: Mengevaluasi kualitas teks yang dihasilkan dengan membandingkannya dengan satu atau beberapa terjemahan referensi berkualitas tinggi menggunakan skor BLEU.
• Skor ROUGE: Mengevaluasi kualitas teks yang dihasilkan dengan membandingkannya dengan ringkasan referensi menggunakan skor ROUGE.
• Kosinus: Juga disebut sebagai kesamaan kosinus mengukur seberapa dekat dua penyematan teks—seperti output model dan teks referensi—selaras dalam makna, membantu menilai kesamaan semantik di antara mereka. Ini dilakukan dengan mengukur jarak mereka di ruang vektor.

Detail:

Skor BLEU (BiLingual Evaluation Understudy) umumnya digunakan dalam pemrosesan bahasa alami (NLP) dan terjemahan mesin. Ini banyak digunakan dalam ringkasan teks dan kasus penggunaan pembuatan teks. Ini mengevaluasi seberapa dekat teks yang dihasilkan cocok dengan teks referensi. Skor BLEU berkisar antara 0 hingga 1, dengan skor yang lebih tinggi menunjukkan kualitas yang lebih baik.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) adalah sekumpulan metrik yang digunakan untuk mengevaluasi ringkasan otomatis dan terjemahan mesin. Ini mengukur tumpang tindih antara ringkasan teks dan referensi yang dihasilkan. ROUGE berfokus pada langkah-langkah berorientasi pengenalan untuk menilai seberapa baik teks yang dihasilkan mencakup teks referensi. Skor ROUGE menyediakan berbagai metrik, termasuk: • ROUGE-1: Tumpang tindih unigram (satu kata) antara teks yang dihasilkan dan referensi. • ROUGE-2: Tumpang tindih bigram (dua kata berturut-turut) antara teks yang dihasilkan dan referensi. • ROUGE-3: Tumpang tindih trigram (tiga kata berturut-turut) antara teks yang dihasilkan dan referensi. • ROUGE-4: Tumpang tindih empat gram (empat kata berturut-turut) antara teks yang dihasilkan dan referensi. • ROUGE-5: Tumpang tindih lima gram (lima kata berturut-turut) antara teks yang dihasilkan dan referensi. • ROUGE-L: Tumpang tindih L-gram (kata berturut-turut L) antara teks yang dihasilkan dan referensi. Ringkasan teks dan perbandingan dokumen adalah di antara kasus penggunaan optimal untuk ROUGE, terutama dalam skenario di mana koherensi dan relevansi teks sangat penting.

Kesamaan kosinus mengukur seberapa dekat dua penyematan teks—seperti output model dan teks referensi—selaras dalam makna, membantu menilai kesamaan semantik di antara mereka. Sama seperti evaluator berbasis model lainnya, Anda perlu menyediakan penyebaran model menggunakan untuk evaluasi.

Prompt kustom

Menggunakan model untuk mengklasifikasikan output ke dalam sekumpulan label tertentu. Evaluator ini menggunakan perintah kustom yang perlu Anda tentukan.