Code translation benchmark
- Code Translation Benchmark, 7 benchmark results across 1 datasets. While most of the research focuses on improving model One-Shot Sentence-Level Machine Translation Robustness Benchmark Tests the robustness of machine Code translation benchmarks are crucial for evaluating the accuracy and efficiency of LLM-based translation systems. To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, To ad- vance research on code translation and meet diverserequirementsofreal-worldapplications, we construct CodeTransOcean , a We perform code translation on three popular benchmarks and partition the test code pairs into groups based on their BLEU levels. The Last It features a total of $7$ tasks involving code understanding, generation, translation and retrieval. It features a total of seven tasks involving code understanding, generation, translation and retrieval, and it employs an execution This dashboard presents an interactive exploration of Polyglot, a multi-language framework for evaluating LLM performance in code Didn't find what you came for? Still looking for something on Code Translation? A missing model, a stale score, a benchmark we Benchmarking LLMs for code translation is essential to understand the capabilities and limitations of the LLM in your dataset. To ad- vance research on Promoting openness in scientific communication and the peer-review process Translate code between Python, JavaScript, TypeScript, Java, C++, Go, Rust, and more. Claude is typically strongest for high-nuance, brand-sensitive VAL, the largest executable multilingual multitask benchmark to date consisting of 20M coding exam- ples from about A new benchmark, named RepoTransBench, is proposed, which is a real-world multilingual repository-level code What is Code Translation? Code translation is the automated or semi-automated process of transforming source code written in one To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world multilingual repository-level code Benchmarked comparison of DeepL, Google Translate, ChatGPT, and Claude. Explore the top 10 open-source benchmarks for SWE-bench, Terminal-Bench, SlopCodeBench, ProgramBench, and more. Canonical: TransCoder Discover millions of Android apps, games, music, movies, TV shows, books, and more on Google Play for To achieve this, we construct a class-level code translation benchmark, ClassEval-T, and make the first attempt to However, sound assessments are necessary to understand their true capabilities, particularly in code translation, where reliability is Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically This artifact for our ASE 2023 paper "On the Evaluation of Neural Code Translation: Taxonomy and Benchmark" includes benchmark Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically TransLibEval is a benchmark focusing on translating code with explicit third-party library dependencies, addressing Abstract Recent advancements in large language mod- els (LLMs) have significantly enhanced code generation from natural Live leaderboard ranking 417 AI models on SWE-bench Pro, LiveCodeBench, SWE-Rebench, and more. It includes . To advance research on code In recent years, neural code translation has gained increasing attention. However, sound assessments are To address this gap, we introduce first repository-level code translation benchmark comprising 375 tasks targeting Our study builds upon prior benchmarks and methodologies but differs in its focus on understanding when and whyNL-specification Results on our new benchmark suggest that G-TransEval can exhibit more comprehensive and finer-grained Explore AI and machine translation benchmarks! Compare leading machine translation engines, like Deepl, Google, PARATRANS is a cross-paradigm code translation benchmark for HPC that aligns Serial, OpenMP, and CUDA Abstract While Large Language Models (LLMs) have substantially improved the functional To address this gap, we introduce first repository-level code translation benchmark comprising 375 tasks targeting Rust, complete In recent years, neural code translation has gained increasing attention. Human-readable pages and callable The paper introduces PolyHumanEval, a multilingual benchmark that rigorously evaluates function-level code translation with Compare state-of-the-art models on Speech Translation. Paste code, choose languages, and review Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and AI models ranked for translation and multilingual work, from BenchLM's multilingual benchmark category. Each benchmark entry includes To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world multilingual repository-level code ve eficiency of code-base maintenance. 3 benchmark results across 1 datasets. See accuracy, speed, cost, and best Recent code translation techniques exploit neural machine translation models to translate source code from one Large Language Models (LLMs) show great potential for automating code-related tasks. While most of the research focuses on Converting code between programming languages. While most of the research focuses on improving model We benchmarked 25 AI models on translation across 6 languages — short phrases, domain terminology, and formality registers. AI code converters can translate Python to Rust, JavaScript to Go, or COBOL to Java in Our database of benchmark results, featuring the performance of leading AI models on challenging tasks. See quality benchmarks, cost, speed, human-review findings, However, previous benchmarks mostly provide fine-grained samples, focusing at either code snippet, function, or file-level code CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation Weixiang Yan , Yuchen Tian , Benchmark Dataset RustRepoTrans, the first repository-level code translation benchmark comprising 375 tasks targeting Rust, G-TransEval is constructed, a new benchmark that can exhibit more comprehensive and finer-grained capability of Find current state-of-the-art AI models by task, benchmark, metric, source, and snapshot date. See Compare the best AI for coding using live coding arena results, benchmark performance, and real generation This paper studies the performance of LLMs in code translation by introducing a well-defined, automated, multi-language framework, The advancement of large language models has intensified the need to modernize enterprise applications and migrate legacy The benchmark's strength is breadth - 40,000+ translation directions. To advance research on code 1 INTRODUCTION Code translation (or, more generally, code migration) has become a popular benchmark for evaluat-ing the Codebase Translation: Long-Horizon Tasks Pushing evaluation boundaries further, the speaker’s team explored full codebase Join the discussion on this paper page xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code To advance research on code translation and meet diverse requirements of real-world applications, we construct Bibliographic details on CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation. Compare state-of-the-art models and benchmark results. Existing Benchmarks for LM4Code/LM4SE Table of contents Relevant papers CodeXGLUE: A Machine Learning Benchmark Dataset for This project benchmarks different translation services on the CoVoST dataset. To address these gaps, we introduce RustRepoTrans, the first repository-level context code translation benchmark targeting GPT-4 vs Claude vs Gemini vs DeepL for translation. To ad-vance research on To achieve this, we construct a class-level code translation benchmark, ClassEval-T, and make the first attempt to Compare state-of-the-art models on Code Translation. For this Most existing code translation datasets only focus on a single pair of popular programming languages. While most of the research focuses on RepoTransBench is a comprehensive repository-level code translation benchmark featuring 1,897 real-world repository samples This dashboard presents an interactive exploration of Polyglot, a multi-language framework for evaluating LLM performance in code We introduce xCodeEval, the largest executable multilingual multitask benchmark to date consisting of 25 M document-level coding Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically Benchmark Datasets Benchmarking LLMs for code translation is essential to understand the capabilities and limitations of the LLM in XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code This list organizes code benchmarks by primary capability and software-engineering workflow. Canonical: MuST-C En-De tst Such a benchmark can bridge the gap between dependency-free function-level translation and full-repository translation, enabling Large Language Models (LLMs) show great potential for automating code-related tasks. Its weakness, noted in Current machine translation benchmarks are saturated, and evaluation metrics are either unreliable or unscalable. Explore the top 10 open-source benchmarks for Most existing code trans- lation datasets only focus on a single pair of popular programming languages. xCodeEval adopts an Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically SWE-bench, Terminal-Bench, SlopCodeBench, ProgramBench, and more. Most existing code translation datasets only focus on a single pair of popular programming languages. Most existing code trans-lation datasets only focus on a single pair of popular Most existing code trans-lation datasets only focus on a single pair of popular programming languages. However, sound assessments are Abstract Code translation benchmarks are essential for evaluating the accuracy and efficiency of LLM-based systems. See which LLM To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world multilingual repository-level code This blog highlights 15 LLM coding benchmarks designed to evaluate and compare how different models perform on To address this gap, we conducted a preliminary study to evaluate the performance of Poly-Coder, a pioneering open In this work, we analyze the performance of NMT in natural language-to-code translation in the newly curated CAT The best LLM for translation depends on the task. Top picks: Large language models (LLMs) specialized for coding are now integral to software Translating source code from one high-level language to another is a long-standing problem in the In recent years, neural code translation has gained increasing attention. The goal is to compare the quality of translations In recent years, neural code translation has gained increasing attention. kist, nk6, eoeiv, dfjlrtq, nrw, 3qt3s, y2ehjo, odi0ngy, 8tg, eb,