AIQB
ResearchOrdinary

PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization

Source: arXiv·

Summary

Natural Language Processing methods have enabled novel solutions and advances in the field of privacy, particularly in the sub-domain of text-to-text privatization, where the goal is to transform a sensitive input text into a privatized output by ideally masking (in)directly identifiable or otherwise private information. The evaluation of text-to-text privatization, however, is not straightforward, and the extant literature has utilized a myriad of techniques and metrics to quantify the privacy-preserving capabilities of privatization methods. Seeking to unify the evaluation of text-to-text privatization, we introduce PrivBench, a holistic and modular benchmarking platform for researchers and practitioners working on text privatization. PrivBench is holistic in that it evaluates privatization on a series of defined desiderata, which are structured into modules. PrivBench is not only modular but also extensible, allowing for future updates and benchmark versions. PrivBench is user-centered and promotes competition via real-time evaluation and a live public leaderboard. The platform is free to use and openly accessible at https://privbench.com/.
CategoryAI Research & Papers
TierOrdinary
Published
Indexed by AIQB
SourcearXiv
AIQB record IDintel-88070d0bf2149b32f46b13fc
Read original source ↗View data sourceMore Research

Related stories

Matched by category, entity tags and source

ResearchOrdinary
MURANO: Design, Run, and Reproduce Mechanistic Interpretability Experiments as Composable Pipelines
This paper presents Murano, an open source framework for designing, running, and reproducing mechanistic interpretability studies of large language models, intended for researchers across disciplines. These studies often combine loading, r…
Source: arXiv
ResearchOrdinary
SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?
Large language model-based multi-agent systems are evolving from fixed interaction topologies toward dynamically orchestrated Agent Swarms. However, existing benchmarks are still largely based on single-agent or general-purpose agent tasks…
Source: arXiv
ResearchOrdinary
Season-Aware Hybrid Convolutional-Transformer for Antarctic Sea Ice Concentration Forecasting
Antarctic sea ice concentration (SIC) forecasting is an important yet challenging task due to the coexistence of complex spatial structure, long-range temporal dependencies, and strong seasonal variability. Conventional convolution-based m…
Source: arXiv
ResearchOrdinary
Fine-Grained Multi Image Object Hallucination Benchmark
Multimodal Large Language Models (MLLMs) are increasingly deployed in multi-image scenarios requiring complex reasoning across visual contexts. However, current MLLMs remain fundamentally limited by object hallucination-generating plausibl…
Source: arXiv · GPT / Gemini
ResearchOrdinary
PyKEEN-NSX: A Modular Framework for Static, Dynamic and Schema-Aware Negative Sampling in PyKEEN
Embedding methods have become popular due to their scalability on link prediction and/or triple classification tasks on Knowledge Graphs (KGs). Embedding models are trained relying on both positive and negative samples of triples. However,…
Source: arXiv
ResearchOrdinary
Geometry of Divergence: Tracking Hidden-State Trajectories for Adaptive Multi-Turn Reasoning
LLM agents need to sustain goal-consistent reasoning across long multi-turn interactions under strict resource constraints. However, as the multi-turn context accumulates, it can destabilize the underlying LLM's internal representation of …
Source: arXiv
ResearchOrdinary
Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models
Removing a specific individual's information from multimodal large language models (MLLMs) is often needed after deployment, but existing methods rely on a retain set, which is hardest to obtain at that point, and rebuilding it recreates t…
Source: arXiv
ResearchOrdinary
What It Costs to Compose, Rebuild, and Correct Precomputed Memory
Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under whi…
Source: arXiv · Llama
ResearchOrdinary
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We prop…
Source: arXiv
ResearchOrdinary
Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL
While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the stand…
Source: arXiv