Part-time Benchmarking Engineer for Revolutionary LLM Evaluation - Remote

Unlock Your Potential as a Benchmarking Engineer at Vals AI Join the pioneering team at Vals AI, where we're redefining the landscape of Large Language Model (LLM) evaluation for enterprise applications. As a part-time Benchmarking Engineer, you'll play a crucial role in shaping the future of AI adoption in business. With a competitive salary and the flexibility of remote work, this is an opportunity to grow your career while making a significant impact. About Vals AI: Revolutionizing LLM Evaluation Vals AI is at the forefront of creating the infrastructure and certification for automatically auditing LLM applications, ensuring they're ready for enterprise consumption. Our mission is to accelerate the automation of all knowledge work by building the barometer for whether AI is useful. With a strong foundation in NLP evaluation research from Stanford and a $5M seed funding from top investors, we're poised to make a substantial difference. The Role: Part-time Benchmarking Engineer We're seeking a highly skilled and curious Benchmarking Engineer to maintain and improve our public LLM benchmarks. This role requires a strong technical background, excellent communication skills, and the ability to work independently. You'll have significant ownership of our benchmarking site and the agency to propose new benchmarks based on your ideas and hypotheses. Key Responsibilities: Creating New Datasets: Collaborate with our data annotators and partner groups to develop new, private datasets that will drive the evaluation of LLMs. Running Benchmarks: Execute our existing benchmarks against new models as they emerge and compile the results. Analyzing Results: Write comprehensive, free-text analyses of the raw quantitative results, answering critical questions about model performance, pricing, and error types. Social Media and Reporting: Craft engaging Twitter and LinkedIn posts that describe our findings and maintain our social media presence. Script Maintenance: Develop and maintain the scripts used to run benchmarks against our datasets, ensuring efficiency and accuracy. Requirements: The Skills We're Looking For Python Expertise: Deep experience with Python, as it's the primary language used for this role. Communication Skills: Strong writing and communication skills to distill complex technical findings into easily consumable reports for non-technical audiences. Team Collaboration: Experience working in teams, including development sprints, Git best practices, and reviewing pull requests. Availability: Approximately 20 hours a week, with flexibility to adapt to spikes in workload when new models are released. Nice to Have: Additional Skills That Shine LLM Familiarity: Knowledge of LLM methods and developments, with an innate interest in the space. ML Research Experience: Background in ML research or data science, ensuring scientific rigor in our benchmarks. What We Offer: A Rewarding Environment At Vals AI, we offer a highly competitive salary, the option to work from our SF office (with lunch, dinner, snacks, coffee, and drinks provided), and the opportunity to grow into a full-time role. Our team is built on a foundation of intelligence, ownership, intensity, and a solutions-oriented mindset. Our Tech Stack: The Tools You'll Work With Our frontend is built with React and TSX, while our backend uses Django. All infrastructure is hosted on AWS, providing a robust and scalable environment for our applications. Culture and Values: What Drives Us We value intelligence over pedigree, ownership over consensus, and intensity in execution. We're looking for individuals who see solutions rather than problems and are driven to craft innovative solutions. Our team has a diverse background, with experience from top companies like NVIDIA, Meta, Microsoft, Palantir, and HRT, and a collective record of over 300 citations in published work. Career Growth and Learning Benefits As a Benchmarking Engineer at Vals AI, you'll have the opportunity to: Work on cutting-edge LLM evaluation techniques. Collaborate with a talented team of engineers and researchers. Develop your skills in Python, data analysis, and technical writing. Contribute to the growth and development of our benchmarking platform. Compensation and Perks We offer a competitive salary, commensurate with experience. Additional benefits include: Optional office work in SF with amenities. Opportunity to transition into a full-time role. A dynamic, innovative work environment. Conclusion: Join the Vals AI Journey If you're passionate about LLMs, evaluation techniques, and driving the adoption of AI in enterprise settings, we want to hear from you. As a part-time Benchmarking Engineer, you'll be at the heart of our mission to create the industry-standard benchmark for LLM applications. Apply now to be part of this exciting journey and shape the future of AI. For further reading on the challenges and importance of LLM evaluation, check out our references: Hugging Face blog on evaluation. Anthropic's blog on challenges in evaluation. New York Times article on issues in benchmarking. Stanford HAI report on hallucinations in legal tech tools. If you believe you have what it takes to be a part of our innovative team, submit your application today. We look forward to hearing from talented individuals like you. Apply for this job