Find the Right Path to Working With DIU

Problem Statement

As artificial intelligence (AI) capabilities evolve at an extraordinary pace, the government requires evaluation infrastructure that can keep pace by continuously assessing new models against mission-specific benchmarks as they are released.

Further, the success of AI systems in national security contexts will depend on human-machine teaming. Evaluation must assess not only whether AI systems can perform tasks in isolation, but whether human-AI teams achieve better mission outcomes than either humans or AI alone.  

Evaluation must also keep pace as AI systems evolve from passive models to active agents that use tools, access systems, and execute multi-step tasks. Beyond model outputs, assessment must account for agent behaviors, including whether agents complete complex missions correctly and safely, use tools appropriately, and maintain auditability.

Desired Solution Attributes

The Department of War (DoW), in partnership with the Office of the Director of National Intelligence (ODNI), seeks an evaluation harness and government-specific benchmarks that together enable rigorous, reproducible, vendor-agnostic assessment of any AI system against government-defined criteria. The Government intends to use this harness across multiple programs. Solutions should be designed for broad applicability rather than single-program optimization. This Area of Interest (AOI) comprises two Lines of Effort (LOE); vendors may respond to one or both. Vendors submitting solutions must specify if they are addressing LOE 1, LOE 2, or both on the title slide or title page in their submission. Submission file titles should likewise indicate “LOE1_”, “LOE2_”, or “LOE1&2_” as a prefix. 

The Government is interested in considering solutions from a wide selection of vendors. All submissions should clearly explain which of the desired solution attributes they do and do not address, with proven examples of prior deployment, if applicable. The Government will consider partial solutions. Vendors are welcome to apply individually or in partnership. The Government may also request teaming arrangements amongst solution providers. Vendors are expected to demonstrate their solution in an unclassified environment as part of the Commercial Solutions Opening.

LOE 1: Evaluation Harness

This AOI seeks an evaluation harness or harnesses to serve as the integrated infrastructure of an execution environment, tooling, and methodology that connects models to benchmarks and produces structured evaluation data. Harness architecture should enable standardized, reproducible assessment of AI systems against defined criteria by providing the following:

• Model Interface: Connects the harness to AI systems under evaluation. Provides a standardized, pluggable architecture for interfacing with diverse model types. 
• Execution Engine: Orchestrates complex evaluation workflows across heterogeneous model and environment configurations. 
• Measurement and Scoring System: Scores model outputs against benchmarks.
• Human Evaluation Integration: Supports human-in-the-loop workflows with interfaces for subject-matter-expert review to measure and compare human workload, usability, and mission performance across human-only, AI-only, and human-AI team scenarios 
• Output and Reporting Layer: Exports all data in open, non-proprietary formats, generates aggregate reports, and provides API access to evaluation data. 
• Continuous Monitoring and Analytics: Automates model ingestion and evaluation. Tracks and monitors performance trends. Provides API access to evaluation data.
• Configuration and Benchmark Management: Defines, versions, validates, and manages benchmarks and evaluation configurations to ensure consistency across environments.
• Degraded Conditions Simulator: Simulates operational stress and network degradation in a controlled, reproducible environment. Enables assessment of model and system resilience under variable conditions to validate performance in mission-critical denied, degraded, intermittent, or limited (DDIL) environments.
• Agentic Evaluation: Evaluates agent actions, tool invocations, and multi-step task execution. Provides for safe testing and maintains comprehensive audit trails of agent decisions and actions.
• Adversarial AI: Supports automated red-teaming, including the execution of adversarial prompts and attack patterns. Scores robustness across attack categories and exports results in open formats.
• Multimodal Inputs: Support for video, audio, and cross-modal datasets and comparison frameworks.

Submissions in response to this LOE should also have the following attributes:

• Modular architecture enabling independent upgrade and addition of components
• Containerized deployment supporting standard government orchestration platforms
• Be deployable as directed across government environments—unclassified, classified cloud, and air-gapped—without fundamental architectural changes
• Harness infrastructure and evaluation content (benchmarks, scoring, attacks) are interoperable but can be de-coupled
• Appropriate access controls and ability to protect sensitive data according to classification requirements

LOE 2: Benchmark Development and Methodology

This AOI seeks solutions from vendors that create benchmarks across unclassified, secret, and top secret workflows, and that provide their methodology for government review and adoption. These benchmarks would be executed using the evaluation harness in LOE 1. 

The benchmarking methodology should address:

• Requirements elicitation: Identifying what capabilities matter for a given mission context.
• Task decomposition: Breaking complex capabilities into measurable evaluation tasks.
• Input design: Constructing scenarios that ensure representativeness (reflection of realistic operational conditions) and operational realism (scenarios that reflect realistic workflows in human-AI teaming tasks).
• Scoring criteria development: Defining what "good" looks like, including rubric construction that prioritizes interpretability (results that are easily understood and can be acted upon by decision makers).
• Baseline establishment: Evaluating open-source/open-weight models to set performance baselines while ensuring fairness (no systemic advantage to particular architectures or vendors).
• Validation: Verifying that benchmarks meet standards for validity (measures of intended capability), reliability (consistent results across runs), and discriminality (clear articulation across performance levels).
• Gaming resistance: Designing benchmarks resistant to optimization without genuine capability improvement.
• Maintenance: How benchmarks are updatable as requirements or model capabilities evolve to maintain their long-term utility.

Training Materials

Materials should enable government personnel to develop and maintain benchmarks without ongoing vendor support, including but not limited to: written methodology guide, worked examples, common pitfalls, quality assurance checklist, and training curriculum.

Vendor Qualifications

This AOI seeks solutions from vendors that are eligible to receive an Other Transaction award in accordance with 10 U.S.C. 4022 and have demonstrated expertise in AI evaluation, security testing, and benchmark creation. Submissions should provide specific, verifiable evidence of the following preferred qualifications as appropriate to the supported LOE:

• Published research on, and/or demonstrable application of, evaluation methodology, benchmark design, measurement robustness, or gaming resistance
• Active contribution to or development of industry-standard frameworks (e.g., HELM, AgentDojo, Inspect), red-teaming tools (e.g., Garak, PyRIT) or established benchmarks (e.g., MMLU, HumanEval)
• Prior collaboration with frontier AI labs on evaluation testing or benchmark creation
• Prior work with the National Institute of Standards and Technology Center for AI Standards and Innovation, United Kingdom AI Safety Institute, or equivalent government evaluation programs
• Personnel holding active security clearances (Secret minimum; TS/SCI preferred) or documented clearability
• Experience deploying evaluation infrastructure in secure government or enterprise environments, specifically within DoW or Intelligence Community (IC) environments (IL5/IL6, JWICS)
• Familiarity with IC analytic tradecraft, DoW doctrine, or domain expertise in national security and military operations
• Experience designing evaluation protocols involving human performance measurement

Awarding Instrument

This Area of Interest is being released in accordance with the Commercial Solutions Opening (CSO) process detailed within HQ0845-20-S-C001 (DIU CSO), posted to SAM.gov on 23 March 2020. This document can be found at: https://sam.gov/opp/c304359f88a0456bab1fa8837a3647f4/view 

Follow-on Production

Any prototype Other Transaction (OT) agreement awarded may result in follow-on production without further competitive procedures. The follow-on may be significantly larger than the prototype OT.

Anticipated follow-on activities include:

• Deployment across additional classification levels and environments
• Benchmark suite expansion for additional mission areas
• Ongoing maintenance, security updates, and capability enhancements
• Training and support for government evaluation personnel

Any prototype OT will include: "In accordance with 10 U.S.C. 4022(f), and upon a determination that the prototype project for this transaction has been successfully completed, this competitively awarded prototype OTA may result in the award of a follow-on production contract or transaction without the use of competitive procedures.”

Problem Statement

The Department of War’s current approach to battlefield electrical energy across the force is fragmented, lacks a framework to meet current and future battlefield power needs, and is tactically and operationally vulnerable to targetable signatures and strained supply lines in areas of contested logistics. Units currently rely on a patchwork of Government-Off-the-Shelf (GOTS) and Commercial-Off-the-Shelf (COTS) generators to both produce and distribute energy to combat systems.

GOTS generators are often oversized for specific battlefield applications, which leads to losses in energy production and wasted fuel in contested environments. These systems typically operate at an output that matches the load but may not be at the most efficient operating point of the generator, potentially causing it to be underloaded. Additionally, they generate noticeable acoustic and thermal signatures, which can pose challenges for silent watch operations. The size and weight of GOTS generators often require separate transport, impacting mobility and, therefore, lethality on the battlefield.

Conversely, while COTS generators may not be specifically suited and hardened for combat scenarios, they can provide flexibility in various operational contexts. However, they typically use gasoline rather than the primary battlefield fuels such as JP-8 or JP-5, which introduces significant logistical burdens.

The integration of energy storage and power conversion systems into the battlefield provides a direct strategy for these problem areas. Energy storage systems can be paired with generators, vehicles, or used as standalone systems as the mission dictates; thereby increasing overall warfighter effectiveness and battlefield capability.

Background

The services lack a flexible, scalable, easily-portable energy storage and power conversion and distribution system, sized between 1kWh GOTS lightweight systems and heavy applications serviced by current 90kWh GOTS energy storage units. Without this capability, combat units will either be underpowered and unable to achieve their mission, or be anchored to oversized generators.

Project Approach

The Portable Resilient Integrated Storage Module (PRISM) project approach under the DIU Commercial Solutions Opening (CSO) will use an iterative prototype development process to provide a clear path to transition of successful commercially demonstrated technology solutions utilizing the following phased approach:

• Project Phase 1: Initial Design and Prototyping (6 - 12 months) - Modifications, as identified, of commercial systems to meet desired solution features and delivery of a first set of prototypes. This phase will include a Limited Lithium Battery Safety Certification and conclude with government acceptance testing to support user evaluations.
• Project Phase 2: Field User Evaluations (6 months).
• Project Phase 3: Updated Design and Prototyping (6 months) - Modification to design, as identified during User Evaluations. Manufacture and delivery of the updated prototype for formal test and evaluation.
• Project Phase 4: Government Military Standard Testing as required (12 months). 

Project Objectives

PRISM will rapidly integrate and test commercially developed energy storage and power conversion and distribution systems and components; enabling warfighters to effectively execute their mission while decreasing operational risk via efficient battlefield power distribution and energy storage, reduced targetable signatures, and decreased logistical requirements.

Desired Solution Features 

Desired solution features include the following attributes and capabilities:

• Power: Will input and output 1 - 15 kilowatts (kW) at various Direct Current (DC) voltages up to 48 Volts (V) and world wide Alternating Current (AC) voltages and frequencies.
• Energy Storage: Will utilize 1 - 30 kilowatt-hours (kWh) of scalable, energy storage modules.
• Interoperability: Will utilize standard military and/or commercial connectors for inputs and outputs.
• Communications: Will utilize a Controller Area Network (CAN) and/or Tactical Microgrid Standard (TMS) (MIL-STD-3071) for communication when necessary.
• Size: Depending on energy storage capacity, compact form factor suitable for manportable transport or in light tactical vehicles.
• Weight: At the subsystem level, each component will weigh no more than 160 pounds (lbs).
• Architecture: Will employ a modular, open system architecture design to allow upgrades of power electronics and energy storage systems as technology improves.
• Performance: Shall allow operation without interference of electronic equipment such as small form factor radars, computers, and tactical radios without degrading their performance.
• Ruggedization: Compliance with MIL-STD-810H or commercial equivalent standard for shock, vibration, and drop.
• Environmental: Designed to operate in austere environments and conditions.
• Signature Management: Minimization of electromagnetic, thermal, audible, and visual signatures is critical.
• Maintainability: Designed for ease of maintenance with readily available components and clear diagnostic capabilities.
• Batteries: If using lithium batteries, it is highly desired to use a common military or commercial battery and keep in mind Section 842 of the FY 2026 National Defense Authorization Act.
• Timeline: Will be ready for full scale operational testing, demonstration, and evaluation within 12 months from time of award.
• Restrictions: Fuel cell powered solutions will not be accepted.

Vendor Solution Brief Submission Options

Vendors have flexibility in how they submit their solution briefs, which can be proposed either independently or through a teaming arrangement. 

Process

Submissions will be evaluated in accordance with CSO HQ084520SC001 available on https://DIU.mil and https://SAM.gov. Vendors selected for Phase 2 will receive an amplification letter with expanded details to help inform their Phase 2 pitches. Those vendors will be expected to provide their Rough Order of Magnitude (ROM) cost breakdown. The vendorsʼ Phase 2 pitches shall address the Phase 2 evaluation factors contained in CSO HQ0845-20-S-C001. DoW requires companies without a CAGE code to register in SAM https://SAM.gov if selected for an agreement award. The Government recommends that prospective companies begin this process as early as possible.

Background

The Department of War (DoW) faces a littoral contested logistics challenge. Increasingly distributed operations in austere, contested littoral environments are met with all-domain threats targeting logistics capabilities, locations, and activities. These threats limit the ability of warfighters to persist in contested environments and remain combat effective.

Problem

The DoW requires the ability to resupply units spread over wide distances in contested littoral environments.

Desired Solution Attributes

The DoW seeks innovative solutions that provide low-cost logistics transport and enable critical inter-theater resupply within littoral environments. 

Proposed solutions should be demonstrable within 180 days of award and built upon principles of the Modular Open Systems Approach (MOSA) to ensure interoperability and future improvement.

Solutions will be evaluated on their problem alignment, technical merit, and innovative approaches to support the delivery and sustainment of maritime autonomous resupply systems, at scale, for operations in contested littoral environments.

The Government will preference solutions that:

• Transport by sea:
  • A minimum of 18,000lbs of cargo; and
  • All classes of supply including bulk liquids. 
• Operate in NATO sea state 5 and survive in NATO sea state 6.
• Operate at a range of 1,000 to 2,000 nautical miles in NATO sea state 5 while fully laden.
• Sustain speed of at least 12 knots in NATO sea state 4 while fully laden.
• Can loiter for up to 96 hours (assuming reduced speed below 12 knots).
• Support transloading of all classes of supply, including standard warehouse pallets, Pallet Containers (PALCONs), & Joint Modular Intermodal Containers (JMICs). Payload dimensions must be capable of safely and reliably transporting:
  • A minimum of six JMICs (JMIC dimensions are 43’’ tall, 51.75’’ wide, 43.75’’ long and weighing 3000 pounds); or
  • Two containers each measuring 44” tall, 41.5” wide, 166” long, and weighing 5100 pounds
• Capable of autonomous operation for transit, in congested waterways, and for port operations, to include perception, navigation, and operations (to include hazard avoidance) using:
  • Active sensing / global positioning system (M-Code GPS); and 
  • Passive sensing during emissions control (EMCON) conditions or when communications are lost (i.e. operating in Denied, Degraded, Intermittent, and Limited (DDIL) communications environments).    
• Provide sufficient navigational accuracy in DDIL environments to assure reaching the objective location. Prior to completion of prototyping, companies will be expected to demonstrate assured Position, Navigation and Timing in DDIL and GPS-degraded and denied environments.
• Employ a low-profile form factor to reduce chances of detection and interdiction by potential adversaries in a contested domain.  
• Provide redundant line of sight and beyond line-of-sight communications capabilities on standard Navy and Marine Corps networks.
• Capable of manual human operations via remote control, particularly for first and last mile terminal guidance.  
• Ability to pass vessel control between operators (both for remote control and for autonomous operations).
• Capable of one human to multiple craft control.
• Ability to conduct dynamic re-tasking, including changing waypoints and transitioning to loitering operations.
• Can conduct shore-to-shore (pier-to-pier), ship-to-ship, and ship-to-shore distribution, including Military Sealift Command vessels and other littoral connectors. Ship or shore-to-beach operations with self-withdrawal after cargo discharge desirable.
• Require minimal off-platform support (e.g., material handling equipment (MHE)) to discharge cargo.  
• Provide over-the-road trailer compatible with both commercial tractor trailers and Marine Corps tactical prime movers.
• Resistant to tampering while underway with the ability to remotely scuttle the vessel.
• Can provide a functional prototype and provide evidence of the ability to scale manufacturing to sustained production levels if selected, including indicators of company health and production readiness.
• Can provide high-level rough order of magnitude (ROM) cost estimates, to include the build and testing of prototype craft and production at scale.

Proposals for holistic solutions are preferred, and partnering with other companies to deliver solutions addressing all attributes is highly encouraged, e.g., hardware-autonomy partnerships. However, companies with compelling, demonstrable capability in one or more of these desired attributes are still encouraged to apply. 

The Government encourages submission of mature commercial designs and willingness to adopt government-owned designs for the physical platform to meet aggressive timelines and to properly allocate ample resources toward the prototyping of the autonomy portion of the solution. Government Furnished Information (GFI) will be provided to companies awarded under the CSO process for use to speed integration, reduce redesign, and ensure interoperability. GFI may include:

• Reference hardware designs for autonomous low-profile vessels (ALPVs)
• Baseline software architectures and orchestration frameworks
• Technical Data Packages (TDPs)
• Interface Control Documents (ICDs)
• Data/message protocol specifications and APIs
• Modeling and simulation environments