LightCycle (Data Mesh)

Problem Statement: 

The Department of Defense (DoD) currently operates multiple network transport layers, which serve different enclaves of separate information domains relying on multiple message brokers for data stream management. Data distribution across this network of enclaves is often managed by individual applications, leading to surges, latency, and geographic dependence. Furthermore, given the nature of operating in Denied, Disrupted, Intermittent and Limited Impact (DDIL) bandwidth areas, these issues are compounded and flood the network with retries when disconnected and re-transmissions of messages when connectivity is restored. This lack of control and management of data distribution results in inefficient usage of network resources and decreased performance for high priority mission data.

Desired Solution Attributes:

DoD is looking for a commercial solution that can reliably and efficiently distribute data across multiple, geographically-dispersed network enclaves that can communicate with each other using routing protocols and devices. The solution should also allow for data sharing between different classification levels and prevent unauthorized access to data. 

Specifically, DoD is looking to prototype a data distribution solution that will manage the transport and caching of data across a globally-distributed infrastructure using an asynchronous publisher-subscriber (Pub/Sub) messaging system. 

This prototype must exhibit the following primary capabilities:

• In General:
  • Run on government-furnished infrastructure/equipment operated by government personnel 
  • Run on contractor-provided infrastructure/equipment operated by DoD-approved contractor(s). 
  • Utilize the authentication methods and infrastructure provided by the government. 
  • Permit data sharing across different classification levels
  • Ensure proper access controls to prevent unauthorized access to data
  • Incorporate plans for implementing Role Based Access Control (RBAC)
  • Facilitate management and governance of data through a Data Product Registry to expedite onboarding and promote use of enterprise data for DoD initiatives.
  • Allow for flexible and abstracted access patterns for real-time data exchange between nodes and external data producers and consumers
  • Facilitate the translation and exchange of mission command data, such as reference/encyclopedic data, environmental data, imagery, task orders, and application/configuration data
  • Monitor network health and dynamically route traffic to optimize performance
  • Easily integrate with enterprise data meshes and/or API gateways to improve interoperability between systems.
• Network Performance
  • Incorporate network awareness into message brokers to determine priority of messages delivered over limited bandwidth connections from application service level policies.
  • Interface with a government-provided network transport solution to manage policies over routing.
  • Report metrics to enable analysis and visualization of cache/bandwidth utilization algorithm performance.
• Data Service
  • Provide data catalog management (tagging/metadata/schema) to enable data object discoverability and governance.
  • Provide analysis of data object use across the network to optimize data distribution within the network.
  • Provide local interface to publish data for dissemination (broker or cache).
  • Translate data objects/messages into standard schema for applications consumption
  • Perform optimized compression on data objects to reduce network bandwidth needs.
  • Perform optimization decisions on message queuing and data streaming in the context of network performance (i.e., DDIL) and policy engine inputs.
• Message Broker
  • Incorporate message flooding safeguards between message brokers to prevent saturation of network bandwidth during DDIL scenarios.
  • Determine optimal message broker architecture such that minimal network bandwidth is consumed between network segments while providing data reliability towards application service-level policies.
  • Service message brokering between applications with multiple input and multiple output points of data origination.
  • Identify approach to synchronize message queues, resolving conflicts, resulting from disconnected operations.
• Caching
  • Orchestrate caching data on constrained infrastructure based on service-level policies and best effort means to reduce network bandwidth. 
  • Set global policies across global, regional, and local cache stores to manage updating or purging data as needed.
  • Implement connection-aware cache control to keep cache contents up-to-date when the source server is unavailable for extended periods.
  • Implement Data-At-Rest (DAR) encryption for all cache storage
  • Ensure data encryption during transit between Data Distribution Network (DDN) brokers/routers and applications.
  • Ensure classification policy and data sovereignty laws are maintained across cache sites.
  • Provide geo-fencing functions/policy engine to prevent data from being stored on geolocated infrastructure.
• Policy Engine
  • Provide an end user interface to establish mission playbook policies, which orchestrate multiple configuration changes to the multiple products (i.e., network transport, message broker(s), cache, etc.) to quickly reprioritize the system for mission context.
  • Support distributed (global, regional, and local) policy engine changes to support DDIL operations.
• Security
  • Solution accreditation is required for full functionality and deployment on DoD systems
  • Existing authority to operate (ATO) and/or certification as a system of record is a plus
  • Vendors must possess or be able to obtain Federal Risk and Authorization Management Program (FedRAMP) High accreditation along with Defense Information Systems Agency (DISA) Impact Level (IL)-6 provisional authority (PA) or offer solutions that are compatible with services that have already attained an ATO and are DISA IL-6-compliant.
  • Final product must be compliant with FedRAMP High Impact Risk Level.