High Level Architecture
Jan 1 2022 at 12:00 AM
The purpose of this document is to describe the general architecture of a Virtual Raptor™ (V-Raptor) and to outline the various software and hardware requirements for the provisioning of a V-Raptor™.
Introduction
This section provides an overview of the V-Raptor™ architecture and will briefly discuss each component.
Figure 1 – High level V-Raptor architecture
Hardware
This section details the required infrastructure that provides networking, compute, and storage on which to run the necessary software components. The Raptor™ is designed to run in the cloud or on premises if machines with suitable specifications are provisioned.
It is important to note that the V-Raptor™ makes use of localised storage in almost all its services as part of its internal messaging system. Messages are persisted before transmission to ensure that a service crash does not result in all currently queued messages being lost. This setting can be turned off to persist only if messages cannot be transmitted. However, should this setting be required a high-performance storage solution should be provided to ensure the transmission throughput is not adversely affected.
Provisioning
The V-Raptor™ deployment currently supports VMWare, AWS, and Azure environment.
Kubernetes is required for all orchestration in the V-Raptor™. Additionally, the V-Raptor™ makes use of Kubernetes objects such as Config Maps and Secrets to store information required by its core service. When deploying the V-Raptor™ to a Kubernetes cluster, it is important to take note of the required permissions. Without the necessary permissions, the V-Raptor™ deployment will be unable to correctly create and configure the necessary objects. The specific permissions required are discussed in a separate deployment document.
The V-Raptor™ makes use of DNS lookups for service discovery. It also requires a storage solution that provides access to volume claims through storage classes in Kubernetes. Please note that the underlying storage and networking infrastructure used by the storage solution should be performant to ensure that throughput is not adversely affected.
By design, each V-Raptor™ maintains its own internal certificate authority which is used to provide each micro-service with its own set of certificates. A more detailed discussion of the security model used by the V-Raptor™ is provided in a following section. Since these certificates are self-signed, an external certificate authority is required for services that need to expose an endpoint to external services or devices.
Application and Data
A V-Raptor™ consists of one or more micro-services. Each micro-service is provisioned in its own Kubernetes pod.
The Micro-services are written in DotNet Core 3.1 and use the ASP.Net Core 3.1 layer to provide HTTPS services for API, Health and Prometheus information.
Each service requires two ports to be open. The first port is a TCP/SSL port which is used for intra-service RPC calls. The second port is an HTTPS port to provide access to the API for the service and for Health and Prometheus metrics scraping.
All services have a link to the Identity service which provides the means to identify and validate all connections to and from the services.
Figure 2 - V-Raptor core services
Each micro-service requires a dedicated file storage to persist service data. This storage need not be located on the micro-service, as the V-Raptor™ currently supports both SQLite and MongoDB. The storage method can be configured and changed at runtime. There are currently three types of storage required:
Figure 3 - V-Raptor data storage in transit
Local Database
Database used to store application specific data.
Encryption of data at rest is handled by the Asp.Net Data Protection layer.
Data Storage in Transit Database
Data Storage in Transit (DSIT) is a mechanism whereby telemetry is persisted durably before transmission and processing when sent between two microservices.
Database used to store requests in flight.
Used for Registration, Telemetry and Command requests.
Ensures that no request that cannot be transmitted, for whatever reason, is lost due to the microservice shutting down or crashing.
Encryption of data at rest is handled by the Asp.Net Data Protection layer.
File Storage
Configuration files for controlling the behaviour of the microservice.
Certificates required by the micro-service.
Log files
Device Integration
This micro-service provides the integration point between the V-Raptor and the physical device.
V-Raptor™ allows multiple micro-services to handle device integration, usually with each micro-service handling a separate device type or protocol. Numerous protocols are currently supported (e.g. HTTPS REST, CoAP, DLMS, ModBus, SNMP, TCP, UDP etc). Transport layer security is provided, where possible, via TLS1.31 or DTLS2.
Each Device Integration micro-service can handle multiple physical devices.
The microservice is responsible for collecting data from the devices, either via a push, or a scheduled pull methodology. Once collected, the data is processed, transformed, filtered, and standardised ready for transmission to the Collector.
Collector
The Collector microservice is the heart (hub) of the V-Raptor. It provides centralized processing and storage of data received from the Device Integration microservices.
Cloud
The Cloud microservice provides a link to the Commander™ Portal. Device Registration and Telemetry packets are sent to the Commander™ and Device Commands are received from the Commander™.
Data Storage in Transit
Once accepted by the V-Raptor™, it guarantees that the data packet will be processed and transmitted to the Commander™ portal without loss. This is achieved via Data Storage in Transit. Each microservice uses a database to store data packets as they arrive and keep track of the data packets until they have been processed and transmitted to the next microservice. Only when the data packet has been successfully passed on to the next micro-service will that packet be removed from the database.
Although highly performant, each microservice also provides a means to push back if data packets are arriving faster than the microservice can process and transmit them onwards. This ensures that a microservice can never be overwhelmed by data packets to a point where all resources have been consumed and the V-Raptor™ crashes.
Integration and Tools
Security and Compliance
The V-Raptor™ solution is deployed in Kubernetes, with each micro-service running in its own sandbox which ensures that in the unlikely event of an exploitation, the attacker will only have access to the sandbox of the exploited service and not the entire solution.
All inter-service communication and communication to the Commander™ portal is protected via TLS1.3 encryption of the transport layer.
Where possible, communication to/from the devices is protected via TLS, DTLS or other forms of encryption. In the cases where this is not possible, a proxy is usually deployed between the device and the V-Raptor™ to handle this.
Device data loss is virtually eliminated by employing Data Storage in Transit via local databases at the micro-service layer to ensure data is not lost in the event of power loss or application crash.
Certificates and Encryption
The V-Raptor™ supports RSA, DSA and Elliptic Curve X509 Certificates3. All certificates have their certificate chain verified before they are used within the V-Raptor™.
All microservices are provided with an RSA X509 certificate. The subject name of the certificate forms the unique Service ID of the micro-service. The Service ID assigned to the Collector service forms the Gateway ID of the entire V-Raptor™.
This certificate is used to generate the digital signature of the Service ID that is used to identify and verify the service when it communicates with other services withing the V-Raptor™ environment.
The V-Raptor™ uses the Asp.Net Core Data Protection4 (ASPNetCoreDataProtection) layer to provide encryption and decryption services. The microservice certificate is used to protect the encryption keys at rest. The V-Raptor™ uses encryption for sensitive data at rest, both in files on the disk and in the databases used by the service.
Identity
The V-Raptor™ uses Identity Server 45 to provide OpenID and OAUTH 2.0 services to the V-Raptor™ microservices. JWT6 tokens are used to verify access authorization.
These services include:
- Inter-service identification and verification
- API key verification
Figure 4 - V-Raptor service authentication flow
One of the primary services. Ensures that no unauthorized micro-service can communicate with any of the V-Raptor™ microservices.
Authentication request sent to the Identity Server containing a digital signature of Service A’s ID created using Service A’s private key.
Identity Server verifies the signature using the public certificate held for Service A. Once verifies, Identity Server generates a signed JWT token and returns this to Service A.
Service A attaches the JWT token to the request and sends it to Service B.
Service B sends the JWT token to Identity Server to be verified.
Identity Server verifies that the JWT token was indeed signed by itself.
Service B accepts the request and processes it, returning the response to Service A.
API key verification
This works in a similar way to the Inter-service identification and verification above.
Service A will send an authentication request containing the API key to the Identity Server. The Identity Server will look in a local database to see if the API key has been registered. If it has, the Identity Server will generate a signed JWT token and return it to Service A.
Monitoring
Each V-Raptor™ solution provides several streams to allow external monitoring of the performance and status of individual microservices and the V-Raptor™.
Prometheus
Each microservice that forms part of the V-Raptor™ solution provides a service specific set of metric values that can be scraped by Prometheus7 and displayed using tools like Grafana8 to provide monitoring, insight and alerting on the performance of each microservice.
Health
Each microservice provides middleware that links into the Asp.Net Core Health Checks 9. The health checks provide both a simple service health endpoint, that can be monitored by Kubernetes to determine whether the service needs to be restarted, and a more detailed health endpoint that will provide in depth health information on all aspects of the systems running within the microservice.
Logs
Th V-Raptor™ uses NLog10 for application logging. This can be easily configured to change the logging level and what is logged. NLog also supports plugins to enable logs to be shipped to logging aggregators like the EFK stack.