Device Store Architecture
Jan 1 2022 at 12:00 AM
- Overview
- Architecture diagram
- Development environment
- Device Driver Service
- Driver packages
- Raptor package manager
- Device Store Service
- Multiple Upstream providers
- Migrate Device Store Drivers
- Purge Device Store
- References
Overview
The following sections describe the device store architecture and implementation. Additionally it provides insight into the available functions.
Introduction
The device store, also known as driver store, is used to host driver packages. These packages are in a NuGet format.
Each driver has two packages.
- Meta package containing the driver deployment configuration and,
- UI package which holds meta-data related to the driver such as driver name, driver protocol, version, description etc.
The device store we are currently supporting originates from an open source NuGet project called BaGet. We’ve cloned the repo and enhanced to include security elements and strip unnecessary components.
Please see references below for additional information related to BaGet. Also see the guides section on how to install and configure a local device store instance
Architecture diagram
Figure 1 - Overview
Development environment
The following information is related to the development environment, or developer’s environment.
Device Driver Service
Information and guides related to on how to start development on a device/driver service can be located here.
The information related to the release process of a driver is currently out of scope of this document. However it involves a process of making the driver assembly available in a public container. The location of this container will be used within the driver deployment configuration package.
Driver packages
Currently the driver packages are compiled in a NuGet package format.
A driver consists of two packages, one UI and UI.Meta where the UI package contains a json file containing information related to the driver. The UI.Meta package also contains a json file containing deployment information.
Driver Meta Package
The deployment meta package is basically a NuGet package containing a deployment.json file. This file contains the configuration and deployment information related to the device service.
Sample NuGet file name of the Teltonika service: IoTnxt.Raptor.Device.Teltonika.UI.Meta.4.0.21.1-rc.nupkg
Example of the config is below:
{
"Configs": {
"Type": "ConfigSection",
"Data": {
"DevicesOptions.json": "{\n\t\"Devices\":{}\n}"
}
},
"Certificate": {
"Type": "CertificateSection",
"AlternateNames": [
"localhost"
]
},
"Kubernetes": {
"Type": "KubernetesSection",
"ContainerName": "containerdesitination",
"RunCommand": "dotnet",
"RunCommandArguments": [
"deviceX.driver.service.dll"
],
"EnvironmentalVariables": null,
"OverrideExisting": true,
"OverrideConfigs": false,
"UsePersistentStorage": false,
"StorageAmountBytes": 1048576,
"IssueCertificates": true
},
"Endpoints": {
"Type": "EndpointSection",
"RpcTargets": {
"default": "collector"
},
"RpcServer": true,
"RpcClient": true
}
}
Driver UI Package
The driver UI package is basically a NuGet package containing a PackageInfo.json file. This file assists that the deployment manager can display the basic meta-data of the driver.
Sample NuGet file name of the Teltonika service: IoTnxt.Raptor.Device.Teltonika.UI.4.0.21.1-rc.nupkg
Example of the file below:
{
"ImageUrl": "data:image/jpeg;base64,base64Value",
"Description": "Teltonika Asset Tracker Device Package",
"DevicePackageVendor": "IoT.nxt",
"ProtocolType": "TCP",
"DeviceCategory": "Protocol"
}
Once uploaded to the device store, the information will be available as follows:
Figure 2 - Teltonika Sample 1
and
Figure 3 - Teltonika Sample 2
Package versions
It is recommended to use the standard NuGet package version guide when assigning versions. Please see Package Versioning.
Listed and unlisted packages
There are instances where you would like to have a package listed or unlisted. Listed packages are usually packages where a Meta package is paired with a UI package. Thus the driver package is available on the UI.
However there are instances where we don’t want the package to be displayed on the UI. Thus on the Meta package is uploaded to the device store. An example would be if driver deployment testing is still underway and you don’t want to make the package available for all clients.
Digitally signed packages
Each UI and UI.Meta driver NuGet package has the option to be signed. When signed, the packages will contain a digital signature which is used to verify the package. The signing of the package or package content is done prior to upload the package to the device store and using the raptor package manager cli tool.
On querying the device store to list package details, the package meta contains information such as if the package has a digital signature, and/or if this digital signature has been verified and the result.
All the available package responses:
- Verified signed package. = Verified digital signature.
- Unverified unsigned package. = No digital signature.
- Invalid signed package. = Invalid verification of digital signature.
Sample Package Details Response:
{
"displayName": null,
"packageIdentifier": "IoTnxt.Raptor.Device.CatTracking.UI",
"devicePackageVendor": {
"vendorName": "IoT.nxt"
},
"imageUrl": "data:image/jpeg;base64,base64Value",
"packageLastUpdated": "0001-01-01T00:00:00",
"description": "Cat Tracker Device Package - Testing Driver",
"deviceCategory": "Protocol",
"protocolType": {
"protocolType": "TCP"
},
"metaTags": null,
"configKey": null,
"devicePackageVersion": "1.0.5-rc",
"isDeploymentPackage": false,
"hasDigitalSignature": true,
"isSignatureValid": true
},
Raptor package manager
Raptor package manager is a cli tool which assist the developer to compile the driver packages.
Please see installation and an execution guide can here. Installing and executing raptor package manager commands.
Functionalities currently covered under the Raptor package manager cli tools are as follows:
- Configure Publish Profiles.
- Configure Device Store Profiles.
- Configure Device Store Security Profiles.
- Secure driver package content files using a installed X509 certificate.
- Verify if the driver package content is secured.
- Distribute driver packages to multiple device stores.
Device Store Service
A device store service is a service which handles communication between the device store(s) and the device store clients such as the deployment agent or deployment manager app. The device store service has been updated to read and deploy devices/drivers from one or more device stores.
Available functionalities.
- Get all available package ids.
- Get all available UI packages with their details.
- Gets the package versions asynchronous.
- Extracts the meta package.
- Gets the common package versions across a number of package Ids asynchronous.
Multiple Upstream providers
The device store architecture has provisioned support for multiple upstream providers.
The following represents an overview of the implemented functionality.
-Points 1 to 3 each indicates a device store instance running hosting artifacts, NuGet packages, of available drivers.
-Points 4 to 6 each indicates a instance of a device store. Can be either a running instance of BaGet, NuGet or Azure Artifact store. We use BaGet.
-Point 7 indicates the device store service which is configured to point to each device store instance. Common configuration includes the device store URL and authentication credentials if required.
-Point 8 indicates a device store client which can execute commands against the device store service. I.e. Get Available packages.
-Point 9 indicates an instance of Postman which can assist in executing the same commands over REST. I.e. Get Available packages.
-Point 10 indicates the result of the command/call to get all available packages from the device store service and the connected device store. Note that this particular command will aggregate all the available drivers into a single response/list.
Additional commands or functionalities that can be executed are. Extract Package; Get Package Version; Get Common Package versions between two packages; Get available packages; List available packages details.
Figure 4 - MultiStoreSupport
Single or Multi-Store Support Configuration
The IoTnxt raptor device store service has the ability to be configured if it will be connecting to a single instance of device store, or multiple.
Configuration for Multiple Upstream support is done in the following configuration files: DeviceStoreServiceOptions.json - Specify if the device store service will connect to a single package provider, multiple instances. For multiple instances:
{
"DeviceStoreServiceOptions": {
"PackageProviderType": "MultiNuget"
}
}
For single instances:
{
"DeviceStoreServiceOptions": {
"PackageProviderType": "SingleNuget"
}
}
Single instance configuration
Single instance configuration is done in the NugetPackageApiOptions.json
Sample
{
"NugetPackageApiOptions": {
"PackageSourceUrl": "http://localhost:5020/v3/index.json",
"PackageSourceTimeoutMs": 10000,
"Username": "admin",
"Password": "Wors123",
"IsPasswordCleartext": true,
"AuthenticationTypes": [ "digest" ],
"CertificateSubjectName": "test.raptor.iotnxt.io"
}
}
- Package Source Url: The URL where the * Password: Associated username’s password to gain access to the NuGet repo. Repo is running.
- Package Source Timeout Ms: Package search and retrieve timeout duration in milliseconds.
- Username: Username is required if authentication type(s) are support and configured within the authentication type property:
- Password: Associated username’s password to gain access to the NuGet repo.
- IsPasswordClearText: Specify if the password saved within the configuration is in clear text, or encrypted. Only clear text supported currently.
- Authentication Types: Specify the authentication type if support by die repo. I.e. Basic or Digest or leave empty for none
- Certificate Subject Name: Certificate CN name the device store will use to verify signed packages.
Multi instance configuration
Multi instance configuration is done within the NugetDevicePackageProviderOptions.json
Same definitions apply as from single instance configuration, only additional property is the Priority property and that we can specify a collection of NuGet Repos.
{
"NugetDevicePackageProviderOptions": {
"PackagesLifetimeMinutes": 1,
"Catalogue": {
"Production": {
"Priority": 1,
"PackageSourceUrl": "http://localhost:5020/v3/index.json",
"PackageSourceTimeoutMs": 10000,
"Username": "Administrator",
"Password": "Wors123!@",
"IsPasswordCleartext": true,
"AuthenticationTypes": [ "Digest" ],
"CertificateSubjectName": "prod.raptor.iotnxt.io"
},
"QA": {
"Priority": 2,
"PackageSourceUrl": "http://localhost:5010/v3/index.json",
"PackageSourceTimeoutMs": 10000,
"Username": "QAAdmin",
"Password": "Wors123!",
"IsPasswordCleartext": true,
"AuthenticationTypes": [ "Digest" ],
"CertificateSubjectName": "qa.raptor.iotnxt.io"
},
"Dev": {
"Priority": 3,
"PackageSourceUrl": "http://localhost:5000/v3/index.json",
"PackageSourceTimeoutMs": 10000,
"Username": "DevAdmin",
"Password": "Wors123",
"IsPasswordCleartext": true,
"AuthenticationTypes": [ "Digest" ],
"CertificateSubjectName": "dev.raptor.iotnxt.io"
}
}
}
}
- Package Source Url: The URL where the * Password: Associated username’s password to gain access to the NuGet repo. Repo is running.
- Package Source Timeout Ms: Package search and retrieve timeout duration in milliseconds.
- Username: Username is required if authentication type(s) are support and configured within the authentication type property:
- Password: Associated username’s password to gain access to the NuGet repo.
- IsPasswordClearText: Specify if the password saved within the configuration is in clear text, or encrypted. Only clear text supported currently.
- Authentication Types: Specify the authentication type if support by die repo. I.e. Basic or Digest or leave empty for none
- Certificate Subject Name: Certificate CN name the device store will use to verify signed packages.
Caching layer
The device store client also has an implemented caching layer to assist in minimizing traffic between the V-Raptor and multiple device stores.
The caching layer is however only available when using the multi-store configuration. Multi instance configuration is done within the NugetDevicePackageProviderOptions.json. Note the PackagesLifetimeMinutes configuration property which specifies the amount of time in minutes which the packages are cached for.
Packages are cleared on the cache when expired, and then will be loaded again if required.
Security Configurations - Digest
The device store service has the ability to specify authentication configuration between the device store service and the device store. Supported authentication implementations are Digset;Basic or None.
Our current recommendation is to work with Digest; See NugetDevicePackageProviderOptions.json for available options.
Migrate Device Store Drivers
Overview.
Migrate drivers or NuGet packages from one NuGet repo instance, also called device or driver store, to another repo.
This is achieved by downloading the NuGet packages using the NuGet api, and then uploading to the destination store/repo by using a tool called raptorpm.
Prerequisite:
In order to execute the device store migration. We need to have dotnet 3.1 installed and also the raptorpm tool to distribute the packages to the destination store once downloaded.
Once dotnet is installed. Execute the bash installRaptorpm.sh script to install the raptorpm tool.
The install raptorpm bash script will execute the following:
- Setup a NuGet configuration file with the NuGet source and credentials.
- The NuGet credentials should be applicable to where the dotnet tool is hosted. In this instance the package-testing artifacts store on azure.
- The script will then install the tool using dotnet commands.
The script has to be executed by supplying the following arguments:
Install the raptorpm dotnet cli tool form a NuGet repository or source.
Usage:
installRaptorpm.sh n=<source name> s=<source url> u=<source username> p=<source password>
Arguments:
Source Name, n Specify a source name. I.e. Azure-Artifact-Repo
Source Url, s Specify the source's url I.e. https://azure-artifact-repo/nuget/v3/index.json
Source Username, u Specify the username required to authentication to the source. I.e. admin
Source Password, p Specify the password. I.e. password
Sample:
./installRaptorpm.sh n=package-testing s=https://pkgs.dev.azure.com/iotnxt/_packaging/package-testing/nuget/v3/index.json u=[email protected] p=luxuhqhab8zz3jwhmwtzzmn2k6ous6ao5paet7lcweb5pzgpyewq
Configure
Migrating the NuGet packages from one store to a next can be achieved by first configuring the source store and then the destination store. The configure.sh script will assist in achieving this.
The script has to be executed by supplying the following arguments:
Configure source and destination NuGet repository details such as authentication and store location.
Usage:
configure.sh s=<source repo url> n=<source repo name> u=<source repo username> p=<source repo password> l=<destination repo url> r=<destination repo username> w=<destination repo password> k=<destination repo api-key> v=<log verbose>
Arguments:
Source Name, n Specify a source name. I.e. Azure-Artifact-Repo
Source Url, s Specify the source's url I.e. https://azure-artifact-repo/nuget/v3/index.json
Source Username, u Specify the username required to authentication to the source. I.e. admin
Source Password, p Specify the password. I.e. password
Destination Url, l Specify the destination repo url. I.e. https://azure-artifact-repo/v3/index.json
Destination Username, r Specify the username required to authenticate to the destination repo. I.e. admin
Destination Password, w Specify the destination password. I.e. password
Note that only to supply username and/or password if applicable. In other words if the repo supports digest authentication.
Sample
./configure.sh s=https://cloudev-vraptor.iotnxt.io:7443/v3/index.json n=cloudev v=true l=http://localhost:5090/v3/index.json r=admin w=Wors123 k=iot-nxt-api-key
Migration
To initiate the migration process, run the migratestore.sh script.
The script has to be executed by supplying the following arguments:
Migrate drivers store drivers or packages from source repo to destination repo.
Usage:
migratestore.sh s=<source name> v=<log verbose> l=<only latest versions>
Arguments:
Source Name, s Specify a source name. I.e. Azure-Artifact-Repo. This value is from the configuration step
Verbose Logging, v Specify true if verbose logging enabled. I.e. v=true, default value [v=false]
Latest Versions, l Specify true if only the latest driver versions should be migrated. Specify false if all the driver package versions should be migrated.
Sample:
./migratestore.sh s=cloudev v=true l=true`
Purge Device Store
Overview.
In the instances where it is required to purge a NuGet repo or driver store, we can use the following.
Configure
Purging NuGet packages from one store or another can be achieved by first configuring the source store. The configure.sh script will assist in achieving this.
The script has to be executed by supplying the following arguments:
Configure source and destination NuGet repository details such as authentication and store location.
Usage:
configure.sh s=<source repo url> n=<source repo name> u=<source repo username> p=<source repo password> l=<destination repo url> r=<destination repo username> w=<destination repo password> k=<destination repo api-key> v=<log verbose>
Arguments:
Source Name, n Specify a source name. I.e. Azure-Artifact-Repo
Source Url, s Specify the source's url I.e. https://azure-artifact-repo/nuget/v3/index.json
Source Username, u Specify the username required to authentication to the source. I.e. admin
Source Password, p Specify the password. I.e. password
Note that only to supply username and/or password if applicable. In other words if the repo supports digest authentication.
Sample
./configure.sh s=http://localhost:5090/v3/index.json n=localDev u=admin p=Wors123
Purge
To initiate the purging process, run the purge.sh script.
The script has to be executed by supplying the following arguments:
Purge drivers or packages from source repo.
Usage:
purge.sh s=<source name> v=<log verbose> k=<api key>
Arguments:
Source Name, s Specify a source name. I.e. Azure-Artifact-Repo. This value is from the configuration step
Verbose Logging, v Specify true if verbose logging enabled. I.e. v=true, default value [v=false]
Api-Key, k Specify the api key required to delete or purge drivers.
Sample:
./purge.sh s=localDev v=true k=iot-nxt-api-key