Hybrid Cloud on Qdrant - Vector Database

Setup Hybrid Cloud

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Creating a Hybrid Cloud Environment

The following instruction set will show you how to properly set up a Qdrant cluster in your Hybrid Cloud Environment.

You can also watch a video demo on how to set up a Hybrid Cloud Environment:

To learn how Hybrid Cloud works, read the overview document.

Prerequisites

Qdrant Hybrid Cloud is available as part of our Enterprise plan. To get access to Qdrant Hybrid Cloud, please contact us.

Create a Cluster

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Creating a Qdrant Cluster in Hybrid Cloud

Once a Hybrid Cloud Environment has been created you can follow the normal process to create a Qdrant cluster in that environment. This page also contains additional information on how to create a production-ready cluster.

Make sure to select your Hybrid Cloud Environment as the target.

Note that in the “Kubernetes Configuration” section you can additionally configure:

NodeSelectors for the Qdrant database pods
Toleration for the Qdrant database pods
TopologySpreadConstraints for the Qdrant database pods
Additional labels for the Qdrant database pods
A service type and annotations for the Qdrant database service

These settings can also be changed after the cluster is created on the cluster detail page.

Configure, Scale & Update Clusters

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Configure, Scale & Update Qdrant Hybrid Cloud Clusters

Configure Clusters

Alongside Hybrid Cloud specific scheduling options, you can also adjust various other advanced configuration options for your clusters. See Configure Clusters for more details.

Scale Clusters

Hybrid cloud clusters can be scaled up and down, horizontall and vertically, at any time. For more details see Scale Clusters.

Automatic Shard Rebalancing

Qdrant Cloud supports automatic shard rebalancing when scaling your cluster horizontally. This ensures that data is evenly distributed across the nodes, optimizing performance and resource utilization. For more details see Shard Rebalancing.

Configure the Qdrant Operator

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Configuring Qdrant Operator: Advanced Options

The Qdrant Operator has several configuration options, which can be configured in the advanced section of your Hybrid Cloud Environment.

The following YAML shows all configuration options with their default values:

# Additional pod annotations
podAnnotations: {}

# Configuration for the Qdrant operator service monitor to scrape metrics
serviceMonitor:
 enabled: false

# Resource requests and limits for the Qdrant operator
resources: {}

# Node selector for the Qdrant operator
nodeSelector: {}

# Tolerations for the Qdrant operator
tolerations: []

# Affinity configuration for the Qdrant operator
affinity: {}

# Configuration for the Qdrant operator (v2)
settings:
 # The log level for the operator
 # Available options: DEBUG | INFO | WARN | ERROR
 logLevel: INFO 
 # Controller related settings
 controller:
 # The period a forced recync is done by the controller (if watches are missed / nothing happened)
 forceResyncPeriod: 10h
 # QPS indicates the maximum QPS to the master from this client.
 # Default is 200
 qps: 200
 # Maximum burst for throttle.
 # Default is 500.
 burst: 500
 # Features contains the settings for enabling / disabling the individual features of the operator
 features:
 # ClusterManagement contains the settings for qdrant (database) cluster management
 clusterManagement:
 # Whether or not the Qdrant cluster features are enabled.
 # If disabled, all other properties in this struct are disregarded. Otherwise, the individual features will be inspected.
 # Default is true.
 enable: true
 # The StorageClass used to make database and snapshot PVCs.
 # Default is nil, meaning the default storage class of Kubernetes.
 storageClass: 
 # The StorageClass used to make database PVCs.
 # Default is nil, meaning the default storage class of Kubernetes.
 #database: 
 # The StorageClass used to make snapshot PVCs.
 # Default is nil, meaning the default storage class of Kubernetes.
 #snapshot:
 # Qdrant config contains settings specific for the database
 qdrant:
 # The config where to find the image for qdrant
 image: 
 # The repository where to find the image for qdrant
 # Default is "qdrant/qdrant"
 repository: qdrant/qdrant
 # Docker image pull policy
 # Default "IfNotPresent", unless the tag is dev, master or latest. Then "Always"
 #pullPolicy:
 # Docker image pull secret name
 # This secret should be available in the namespace where the cluster is running
 # Default not set
 #pullSecretName:
 # storage contains the settings for the storage of the Qdrant cluster
 storage:
 performance:
 # CPU budget, how many CPUs (threads) to allocate for an optimization job.
 # If 0 - auto selection, keep 1 or more CPUs unallocated depending on CPU size
 # If negative - subtract this number of CPUs from the available CPUs.
 # If positive - use this exact number of CPUs.
 optimizerCpuBudget: 0
 # Enable async scorer which uses io_uring when rescoring.
 # Only supported on Linux, must be enabled in your kernel.
 # See: <https://qdrant.tech/articles/io_uring/#and-what-about-qdrant>
 asyncScorer: false
 # Qdrant DB log level
 # Available options: DEBUG | INFO | WARN | ERROR 
 # Default is "INFO"
 logLevel: INFO
 # Default Qdrant security context configuration
 securityContext:
 # Enable default security context
 # Default is false
 enabled: false
 # Default user for qdrant container
 # Default not set
 #user: 1000
 # Default fsGroup for qdrant container
 # Default not set
 #fsUser: 2000
 # Default group for qdrant container
 # Default not set
 #group: 3000
 # Network policies configuration for the Qdrant databases
 networkPolicies:
 ingress:
 - ports:
 - protocol: TCP
 port: 6333
 - protocol: TCP
 port: 6334
 # Allow DNS resolution from qdrant pods at Kubernetes internal DNS server
 egress:
 - ports:
 - protocol: UDP
 port: 53
 # Scheduling config contains the settings specific for scheduling
 scheduling:
 # Default topology spread constraints (list from type corev1.TopologySpreadConstraint)
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: "kubernetes.io/hostname"
 whenUnsatisfiable: "ScheduleAnyway"
 # Default pod disruption budget (object from type policyv1.PodDisruptionBudgetSpec)
 podDisruptionBudget:
 maxUnavailable: 1
 # ClusterManager config contains the settings specific for cluster manager
 clusterManager:
 # Whether or not the cluster manager (on operator level).
 # If disabled, all other properties in this struct are disregarded. Otherwise, the individual features will be inspected.
 # Default is false.
 enable: true
 # The endpoint address the cluster manager could be reached
 # If set, this should be a full URL like: http://cluster-manager.qdrant-cloud-ns.svc.cluster.local:7333
 endpointAddress: http://qdrant-cluster-manager:80
 # InvocationInterval is the interval between calls (started after the previous call is retured)
 # Default is 10 seconds
 invocationInterval: 10s
 # Timeout is the duration a single call to the cluster manager is allowed to take.
 # Default is 30 seconds
 timeout: 30s
 # Specifies overrides for the manage rules
 manageRulesOverrides:
 #dry_run: 
 #max_transfers:
 #max_transfers_per_collection:
 #rebalance:
 #replicate:
 # Ingress config contains the settings specific for ingress
 ingress:
 # Whether or not the Ingress feature is enabled.
 # Default is true.
 enable: false
 # Which specific ingress provider should be used
 # Default is KubernetesIngress
 provider: KubernetesIngress
 # The specific settings when the Provider is QdrantCloudTraefik
 qdrantCloudTraefik:
 # Enable tls
 # Default is false
 tls: false
 # Secret with TLS certificate
 # Default is None
 secretName: ""
 # List of Traefik middlewares to apply
 # Default is an empty list
 middlewares: []
 # IP Allowlist Strategy for Traefik
 # Default is None
 ipAllowlistStrategy:
 # Enable body validator plugin and matching ingressroute rules
 # Default is false
 enableBodyValidatorPlugin: false
 # The specific settings when the Provider is KubernetesIngress
 kubernetesIngress:
 # Name of the ingress class
 # Default is None
 #ingressClassName:
 # TelemetryTimeout is the duration a single call to the cluster telemetry endpoint is allowed to take.
 # Default is 3 seconds
 telemetryTimeout: 3s
 # MaxConcurrentReconciles is the maximum number of concurrent Reconciles which can be run. Defaults to 20.
 maxConcurrentReconciles: 20 
 # VolumeExpansionMode specifies the expansion mode, which can be online or offline (e.g. in case of Azure).
 # Available options: Online, Offline
 # Default is Online
 volumeExpansionMode: Online
 # BackupManagementConfig contains the settings for backup management
 backupManagement:
 # Whether or not the backup features are enabled.
 # If disabled, all other properties in this struct are disregarded. Otherwise, the individual features will be inspected.
 # Default is true.
 enable: true
 # Snapshots contains the settings for snapshots as part of backup management.
 snapshots:
 # Whether or not the Snapshot feature is enabled.
 # Default is true.
 enable: true
 # The VolumeSnapshotClass used to make VolumeSnapshots.
 # Default is "csi-snapclass".
 volumeSnapshotClass: "csi-snapclass"
 # The duration a snapshot is retained when the phase becomes Failed or Skipped
 # Default is 72h (3d).
 retainUnsuccessful: 72h
 # MaxConcurrentReconciles is the maximum number of concurrent Reconciles which can be run. Defaults to 1.
 maxConcurrentReconciles: 1
 # ScheduledSnapshots contains the settings for scheduled snapshot as part of backup management.
 scheduledSnapshots:
 # Whether or not the ScheduledSnapshot feature is enabled.
 # Default is true.
 enable: true
 # MaxConcurrentReconciles is the maximum number of concurrent Reconciles which can be run. Defaults to 1.
 maxConcurrentReconciles: 1
 # Restores contains the settings for restoring (a snapshot) as part of backup management.
 restores:
 # Whether or not the Restore feature is enabled.
 # Default is true.
 enable: true
 # MaxConcurrentReconciles is the maximum number of concurrent Reconciles which can be run. Defaults to 1.
 maxConcurrentReconciles: 1

Networking, Logging & Monitoring

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Configuring Networking, Logging & Monitoring in Qdrant Hybrid Cloud

Configure network policies

For security reasons, each database cluster is secured with network policies. By default, database pods only allow egress traffic between each and allow ingress traffic to ports 6333 (rest) and 6334 (grpc) from within the Kubernetes cluster.

You can modify the default network policies in the Hybrid Cloud environment configuration:

qdrant:
 networkPolicies:
 ingress:
 - from:
 - ipBlock:
 cidr: 192.168.0.0/22
 - podSelector:
 matchLabels:
 app: client-app
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: client-namespace
 - podSelector:
 matchLabels:
 app: traefik
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kube-system
 ports:
 - port: 6333
 protocol: TCP
 - port: 6334
 protocol: TCP

Logging

You can access the logs with kubectl or the Kubernetes log management tool of your choice. For example:

Deployment Platforms

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Qdrant Hybrid Cloud: Hosting Platforms & Deployment Options

This page provides an overview of how to deploy Qdrant Hybrid Cloud on various managed Kubernetes platforms.

For a general list of prerequisites and installation steps, see our Hybrid Cloud setup guide. This platform specific documentation also applies to Qdrant Private Cloud.

Akamai (Linode)

The Linode Kubernetes Engine (LKE) is a managed container orchestration engine built on top of Kubernetes. LKE enables you to quickly deploy and manage your containerized applications without needing to build (and maintain) your own Kubernetes cluster. All LKE instances are equipped with a fully managed control plane at no additional cost.