Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions packages/couchbase/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ For example, you could use the data from this integration to know when there are

The Couchbase integration collects metrics data.

Metrics give you insight into the state of the Couchbase. Metrics data streams collected by the Couchbase integration include [Bucket](https://docs.couchbase.com/server/current/rest-api/rest-buckets-summary.html), [Cluster](https://docs.couchbase.com/server/current/rest-api/rest-cluster-details.html), [Couchbase Lite Replication](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cbl_replication_pull), [Delta Sync](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#delta_sync), [GSI views](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#gsi_views), [Import](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#shared_bucket_import), [Resource Utilization](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#resource_utilization), and [Security](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#security) metrics from [Couchbase](https://www.couchbase.com/) so that the user could monitor and troubleshoot the performance of the Couchbase instances.
Metrics give you insight into the state of the Couchbase. Metrics data streams collected by the Couchbase integration include [Bucket](https://docs.couchbase.com/server/current/rest-api/rest-buckets-summary.html), [Cluster](https://docs.couchbase.com/server/current/rest-api/rest-cluster-details.html), [Cache](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cache), [Couchbase Lite Replication](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#cbl_replication_pull), [Database](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#database), [Delta Sync](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#delta_sync), [GSI views](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#gsi_views), [Import](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#shared_bucket_import), [Resource Utilization](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#resource_utilization), [Security](https://docs.couchbase.com/sync-gateway/current/stats-monitoring.html#security), and [XDCR](https://docs.couchbase.com/server/current/rest-api/rest-bucket-stats.html) metrics from [Couchbase](https://www.couchbase.com/) so that the user could monitor and troubleshoot the performance of the Couchbase instances.

This integration uses:
- `http` metricbeat module to collect `bucket` and `cluster` metrics.
- `prometheus` metricbeat module to collect `cbl_replication`, `delta_sync`, `gsi_views`, `import`, `resource`, and `security` metrics.
- `http` metricbeat module to collect `bucket`, `cluster`, and `xdcr` metrics.
- `prometheus` metricbeat module to collect `cache`, `cbl_replication`, `database_stats`, `delta_sync`, `gsi_views`, `import`, `resource`, and `security` metrics.

Note: For Couchbase cluster setup, there is an ideal scenario of a single host with administrator access for the entire cluster to collect metrics. Providing multiple hosts from the same cluster might lead to data duplication. In the case of multiple clusters, adding a new integration to collect data from different cluster hosts is a good option.

Expand Down Expand Up @@ -94,3 +94,13 @@ This is the `resource` data stream. The Resource Utilization metrics are related
{{event "resource"}}

{{fields "resource"}}

### XDCR

This is the `xdcr` data stream. Cross Data Center Replication (XDCR) replicates data between a source bucket and a target bucket. XDCR collects metrics related to statistics of XDCR. Metrics can be fetched from multiple buckets.

Note: It is preferable to add a new integration if user requires to fetch metrics from multiple hosts for XDCR data stream.

{{event "xdcr"}}

{{fields "xdcr"}}
5 changes: 4 additions & 1 deletion packages/couchbase/_dev/deploy/docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@ services:
couchbase:
image: couchbase/server-sandbox:${COUCHBASE_VERSION:-7.1.0}
hostname: cb
volumes:
- ./files/setup.sh:/setup.sh
- ./files/entrypoint.sh:/entrypoint.sh
healthcheck:
test: ["CMD", "curl", "-f", "http://admin:password@localhost:8091/pools/default"]
test: ["CMD", "curl", "-f", "http://admin:password@localhost:8091/pools/default/"]
interval: 2s
timeout: 2s
retries: 40
Expand Down
60 changes: 60 additions & 0 deletions packages/couchbase/_dev/deploy/docker/files/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/bin/bash
set -e

staticConfigFile=/opt/couchbase/etc/couchbase/static_config
restPortValue=8091

# see https://developer.couchbase.com/documentation/server/current/install/install-ports.html
function overridePort() {
portName=$1
portNameUpper=$(echo $portName | awk '{print toupper($0)}')
portValue=${!portNameUpper}

# only override port if value available AND not already contained in static_config
if [ "$portValue" != "" ]; then
if grep -Fq "{${portName}," ${staticConfigFile}
then
echo "Don't override port ${portName} because already available in $staticConfigFile"
else
echo "Override port '$portName' with value '$portValue'"
echo "{$portName, $portValue}." >> ${staticConfigFile}

if [ ${portName} == "rest_port" ]; then
restPortValue=${portValue}
fi
fi
fi
}

overridePort "rest_port"
overridePort "mccouch_port"
overridePort "memcached_port"
overridePort "query_port"
overridePort "ssl_query_port"
overridePort "fts_http_port"
overridePort "moxi_port"
overridePort "ssl_rest_port"
overridePort "ssl_capi_port"
overridePort "ssl_proxy_downstream_port"
overridePort "ssl_proxy_upstream_port"


[[ "$1" == "couchbase-server" ]] && {

if [ "$(whoami)" = "couchbase" ]; then
# Ensure that /opt/couchbase/var is owned by user 'couchbase' and
# is writable
if [ ! -w /opt/couchbase/var -o \
$(find /opt/couchbase/var -maxdepth 0 -printf '%u') != "couchbase" ]; then
echo "/opt/couchbase/var is not owned and writable by UID 1000"
echo "Aborting as Couchbase Server will likely not run"
exit 1
fi
fi
echo "Starting Couchbase Server -- Web UI available at http://<ip>:$restPortValue"
echo "and logs available in /opt/couchbase/var/lib/couchbase/logs"
sh -c /setup.sh &
exec runsvdir -P /etc/service 'log: ...........................................................................................................................................................................................................................................................................................................................................................................................................'
}

exec "$@"
20 changes: 20 additions & 0 deletions packages/couchbase/_dev/deploy/docker/files/setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# wait for the couchbase-server starts
until curl -f http://Administrator:password@localhost:8091/pools/default
do
sleep 5s
done

# add "beer-sample" bucket from sampleBuckets
curl -v -u Administrator:password -X POST http://127.0.0.1:8091/sampleBuckets/install -d '["beer-sample"]'

# using couchbase-cli run xdcr-setup for the cluster
couchbase-cli xdcr-setup -c 127.0.0.1 -u Administrator -p password --create --xdcr-cluster-name cluster --xdcr-hostname 127.0.0.1 --xdcr-username Administrator --xdcr-password password

# wait till the xdcr-setup creates cluster
until curl -f http://Administrator:password@localhost:8091/pools/default/buckets/beer-sample/stats
do
sleep 5s
done

# perform replication from "beer-sample" to "travel-sample"
curl -v -X POST -u Administrator:password http://127.0.0.1:8091/controller/createReplication -d fromBucket=beer-sample -d toCluster=cluster -d toBucket=travel-sample -d replicationType=continuous -d enableCompression=1
4 changes: 0 additions & 4 deletions packages/couchbase/_dev/deploy/variants.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
variants:
v7.1.0:
COUCHBASE_VERSION: 7.1.0
v7.0.2:
COUCHBASE_VERSION: 7.0.2
v6.6.0:
COUCHBASE_VERSION: 6.6.0
Comment thread
ritalwar marked this conversation as resolved.
default: v7.1.0
5 changes: 5 additions & 0 deletions packages/couchbase/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "0.6.0"
changes:
- description: Couchbase integration package with "xdcr" data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/4073
- version: "0.5.0"
changes:
- description: Couchbase integration package with "resource" data stream.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,6 @@ processors:
value: [database]
ignore_failure: true
ignore_empty_value: true
- set:
field: event.ingested
value: "{{{_ingest.timestamp}}}"
ignore_failure: true
ignore_empty_value: true
- set:
field: event.module
value: couchbase
Expand Down
33 changes: 17 additions & 16 deletions packages/couchbase/data_stream/bucket/sample_event.json
Original file line number Diff line number Diff line change
@@ -1,40 +1,40 @@
{
"@timestamp": "2022-07-22T10:40:36.032Z",
"@timestamp": "2022-09-22T12:12:39.838Z",
"agent": {
"ephemeral_id": "b6b8e21b-ded1-41d8-a193-c5aead533ff1",
"id": "5d67808a-0fe5-4f5f-9636-ec161f0cdcf0",
"ephemeral_id": "c8726d7e-0c72-46ee-bc4a-fc7b5baf11ce",
"id": "e1c61e89-8171-47ed-be0b-eb7f11396b0d",
"name": "docker-fleet-agent",
"type": "metricbeat",
"version": "8.3.2"
"version": "8.4.1"
},
"couchbase": {
"bucket": {
"data": {
"used": {
"bytes": 20892210
"bytes": 4578094
}
},
"disk": {
"fetches": 0,
"used": {
"bytes": 20914347
"bytes": 15977057
}
},
"item": {
"count": 7303
},
"memory": {
"used": {
"bytes": 34972008
"bytes": 35163472
}
},
"name": "beer-sample",
"operations_per_sec": 0,
"ram": {
"quota": {
"bytes": 104857600,
"bytes": 209715200,
"used": {
"pct": 33.35190582275391
"pct": 16.76725006103516
}
}
},
Expand All @@ -50,18 +50,18 @@
"version": "8.3.0"
},
"elastic_agent": {
"id": "5d67808a-0fe5-4f5f-9636-ec161f0cdcf0",
"id": "e1c61e89-8171-47ed-be0b-eb7f11396b0d",
"snapshot": false,
"version": "8.3.2"
"version": "8.4.1"
},
"event": {
"agent_id_status": "verified",
"category": [
"database"
],
"dataset": "couchbase.bucket",
"duration": 6674276,
"ingested": "2022-07-22T10:40:39Z",
"duration": 106105674,
"ingested": "2022-09-22T12:12:43Z",
"kind": "metric",
"module": "couchbase",
"type": [
Expand All @@ -72,17 +72,18 @@
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "51511c1493f34922b559a964798246ec",
"ip": [
"172.26.0.7"
"192.168.128.7"
],
"mac": [
"02:42:ac:1a:00:07"
"02:42:c0:a8:80:07"
],
"name": "docker-fleet-agent",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.4.0-110-generic",
"kernel": "5.4.0-126-generic",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,6 @@ processors:
value: [database]
ignore_failure: true
ignore_empty_value: true
- set:
field: event.ingested
value: "{{{_ingest.timestamp}}}"
ignore_failure: true
ignore_empty_value: true
- set:
field: event.module
value: couchbase
Expand Down
25 changes: 13 additions & 12 deletions packages/couchbase/data_stream/cbl_replication/sample_event.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{
"@timestamp": "2022-08-01T12:29:02.626Z",
"@timestamp": "2022-09-22T12:19:30.637Z",
"agent": {
"ephemeral_id": "e90e29db-34fd-4bba-a4bf-da0c2e5f1d15",
"id": "980dcead-2dc9-4d20-a59b-a81f43d3a52c",
"ephemeral_id": "bdda1dcc-d922-49f1-a3ea-9e3bdfb6a767",
"id": "e1c61e89-8171-47ed-be0b-eb7f11396b0d",
"name": "docker-fleet-agent",
"type": "metricbeat",
"version": "8.3.2"
"version": "8.4.1"
},
"couchbase": {
"cbl_replication": {
Expand Down Expand Up @@ -61,7 +61,7 @@
},
"sync": {
"function": {
"time": 1732351964
"time": 125292068
}
},
"write": {
Expand All @@ -81,18 +81,18 @@
"version": "8.3.0"
},
"elastic_agent": {
"id": "980dcead-2dc9-4d20-a59b-a81f43d3a52c",
"id": "e1c61e89-8171-47ed-be0b-eb7f11396b0d",
"snapshot": false,
"version": "8.3.2"
"version": "8.4.1"
},
"event": {
"agent_id_status": "verified",
"category": [
"database"
],
"dataset": "couchbase.cbl_replication",
"duration": 50168237,
"ingested": "2022-08-01T12:29:03Z",
"duration": 92869141,
"ingested": "2022-09-22T12:19:32Z",
"kind": "metric",
"module": "couchbase",
"type": [
Expand All @@ -103,17 +103,18 @@
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "51511c1493f34922b559a964798246ec",
"ip": [
"192.168.192.6"
"192.168.128.7"
],
"mac": [
"02:42:c0:a8:c0:06"
"02:42:c0:a8:80:07"
],
"name": "docker-fleet-agent",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.4.0-110-generic",
"kernel": "5.4.0-126-generic",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,6 @@ processors:
value: [database]
ignore_failure: true
ignore_empty_value: true
- set:
field: event.ingested
value: "{{{_ingest.timestamp}}}"
ignore_failure: true
ignore_empty_value: true
- set:
field: event.module
value: couchbase
Expand Down
Loading