Skip to content

Commit

Permalink
Merge branch 'master' into DOC-960
Browse files Browse the repository at this point in the history
  • Loading branch information
Simplychee authored Jan 8, 2025
2 parents b486fa0 + 812b235 commit 94a0fd4
Show file tree
Hide file tree
Showing 20 changed files with 299 additions and 176 deletions.
7 changes: 3 additions & 4 deletions docs/_include/general-shipping/data-shipping-tokens_vars.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@

From your account, go to the <a href="https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping" target ="_blank"> **Manage Tokens** > **Data shipping tokens** tab.</a> of your Operations workspace. It can be reached by selecting **<i class="li li-gear"></i> > Settings > Manage tokens** in the **ADMIN ZONE** section of the side navigation menu.
From your account, go to the <a href="https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping" target ="_blank"> **Manage Tokens** > **Data shipping tokens** tab</a>. Or select **<i class="li li-gear"></i> > Settings > Manage tokens** in the side navigation menu.

+ Click the product option.
+ Select the relevant telemetry (Logs, Metrics, or Traces).
+ The Listener URL for your account is displayed above the token table.
+ The token for each account is listed in the table along with the date it was created.

+ The token for each account is listed in the table along with the date it was created.
2 changes: 1 addition & 1 deletion docs/shipping/AWS/aws-ec2.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ service:
exporters: [prometheusremotewrite]
telemetry:
logs:
level: "debug"
level: "info"
metrics:
address: localhost:8888
```
Expand Down
2 changes: 1 addition & 1 deletion docs/shipping/AWS/aws-ecs-fargate.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ service:
exporters: [ prometheusremotewrite ]
telemetry:
logs:
level: "debug"
level: "info"
```
### AWS::Lambda::Function
The `logzioFirehoseSubscriptionFiltersFunction` Lambda function is designed to handle the process of filtering log data for Logz.io, and add and remove cloudwatch logs subscription filters
Expand Down
8 changes: 6 additions & 2 deletions docs/shipping/App360/App360.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,11 +174,15 @@ service:
traces:
receivers: [otlp]
processors: [tail_sampling, batch]
exporters: [logzio/traces, spanmetrics]
exporters: [logzio/traces]
traces/spm:
receivers: [otlp]
processors: [batch]
exporters: [spanmetrics]
metrics/spanmetrics:
receivers: [spanmetrics]
processors: [metricstransform/metrics-rename, metricstransform/labels-rename]
exporters: [prometheusremotewrite/spm]
exporters: [prometheusremotewrite/spm]
telemetry:
logs:
level: "debug"
Expand Down
225 changes: 112 additions & 113 deletions docs/shipping/Code/go.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,118 @@ if err != nil {
l.Stop() // Drains the log buffer
```

</TabItem>
<TabItem value="OpenTelemetry" label="OpenTelemetry">

This integration uses the OpenTelemetry logging exporter to send logs to Logz.io via the OpenTelemetry Protocol (OTLP) listener.

### Prerequisites

- Go 1.21 or newer

:::note
If you need an example aplication to test this integration, please refer to our [Go OpenTelemetry repository](https://github.com/logzio/opentelemetry-examples/tree/main/go/logs).
:::

### Configure the instrumentation


1. Install OpenTelemetry dependencies:

```bash
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp
go get go.opentelemetry.io/otel/exporters/stdout/stdoutlog
```

2. Create a new file named `otel.go` and add the following code to set up OpenTelemetry logging:


```go
package main

import (
"context"
"fmt"
"log"

"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
"go.opentelemetry.io/otel/exporters/stdout/stdoutlog"
"go.opentelemetry.io/otel/log/global"
sdklog "go.opentelemetry.io/otel/sdk/log"
)

func newLoggerProvider() (*sdklog.LoggerProvider, error) {
// Create stdout log exporter
stdoutExporter, err := stdoutlog.New(stdoutlog.WithPrettyPrint())
if err != nil {
return nil, fmt.Errorf("failed to create stdout exporter: %w", err)
}

// Create OTLP HTTP log exporter for Logz.io
httpExporter, err := otlploghttp.New(context.Background(),
otlploghttp.WithEndpoint("otlp-listener.logz.io"),
otlploghttp.WithHeaders(map[string]string{
"Authorization": "Bearer <LOG-SHIPPING-TOKEN>",
"user-agent": "logzio-go-logs-otlp",
}),
otlploghttp.WithURLPath("/v1/logs"),
)
if err != nil {
return nil, fmt.Errorf("failed to create OTLP HTTP exporter: %w", err)
}

// Create a logger provider with both exporters
loggerProvider := sdklog.NewLoggerProvider(
sdklog.WithProcessor(sdklog.NewBatchProcessor(stdoutExporter)), // For stdout
sdklog.WithProcessor(sdklog.NewBatchProcessor(httpExporter)), // For HTTP export
)

return loggerProvider, nil
}

// setupOTelSDK bootstraps the OpenTelemetry logging pipeline.
func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) {
// Set up logger provider.
loggerProvider, err := newLoggerProvider()
if err != nil {
return nil, err
}

// Set the global logger provider
global.SetLoggerProvider(loggerProvider)

// Return a shutdown function
shutdown = func(ctx context.Context) error {
err := loggerProvider.Shutdown(ctx)
if err != nil {
log.Printf("Error during logger provider shutdown: %v", err)
}
return err
}

return shutdown, nil
}

```


{@include: ../../_include/log-shipping/log-shipping-token.md}
Update the `listener.logz.io` part in `https://otlp-listener.logz.io/v1/logs` with the URL for [your hosting region](https://docs.logz.io/docs/user-guide/admin/hosting-regions/account-region).


3. Run your application.

### Check Logz.io for your logs


Allow some time for data ingestion, then open [Open Search Dashboards](https://app.logz.io/#/dashboard/osd).

Encounter an issue? See our [log shipping troubleshooting](https://docs.logz.io/docs/user-guide/log-management/troubleshooting/log-shipping-troubleshooting/) guide.


</TabItem>
</Tabs>

## Metrics

Expand Down Expand Up @@ -379,119 +491,6 @@ Install the pre-built dashboard to enhance the observability of your metrics.

{@include: ../../_include/metric-shipping/generic-dashboard.html}

</TabItem>
<TabItem value="OpenTelemetry" label="OpenTelemetry">

This integration uses the OpenTelemetry logging exporter to send logs to Logz.io via the OpenTelemetry Protocol (OTLP) listener.

### Prerequisites

- Go 1.21 or newer

:::note
If you need an example aplication to test this integration, please refer to our [Go OpenTelemetry repository](https://github.com/logzio/opentelemetry-examples/tree/main/go/logs).
:::

### Configure the instrumentation


1. Install OpenTelemetry dependencies:

```bash
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp
go get go.opentelemetry.io/otel/exporters/stdout/stdoutlog
```

2. Create a new file named `otel.go` and add the following code to set up OpenTelemetry logging:


```go
package main

import (
"context"
"fmt"
"log"

"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
"go.opentelemetry.io/otel/exporters/stdout/stdoutlog"
"go.opentelemetry.io/otel/log/global"
sdklog "go.opentelemetry.io/otel/sdk/log"
)

func newLoggerProvider() (*sdklog.LoggerProvider, error) {
// Create stdout log exporter
stdoutExporter, err := stdoutlog.New(stdoutlog.WithPrettyPrint())
if err != nil {
return nil, fmt.Errorf("failed to create stdout exporter: %w", err)
}

// Create OTLP HTTP log exporter for Logz.io
httpExporter, err := otlploghttp.New(context.Background(),
otlploghttp.WithEndpoint("otlp-listener.logz.io"),
otlploghttp.WithHeaders(map[string]string{
"Authorization": "Bearer <LOG-SHIPPING-TOKEN>",
"user-agent": "logzio-go-logs-otlp",
}),
otlploghttp.WithURLPath("/v1/logs"),
)
if err != nil {
return nil, fmt.Errorf("failed to create OTLP HTTP exporter: %w", err)
}

// Create a logger provider with both exporters
loggerProvider := sdklog.NewLoggerProvider(
sdklog.WithProcessor(sdklog.NewBatchProcessor(stdoutExporter)), // For stdout
sdklog.WithProcessor(sdklog.NewBatchProcessor(httpExporter)), // For HTTP export
)

return loggerProvider, nil
}

// setupOTelSDK bootstraps the OpenTelemetry logging pipeline.
func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) {
// Set up logger provider.
loggerProvider, err := newLoggerProvider()
if err != nil {
return nil, err
}

// Set the global logger provider
global.SetLoggerProvider(loggerProvider)

// Return a shutdown function
shutdown = func(ctx context.Context) error {
err := loggerProvider.Shutdown(ctx)
if err != nil {
log.Printf("Error during logger provider shutdown: %v", err)
}
return err
}

return shutdown, nil
}

```


{@include: ../../_include/log-shipping/log-shipping-token.md}
Update the `listener.logz.io` part in `https://otlp-listener.logz.io/v1/logs` with the URL for [your hosting region](https://docs.logz.io/docs/user-guide/admin/hosting-regions/account-region).


3. Run your application.

### Check Logz.io for your logs


Allow some time for data ingestion, then open [Open Search Dashboards](https://app.logz.io/#/dashboard/osd).

Encounter an issue? See our [log shipping troubleshooting](https://docs.logz.io/docs/user-guide/log-management/troubleshooting/log-shipping-troubleshooting/) guide.


</TabItem>
</Tabs>


## Traces

Expand Down
21 changes: 13 additions & 8 deletions docs/shipping/Code/nestjs.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,19 +327,24 @@ logzio-k8s-telemetry logzio-helm/logzio-k8s-telemetry
{@include: ../../_include/tracing-shipping/replace-tracing-token.html}
`<<LOGZIO_ACCOUNT_REGION_CODE>>` - Your Logz.io account region code. [Available regions](https://docs.logz.io/docs/user-guide/admin/hosting-regions/account-region/#available-regions).

#### Define the logzio-k8s-telemetry service dns
#### Define the service DNS

You'll need the following service DNS:

`http://<<CHART-NAME>>-otel-collector.<<NAMESPACE>>.svc.cluster.local:<<PORT>>/`.

Replace `<<CHART-NAME>>` with the relevant service you're using (`logzio-k8s-telemetry`, `logzio-monitoring`).
Replace `<<NAMESPACE>>` with your Helm chart's deployment namespace (e.g., default or monitoring).
Replace `<<PORT>>` with the [port for your agent's protocol](https://github.com/logzio/logzio-helm/blob/master/charts/logzio-telemetry/values.yaml#L249-L267) (Default is 4317).

If you're not sure what your cluster domain name is, you can run the following command to look it up:

In most cases, the service dns will be `logzio-k8s-telemetry.default.svc.cluster.local`, where `default` is the namespace where you deployed the helm chart and `svc.cluster.name` is your cluster domain name.

If you are not sure what your cluster domain name is, you can run the following command to look it up:

```shell
kubectl run -it --image=k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Never shell -- \
sh -c 'nslookup kubernetes.default | grep Name | sed "s/Name:\skubernetes.default//"'
sh -c 'nslookup kubernetes.<<NAMESPACE>> | grep Name | sed "s/Name:\skubernetes.<<NAMESPACE>>//"'
```

It will deploy a small pod that extracts your cluster domain name from your Kubernetes environment. You can remove this pod after it has returned the cluster domain name.

It will deploy a small pod that extracts your cluster domain name from your Kubernetes environment. You can remove this pod after it has returned the cluster domain name.

#### Download instrumentation packages

Expand Down
20 changes: 12 additions & 8 deletions docs/shipping/Code/node-js.md
Original file line number Diff line number Diff line change
Expand Up @@ -929,21 +929,25 @@ logzio-k8s-telemetry logzio-helm/logzio-k8s-telemetry
{@include: ../../_include/tracing-shipping/replace-tracing-token.html}
`<<LOGZIO_ACCOUNT_REGION_CODE>>` - Your Logz.io account region code. [Available regions](https://docs.logz.io/docs/user-guide/admin/hosting-regions/account-region/#available-regions).
#### Define the logzio-k8s-telemetry dns name
#### Define the DNS name
In most cases, the service name will be `logzio-k8s-telemetry.default.svc.cluster.local`, where `default` is the namespace where you deployed the helm chart and `svc.cluster.name` is your cluster domain name.
You'll need the following service DNS:
`http://<<CHART-NAME>>-otel-collector.<<NAMESPACE>>.svc.cluster.local:<<PORT>>/`.
Replace `<<CHART-NAME>>` with the relevant service you're using (`logzio-k8s-telemetry`, `logzio-monitoring`).
Replace `<<NAMESPACE>>` with your Helm chart's deployment namespace (e.g., default or monitoring).
Replace `<<PORT>>` with the [port for your agent's protocol](https://github.com/logzio/logzio-helm/blob/master/charts/logzio-telemetry/values.yaml#L249-L267) (Default is 4317).
If you're not sure what your cluster domain name is, you can run the following command to look it up:
To find your cluster domain name, run the following command:
```shell
kubectl run -it --image=k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Never shell -- \
shell -c 'nslookup kubernetes.default | grep Name | sed "s/Name:\skubernetes.default//"'
shell -c 'nslookup kubernetes.<<NAMESPACE>> | grep Name | sed "s/Name:\skubernetes.<<NAMESPACE>>//"'
```
This command deploys a temporary pod to extract your cluster domain name. You can remove the pod after retrieving the domain name.
{@include: ../../_include/tracing-shipping/node-steps.md}
Expand Down
Loading

0 comments on commit 94a0fd4

Please sign in to comment.