Mastering Go: Part 12 - Program debugging , Profiling and Performance Evaluation

The Go has several APIs and tools available to evaluate the performance and logic of the program. All the diagnostics tools fall under different categories. The aim of these tools is to identify the performance and/or logical issues with the Go program.

Profiling

Profiling is the process of identifying the expensive code blocks in the project. The pprof tool will be used to visualize the data generated by the profiling. Go has a standard library 'pprof' and can be used to visualize the profiling data. the profiler generated the data in the specific format expected by the pprof tool. The profiling data can be collected during testing via go test or endpoints made available from the net/http/pprof package.  

The ppfrof package contains several of the APIs to provide data in a specified/defined structure.

There are few built-in profiles included in the pprof package

CPU - CPU profile reports

HEAP - heap profile reports

threadcreate - profile reports for the  program that creates new process threads

goroutine - goroutine profile report

block - program block waiting on synchronization  profile report

mutex -lock contention report

Install profiling Tools

To generate and visualize the profile report we need a few tools installed. Let's first make sure all the tools required are installed.

First, install the pprof tool using the below command :

$ go install GitHub.com/google/pprof@latest

Run the below command to make sure pprof is working:

$ pprof —version

Now, install the data visualization tool Graphviz. We can install it using Homebrew (for macOS users) or follow the instructions on this Graphviz download.

Bash command:

$ brew install graphviz

Note: If Homebrew is not already installed on your machine, either use the macOS .dmg file or install Homebrew first.

Generate profile reports

There are several ways to generate a program profile. We can generate profiles while running unit tests, by writing code to generate profiles, or by generating live profiles while running the application/services.

Examples: Generate a profile with a unit test. We are using the unit test written as part of this learning series: Part 11: Testing Go Projects.

To generate the profile using a unit test run, please use the command below:

$ go test -cpuprofile cpu.prof -memprofile mem.prof -bench .

 

After the completion of the test, two files, cpu.prof and mem.prof, will be generated. To open and diagnose these files, we need to use the pprof tool.

For any issue/error installing ppform refer to the appendix section of this blog.

Visualize the profile report

In Terminal:

To View the report in the terminal use the below command:

$pprof mem.prof

S*****s-MacBook-Pro:string_formater s********$ pprof mem.prof
File: string_formater.test
Type: alloc_space
Time: May 24, 2024 at 6:37am (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

S******s-MacBook-Pro:string_formater s********i$ pprof cpu.prof
File: string_formater.test
Type: cpu
Time: May 24, 2024 at 6:37am (EDT)
Duration: 201.83ms, Total samples = 0
No samples were found with the default sample value type.
Try "sample_index" command to analyze different sample values.
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

After executing the above command, we can run several other commands to get insights from the profile. Some of these commands are top, web, etc. Additionally, pprof has several functions to generate specific profiles based on our specific needs.

In the web interface:

We can use the web interface to visualize and read the .prof file.
Command:

$ pprof -http=:8090 cpu.prof

Now, you can visualize the result in the browser using Graphviz.

Note: If you encounter any errors please refer to the common errors sections to fix them:

Writing profiling in the program: 

To write profiling code in your Go main package, you'll need to use the runtime/pprof and os packages to create CPU and memory profiles. Here's an example of how to add profiling code to your main.go file:
We can use a build tag to execute the profiling code only in debug mode.

Example: 

Continue from the previous exercise. Modify the main.go to the file and add the code below to generate CPU and memory profiles. The profile will be generated after each execution of the main. To avoid extra resource consumption in production, we can move the profiling code into a separate file and only execute it in debug mode.

package main

import (

"encoding/json"

"errors"

"flag"

"fmt"

"io"

dateformater "learngo/date_formater" // import data formater package

stringformater "learngo/string_formater" // import string formater package

"log"

"net/http"

"os"

"runtime"

"runtime/pprof"

)

var cpuprofile = flag.String("cpuprofile", "cpu.prof", "write cpu profile to `file`")

var memprofile = flag.String("memprofile", "mem.prof", "write memory profile to `file`")

func main() {

flag.Parse()

// CPU profile

if *cpuprofile != "" {

f, err := os.Create(*cpuprofile)

if err != nil {

log.Fatal("could not create CPU profile: ", err)

}

defer f.Close() // error handling omitted for example

if err := pprof.StartCPUProfile(f); err != nil {

log.Fatal("could not start CPU profile: ", err)

}

defer pprof.StopCPUProfile()

}

.................. other code

......................

// memory profile

if *memprofile != "" {

f, err := os.Create(*memprofile)

if err != nil {

log.Fatal("could not create memory profile: ", err)

}

defer f.Close() // error handling omitted for example

runtime.GC() // get up-to-date statistics

if err := pprof.WriteHeapProfile(f); err != nil {

log.Fatal("could not write memory profile: ", err)

}

}

}

Tracing

The GO has packages runtime/trace to capture the trace of the program execution. It can capture the trace of the individual blocks/goroutine of the code. Using the trace visualization tools, we can visualize and analyze the trace to pinpoint the issues. The trace command captures various execution events i.e. GC events, heap sizes, blocking/unblocking, events and goroutines details, etc.
Command to generate a trace for a test run.

$go test -trace=trace.out

To visualize the trace, we can use the pprof package's tool:

$go tool trace trace.out

After executing the trace visualize tool, we can see the trace in the browser:

Similar to writing profile reports in the program, we can start writing tracing in the program and capture all the events while executing the program for debugging purposes.

We can use the different annotations to trace different data they are:

log - capture the trace of execution logs
region - capture the time interval of the goroutines

task - capture the trace of the logical operations such as RPC, and HTTP requests.

Debugging

To debug a Go program, we need to set up the development environment with a Go code debugger. One of the common and popular Go debuggers is Delve. We can easily install Delve on Visual Studio Code as an extension.

Command to install Delv:

$ go install github.com/go-delve/delve/cmd/dlv@latest

We can use the command terminal to start and use the devl debugging, some commands are:

(dlv)$ start - start debugger

(dlv )$ break main.go : 35 - add the breakpoint in the main.go to file line no 35

(dlv )$ continue - continue the execution

(dlv)$ step - step through the code line

(dlv)$ next - go to the next line

(dlv)$ print variable name - print the value of the variable

(dlv)$ clear main.go:35   -  remove the breakpoint from the line no 35

After installing Delve in VS Code, we can execute the Go program in debug mode by selecting "Run" and then "Start Debugging". The program will start in debug mode, and we can navigate through the lines using the debugging mode panels.

Reference

https://go.dev/blog/pprof

https://go.dev/doc/diagnostics

https://pkg.go.dev/runtime/trace

https://go.dev/blog/execution-traces-2024

https://github.com/google/pprof/blob/main/doc/README.md

https://www.practical-go-lessons.com/chap-36-program-profiling