Executing Shell Commands in Golang

In this tutorial we will learn how to execute shell commands (like ls, mkdir or grep) in Golang.

We will also learn how to pass I/O to a running command through stdin and stdout, as well as manage long running commands.

If you just want to see the code, you can view it on Github

The Exec Package

We can use the official os/exec package to run external commands.

When we execute shell commands, we are running code outside of our Go application. In order to do this, we need to run these commands in a child process.

Each command is run as a child process within the running Go application, and exposes Stdin and Stdout attributes that we can use to read and write data from the process.

Running Basic Shell Commands

To run a simple command and read its output, we can create a new *exec.Cmd instance and run it.

In this example, let’s list the files in our current directory using ls, and print the output from our code:

// create a new *Cmd instance
// here we pass the command as the first argument and the arguments to pass to the command as the
// remaining arguments in the function
cmd := exec.Command("ls", "./")

// The `Output` method executes the command and
// collects the output, returning its value
out, err := cmd.Output()
if err != nil {
  // if there was any error, print it here
  fmt.Println("could not run command: ", err)
}
// otherwise, print the output from running the command
fmt.Println("Output: ", string(out))

Since I am running this code within the example repository, it prints the files in the project root:

> go run shellcommands/main.go

Output:  LICENSE
README.md
go.mod
shellcommands

Note that when we run exec, our application does not spawn a shell, and runs the given command directly. This means that any shell-based processing, like glob patterns or expansions will not be done.

So for example, when we run ls ./*.md , it is not expanded into README.md like we expect when running in our bash shell.

Executing Long Running Commands

The previous example executed the ls command that returned its output immediately. What about commands whose output is continuous, or takes a long time to retrieve?

For example, when we run the ping command, we get continuous output at periodic intervals:

➜  ~ ping google.com
PING google.com (142.250.77.110): 56 data bytes
64 bytes from 142.250.77.110: icmp_seq=0 ttl=116 time=11.397 ms
64 bytes from 142.250.77.110: icmp_seq=1 ttl=116 time=17.646 ms  ## this is received after 1 second
64 bytes from 142.250.77.110: icmp_seq=2 ttl=116 time=10.036 ms  ## this is received after 2 seconds
64 bytes from 142.250.77.110: icmp_seq=3 ttl=116 time=9.656 ms   ## and so on
# ...

If we tried executing this type of command using cmd.Output, we wouldn’t get any output, since the Output method waits for the command to execute, and the ping command executes indefinitely.

Instead, we can a custom Stdout attribute to read output continuously:

cmd := exec.Command("ping", "google.com")

// pipe the commands output to the applications
// standard output
cmd.Stdout = os.Stdout

// Run still runs the command and waits for completion
// but the output is instantly piped to Stdout
if err := cmd.Run(); err != nil {
  fmt.Println("could not run command: ", err)
}

Output:

> go run shellcommands/main.go

PING google.com (142.250.195.142): 56 data bytes
64 bytes from 142.250.195.142: icmp_seq=0 ttl=114 time=9.397 ms
64 bytes from 142.250.195.142: icmp_seq=1 ttl=114 time=37.398 ms
64 bytes from 142.250.195.142: icmp_seq=2 ttl=114 time=34.050 ms
64 bytes from 142.250.195.142: icmp_seq=3 ttl=114 time=33.272 ms

# ...
# and so on

By assigning the Stdout attribute directly, we can capture the output throughout the commands lifecycle, and process it as soon as it is received.

Custom Output Writer

Instead of using os.Stdout, we can create our own writer that implements the io.Writer interface.

Let’s create a writer that adds a "received output:" prefix before each output chunk:

type customOutput struct{}

func (c customOutput) Write(p []byte) (int, error) {
	fmt.Println("received output: ", string(p))
	return len(p), nil
}

Now we can assign a new instance of customWriter as the output writer:

cmd.Stdout = customOutput{}

If we run the application now, we will get the below output:

received output:  PING google.com (142.250.195.142): 56 data bytes
64 bytes from 142.250.195.142: icmp_seq=0 ttl=114 time=187.825 ms

received output:  64 bytes from 142.250.195.142: icmp_seq=1 ttl=114 time=19.489 ms

received output:  64 bytes from 142.250.195.142: icmp_seq=2 ttl=114 time=117.676 ms

received output:  64 bytes from 142.250.195.142: icmp_seq=3 ttl=114 time=57.780 ms

Passing Input To Commands With STDIN

In the previous examples, we executed commands without giving any input (or providing limited inputs as arguments). In most cases, input is given through the STDIN stream.

One popular example of this is the grep command, where we can pipe the input from another command:

➜  ~ echo "1. pear\n2. grapes\n3. apple\n4. banana\n" | grep apple
3. apple

Here, the input is passed to the grep command through STDIN. In this case the input is a list of fruit, and grep filters the line that contains "apple"

The *Cmd instance provides us with an input stream which we can write into. Let’s use it to pass input to a grep child process:

cmd := exec.Command("grep", "apple")

// Create a new pipe, which gives us a reader/writer pair
reader, writer := io.Pipe()
// assign the reader to Stdin for the command
cmd.Stdin = reader
// the output is printed to the console
cmd.Stdout = os.Stdout

go func() {
  defer writer.Close()
  // the writer is connected to the reader via the pipe
  // so all data written here is passed on to the commands
  // standard input
  writer.Write([]byte("1. pear\n"))
  writer.Write([]byte("2. grapes\n"))
  writer.Write([]byte("3. apple\n"))
  writer.Write([]byte("4. banana\n"))
}()

if err := cmd.Run(); err != nil {
  fmt.Println("could not run command: ", err)
}

Output:

3. apple

Killing a Child Process

There are several commands that run indefinitely, or need an explicit signal to stop.

For example, if we start a web server using python3 -m http.server or execute sleep 10000 the resulting child processes will run for a very long time (or indefinitely).

To stop these processes, we need to send a kill signal from our application. We can do this by adding a context instance to the command.

If the context gets cancelled, the command terminates as well.

ctx := context.Background()
// The context now times out after 1 second
// alternately, we can call `cancel()` to terminate immediately
ctx, cancel = context.WithTimeout(ctx, 1*time.Second)

cmd := exec.CommandContext(ctx, "sleep", "100")

out, err := cmd.Output()
if err != nil {
  fmt.Println("could not run command: ", err)
}
fmt.Println("Output: ", string(out))

This will give the following output after 1 second has elapsed:

could not run command:  signal: killed
Output:  

Terminating child processes is useful when you want to limit the time spent in running a command or want to create a fallback incase a command doesn’t return a result on time.

Conclusion

So far, we learned multiple ways to execute and interact with unix shell commands. Here are some things to keep in mind when using the os/exec package:

• Use cmd.Output when you want to execute simple commands that don’t usually give too much output
• For functions with continuous or long-running output, you should use cmd.Run and interact with the command using cmd.Stdout and cmd.Stdin
• In production applications, its useful to keep a timeout and kill a process if it isn’t responding for a given time. We can send termination commands using context cancellation.

If you want to read more about the different functions and configuration options, you can view the official documentation page.

You can view the working code for all examples on Github.