Structuring your Golang app: Flat structure vs. layered architecture

Building a great application begins with its structure. An app’s structure sets the tone for the development of the application, and it is important to get it right from the start.

Go is a relatively simple language that has no opinions on how applications should be structured. In this article, we’ll explore two primary ways you can structure your Go application.

Before we continue, it’s important to note that no one structure is perfect for all applications. Some of what we’ll cover may not be suited to your library or project. However, you should understand what’s available to use so you can easily decide on how to best build your application.

Building a Go app with a flat structure

This method of structuring projects keeps all the files and packages in the same directory.

Initially, this might look like a poor way of structuring projects, but some builds are perfectly suited for it. A sample project using a flat structure would have the following structure:

flat_app/
  main.go
  lib.go
  lib_test.go
  go.mod
  go.sum

The main advantage of using this structure is that it’s easy to work with. All created packages are located in the same directory, so they can be easily modified and used when required.

This structure is best used for building libraries, simple scripts, or simple CLI applications. HttpRouter, a widely used routing library for building APIs, uses a similar flat structure.

One major drawback, however, is that as the project becomes more complex, it will become almost impossible to maintain. For example, a structure like this would not be suited for building a REST API because the API has different components that make it function well, such as controllers, models, configs, and middleware. These components should not all be kept in one file directory.

Ideally, you should use a flat structure when bootstrapping an application. Once you become uncomfortable with the clutter, you can upgrade to any of the other project structures.

Building a simple API using a flat structure

To demonstrate a flat structure, let’s build an API for a note-taking application.

Create a new directory for this project by running:

mkdir notes_api_flat

The directory is named notes_api_flat because there could be variations of this app using other structures that we’ll cover later.

Now, initialize the project:

go mod init github.com/username/notes_api_flat

This application would allow users to store notes. We’ll use SQLite3 for storing the notes and Gin for routing. Run the snippet below to install them:

go get github.com/mattn/go-sqlite3
go get github.com/gin-gonic/gin

Next, create the following files:

• main.go: entry point to the application
• models.go: manages access to the database
• migration.go: manages creating tables

After creating them, the folder structure should look like this:

notes_api_flat/
  go.mod
  go.sum
  go.main.go
  migration.go
  models.go

Writing migration.go

Add the following to migration.go to create the table that will store our notes.

package main
import (
  "database/sql"
  "log"
)
const notes = `
  CREATE TABLE IF NOT EXISTS notes (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    title VARCHAR(64) NOT NULL,
    body MEDIUMTEXT NOT NULL,
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
  )
`
func migrate(dbDriver *sql.DB) {
  statement, err := dbDriver.Prepare(notes)
  if err == nil {
    _, creationError := statement.Exec()
    if creationError == nil {
      log.Println("Table created successfully")
    } else {
      log.Println(creationError.Error())
    }
  } else {
    log.Println(err.Error())
  }
}

In the above snippet, we are declaring the package to be main. Note that we cannot set it to be something different from what would be in main.go, as they are in the same directory. Hence, everything that is done in each file would be available globally because all of the files are located in the same package.

Notice that we imported the packages that would be required to interact with SQL, as well as the log package to log any errors that would occur.

Next, we have the SQL query that creates a notes table with the following fields: id, title, body, created_at, and updated_at.

Finally, we defined the function migrate, which executes the query that was written above and prints any errors that occur in the process.

Creating models.go

Add the following to models.go:

package main
import (
  "log"
  "time"
)
type Note struct {
  Id        int       `json:"id"`
  Title     string    `json:"title"`
  Body      string    `json:"body"`
  CreatedAt time.Time `json:"created_at"`
  UpdatedAt time.Time `json:"updated_at"`
}
func (note *Note) create(data NoteParams) (*Note, error) {
  var created_at = time.Now().UTC()
  var updated_at = time.Now().UTC()
  statement, _ := DB.Prepare("INSERT INTO notes (title, body, created_at, updated_at) VALUES (?, ?, ?, ?)")
  result, err := statement.Exec(data.Title, data.Body, created_at, updated_at)
  if err == nil {
    id, _ := result.LastInsertId()
    note.Id = int(id)
    note.Title = data.Title
    note.Body = data.Body
    note.CreatedAt = created_at
    note.UpdatedAt = updated_at
    return note, err
  }
  log.Println("Unable to create note", err.Error())
  return note, err
}
func (note *Note) getAll() ([]Note, error) {
  rows, err := DB.Query("SELECT * FROM notes")
  allNotes := []Note{}
  if err == nil {
    for rows.Next() {
      var currentNote Note
      rows.Scan(
        &currentNote.Id,
        &currentNote.Title,
        &currentNote.Body,
        &currentNote.CreatedAt,
        &currentNote.UpdatedAt)
      allNotes = append(allNotes, currentNote)
    }
    return allNotes, err
  }
  return allNotes, err
}
func (note *Note) Fetch(id string) (*Note, error) {
  err := DB.QueryRow(
    "SELECT id, title, body, created_at, updated_at FROM notes WHERE id=?", id).Scan(
    &note.Id, &note.Title, &note.Body, &note.CreatedAt, &note.UpdatedAt)
  return note, err
}

The model contains the note structure definition and the three methods that allow the note to interact with the database. The note structure contains all the data that a note can have and should be synchronized with the columns in the database.

The create method is responsible for creating a new note and returns the newly created note and any errors that occur in the process.

The getAll method gets all notes in the database as a slice and returns it with any errors that occur in the process.

The Fetch method gets a specific note from its id. All of these methods can be used in the future to get notes directly.

Completing the API in Go

The final piece remaining in the API is routing. Modify main.go to include the following code:

package main
import (
  "database/sql"
  "log"
  "net/http"
  "github.com/gin-gonic/gin"
  _ "github.com/mattn/go-sqlite3"
)
// Create this to store instance to SQL
var DB *sql.DB
func main() {
  var err error
  DB, err = sql.Open("sqlite3", "./notesapi.db")
  if err != nil {
    log.Println("Driver creation failed", err.Error())
  } else {
    // Create all the tables
    migrate(DB)
    router := gin.Default()
    router.GET("/notes", getAllNotes)
    router.POST("/notes", createNewNote)
    router.GET("/notes/:note_id", getSingleNote)
    router.Run(":8000")
  }
}
type NoteParams struct {
  Title string `json:"title"`
  Body  string `json:"body"`
}
func createNewNote(c *gin.Context) {
  var params NoteParams
  var note Note
  err := c.BindJSON(&params)
  if err == nil {
    _, creationError := note.create(params)
    if creationError == nil {
      c.JSON(http.StatusCreated, gin.H{
        "message": "Note created successfully",
        "note":    note,
      })
    } else {
      c.String(http.StatusInternalServerError, creationError.Error())
    }
  } else {
    c.String(http.StatusInternalServerError, err.Error())
  }
}
func getAllNotes(c *gin.Context) {
  var note Note
  notes, err := note.getAll()
  if err == nil {
    c.JSON(http.StatusOK, gin.H{
      "message": "All Notes",
      "notes":   notes,
    })
  } else {
    c.String(http.StatusInternalServerError, err.Error())
  }
}
func getSingleNote(c *gin.Context) {
  var note Note
  id := c.Param("note_id")
  _, err := note.Fetch(id)
  if err == nil {
    c.JSON(http.StatusOK, gin.H{
      "message": "Single Note",
      "note":    note,
    })
  } else {
    c.String(http.StatusInternalServerError, err.Error())
  }
}

Here, we import all the required packages. Note the final import:

"github.com/mattn/go-sqlite3"

This code snippet is required to work with SQLite, although it’s not being used directly. The main function initializes the database first, then exits if it fails to do so. The database instance is stored on the DB global variable so that it can easily be accessed.

Next, we migrate the tables by calling the migrate function, which was defined in migrations.go.

We do not need to import anything to use this function because it is in the main package and available globally.

Next, define the routes. We only need three routes:

• A GET request to /notes that retrieves all the notes that have been created and stored in the database
• A POST request to /notes creates a new note and persists it to the database
• A GET request to /note/:note_id retrieves a note by its id

These routes have individual handlers that use the note model to perform the required database actions.

Benefits of using a flat structure

We can see that by using the flat structure, we can build simple APIs quickly without managing multiple packages. This is especially useful for library authors because most modules are required to be a part of the base package.

Cons of using a flat structure

Despite all the benefits of using a flat structure, it’s not the best option when it comes to building APIs. First, this structure is quite limiting, and it automatically makes functions and variables available globally.

There is also no true separation of concerns. We tried to separate the model from the migration and the routing, but it was almost impossible because they can still be accessed directly from one another. This may cause one file to modify an item that it wasn’t supposed to or without the knowledge of another file, so this app would not be easily maintainable.

The next structure we’ll cover address many of the problems with using a flat structure.

Using a layered architecture (classic MVC structure) in Go

This structure groups files according to their functionalities. Packages that handle communication with the database (models) are grouped and stored differently from packages that handle the requests from the routes.

Let’s see what a layered architecture structure looks like:

layered_app/
  app/
    models/
      User.go         
    controllers/
      UserController.go
  config/
    app.go
  views/
    index.html
  public/
    images/
      logo.png
  main.go
  go.mod
  go.sum

Notice the separation. Because of it, it’s easy to maintain projects that are structured this way, and you’ll have less clutter in your code using an MVC structure.

Although layered architecture is not ideal for building simple libraries, it’s well suited for building APIs and other large applications. This is often the default structure for apps built using Revel, a popular Go Framework for building REST APIs.

Updating the Go app with a layered architecture

Now that you’ve seen an example project using layered architecture, let’s upgrade our project from a flat structure to an MVC structure.

Create a new folder called notes_api_layered and initialize a Go module in it by running the snippet below:

mkdir notes_api_layered
go mod init github.com/username/notes_api_layered

Install the required SQLite and Gin packages.

go get github.com/mattn/go-sqlite3
go get github.com/gin-gonic/gin

Now, update the project folder structure to look like this:

notes_api_layered/
  config/
    db.go
  controllers/
    note.go
  migrations/
    main.go
    note.go
  models/
    note.go
  go.mod
  go.sum
  main.go

As you can see from the new folder structure, all of the files have been arranged based on their functionalities. All models are located in the model’s directory, and the same goes for migrations, controllers, and configurations.

Next, we refactor the work we did in the flat structure implementation into this new structure.

Starting with the database configuration, add the following to config/db.go:

package config
import (
  "database/sql"
  _ "github.com/mattn/go-sqlite3"
)
var DB *sql.DB
func InitializeDB() (*sql.DB, error) {
  // Initialize connection to the database
  var err error
  DB, err = sql.Open("sqlite3", "./notesapi.db")
  return DB, err
}

Here, we’re declaring a package named config and importing all the relevant libraries to enable communication with the database. Note that we can declare multiple packages because they are not all in the same directory.

Next, we create a DB variable that will hold the connection to the database, as it’s not ideal for each model to have different instances of the database. Note: starting a variable name or function name with capital letters means they should be exported.

Then we declare and export an InitializeDB function, which opens the database and stores the database reference in the DB variable.

Once we are done with the database setup, we next work on the migrations. We have two files in the migrations folder: main.go and note.go.

main.go handles loading the queries to be performed, then performing them, while note.go contains SQL queries specific to the notes table.

If we were to have other tables, e.g., one for comments, they would also have a migration file that would contain the query to create the comments table.

Now, add the following to migrations/note.go:

package migrations
const Notes = `
CREATE TABLE IF NOT EXISTS notes (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  title VARCHAR(64) NOT NULL,
  body MEDIUMTEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL
)
`

Update migrations/main.go to include:

package migrations
import (
  "database/sql"
  "log"
  "github.com/username/notes_api_layered/config"
)
func Run() {
  // Migrate notes
  migrate(config.DB, Notes)
  // Other migrations can be added here.
}
func migrate(dbDriver *sql.DB, query string) {
  statement, err := dbDriver.Prepare(query)
  if err == nil {
    _, creationError := statement.Exec()
    if creationError == nil {
      log.Println("Table created successfully")
    } else {
      log.Println(creationError.Error())
    }
  } else {
    log.Println(err.Error())
  }
}

As explained earlier, migrations/main.go handles loading the query from the individual migration files and running it when the Run method gets called. migrate is a private function and cannot be used outside this module. The only function exported to the external world is Run.

After running the migrations, we need to update the models. The change between the layered structure implementation and the flat structure implementation here is pretty small.

All methods to be used externally should be exported, and all references to DB should be changed to config.DB.

After applying these changes, models/note.go should look like this:

package models
import (
  "log"
  "time"
  "github.com/username/notes_api_layered/config"
)
type Note struct {
  Id        int       `json:"id"`
  Title     string    `json:"title"`
  Body      string    `json:"body"`
  CreatedAt time.Time `json:"created_at"`
  UpdatedAt time.Time `json:"updated_at"`
}
type NoteParams struct {
  Title string
  Body  string
}
func (note *Note) Create(data NoteParams) (*Note, error) {
  var created_at = time.Now().UTC()
  var updated_at = time.Now().UTC()
  statement, _ := config.DB.Prepare("INSERT INTO notes (title, body, created_at, updated_at) VALUES (?, ?, ?, ?)")
  result, err := statement.Exec(data.Title, data.Body, created_at, updated_at)
  if err == nil {
    id, _ := result.LastInsertId()
    note.Id = int(id)
    note.Title = data.Title
    note.Body = data.Body
    note.CreatedAt = created_at
    note.UpdatedAt = updated_at
    return note, err
  }
  log.Println("Unable to create note", err.Error())
  return note, err
}
func (note *Note) GetAll() ([]Note, error) {
  rows, err := config.DB.Query("SELECT * FROM notes")
  allNotes := []Note{}
  if err == nil {
    for rows.Next() {
      var currentNote Note
      rows.Scan(
        &currentNote.Id,
        &currentNote.Title,
        &currentNote.Body,
        &currentNote.CreatedAt,
        &currentNote.UpdatedAt)
      allNotes = append(allNotes, currentNote)
    }
    return allNotes, err
  }
  return allNotes, err
}
func (note *Note) Fetch(id string) (*Note, error) {
  err := config.DB.QueryRow(
    "SELECT id, title, body, created_at, updated_at FROM notes WHERE id=?", id).Scan(
    &note.Id, &note.Title, &note.Body, &note.CreatedAt, &note.UpdatedAt)
  return note, err
}

We’ve declared a new package, models, and we imported the config from github.com/username/notes_api_layered/config. With that, we have access to the DB that would have been initialized once the InitializeDB function gets called.

Changes to the controller are pretty small, too, and mostly consist of exporting the functions and importing the model from the newly created model.

Change this code snippet:

var note Note
var params NoteParams

To this one:

var note models.Note
var params models.NoteParams

After this modification, the controller will look like this:

package controllers
import (
  "net/http"
  "github.com/gin-gonic/gin"
  "github.com/username/notes_api_layered/models"
)
type NoteController struct{}
func (_ *NoteController) CreateNewNote(c *gin.Context) {
  var params models.NoteParams
  var note models.Note
  err := c.BindJSON(&params)
  if err == nil {
    _, creationError := note.Create(params)
    if creationError == nil {
      c.JSON(http.StatusCreated, gin.H{
        "message": "Note created successfully",
        "note":    note,
      })
    } else {
      c.String(http.StatusInternalServerError, creationError.Error())
    }
  } else {
    c.String(http.StatusInternalServerError, err.Error())
  }
}
func (_ *NoteController) GetAllNotes(c *gin.Context) {
  var note models.Note
  notes, err := note.GetAll()
  if err == nil {
    c.JSON(http.StatusOK, gin.H{
      "message": "All Notes",
      "notes":   notes,
    })
  } else {
    c.String(http.StatusInternalServerError, err.Error())
  }
}
func (_ *NoteController) GetSingleNote(c *gin.Context) {
  var note models.Note
  id := c.Param("note_id")
  _, err := note.Fetch(id)
  if err == nil {
    c.JSON(http.StatusOK, gin.H{
      "message": "Single Note",
      "note":    note,
    })
  } else {
    c.String(http.StatusInternalServerError, err.Error())
  }
}

From the above snippet, we converted the functions to methods so that they can be accessed via NoteController.Create instead of controller.Create. This is to account for the possibility of having multiple controllers, which would be the case for most modern applications.

Finally, we update main.go to reflect the refactoring:

package main
import (
  "log"
  "github.com/gin-gonic/gin"
  "github.com/username/notes_api_layered/config"
  "github.com/username/notes_api_layered/controllers"
  "github.com/username/notes_api_layered/migrations"
)
func main() {
  _, err := config.InitializeDB()
  if err != nil {
    log.Println("Driver creation failed", err.Error())
  } else {
    // Run all migrations
    migrations.Run()

    router := gin.Default()

    var noteController controllers.NoteController
    router.GET("/notes", noteController.GetAllNotes)
    router.POST("/notes", noteController.CreateNewNote)
    router.GET("/notes/:note_id", noteController.GetSingleNote)
    router.Run(":8000")
  }
}

Following refactoring, we have a lean main package that imports the required packages: config, controllers, and models. Then, it initializes the database by calling config.InitializeDB().

Now we can move on to routing. The routes should be updated to use the newly created note controller for handling requests.

Benefits of using the layered structure in Go

The largest perk to using a layered structure is that right from the directory structure, you can understand what each file and/or folder is doing. There is also a clear separation of concerns, as each package has a single function to perform.

With a layered architecture, this project is easily extensible. For example, if a new feature to allow users to comment on notes gets added, it will be easy to implement because all of the groundwork has been done. In that case, the model, migration, and controller would just need to be created, then the routes updated, and viola! The feature has been added.

Drawbacks of using the layered structure

For simple projects, this architecture might be overkill and it requires a great deal of planning before implementing it.

Conclusion

In conclusion, we’ve seen that choosing a structure for your Go application depends on what you are building, how complex the project is, and how long you intend to work on it.

For creating simple projects, using a flat structure is just fine. When the project is more complicated, though, it’s important to take a step back to rethink the application and choose a better-suited structure for the application.

Additional structures that are popular to use when building Go apps are domain-driven development and the hexagonal architecture. It might be worth learning about those as well if your application continues to scale.