Arief's Blog

Hosting Custom Go Packages: A Deep Dive

Table of Contents

Introduction

When you want to distribute Go packages, the standard approach is to host them on platforms like GitHub, GitLab, or Bitbucket. Alongside that, Go also provides a powerful mechanism that allows you to host packages under your own domain while keeping the actual code in any version control system. This approach, known as ✨vanity import paths✨, gives you control over your package’s import path and branding.

For example, instead of importing a package as github.com/ariefrahmansyah/pkg, I can use ariefrahmansyah.com/pkg. This provides flexibility where I can change the underlying repository location without breaking existing code that imports my package.

In this deep dive, we’ll explore how Go’s import path discovery mechanism works, how to implement custom package hosting, and best practices for maintaining your own Go package domain.

How Go Discovers Packages

When you run go get ariefrahmansyah.com/pkg, Go doesn’t immediately know where to find the code. Instead, it follows a discovery process that involves querying the domain and looking for specific HTML meta tags.

The implementation of this discovery mechanism is part of the Go toolchain itself and you can find the relevant code in the Go source repository:

This mechanism is built into the go get command, so no additional configuration is needed on the client side where Go automatically handles the discovery process.

The Discovery Flow

Here’s what happens when Go encounters a custom import path:

sequenceDiagram
    participant Developer
    participant GoTool as go get
    participant DomainServer
    participant VCS as Version Control<br/>(GitHub/GitLab/etc)

    Developer->>GoTool: go get example.com/pkg
    GoTool->>DomainServer: HTTP GET https://example.com/pkg?go-get=1
    DomainServer-->>GoTool: HTML with go-import meta tag
    GoTool->>GoTool: Parse meta tag<br/>extract VCS type and repo URL
    GoTool->>VCS: Clone repository from extracted URL
    VCS-->>GoTool: Package source code
    GoTool-->>Developer: Package installed

The key insight is that Go makes an HTTP request to the domain with a special query parameter ?go-get=1. The server must respond with an HTML page containing specific meta tags that tell Go where to find the actual source code.

The go-import Meta Tag

The go-import meta tag is the heart of custom package hosting. It has the following format:

<meta name="go-import" content="import-prefix vcs repo-root" />

The content attribute contains three space-separated parts:

  1. Import Path Prefix: The prefix of the import path (e.g., ariefrahmansyah.com/pkg)
  2. Version Control System: The VCS type (git, svn, hg, or bzr)
  3. Repository Root URL: The URL where the repository can be cloned

For example:

<meta
  name="go-import"
  content="ariefrahmansyah.com/pkg git https://github.com/ariefrahmansyah/pkg"
/>

This tells Go that:

The go-source Meta Tag

While go-import is required for package discovery, go-source is optional but highly recommended. It provides links to view the source code, directory listings, and individual files. This enhances the developer experience when browsing your package on pkg.go.dev.

The format is:

<meta name="go-source" content="import-prefix home directory file" />

The content attribute contains four space-separated parts:

  1. Import Path Prefix: Same as in go-import
  2. Home URL: Link to the project’s home page
  3. Directory URL Template: Template for viewing directories (supports {/dir} placeholder)
  4. File URL Template: Template for viewing files (supports {/dir}, {file}, and {line} placeholders)

For example:

<meta
  name="go-source"
  content="ariefrahmansyah.com/pkg https://github.com/ariefrahmansyah/pkg https://github.com/ariefrahmansyah/pkg/tree/main{/dir} https://github.com/ariefrahmansyah/pkg/blob/main{/dir}/{file}#L{line}"
/>

This enables:

Implementation Tutorial

Now that we understand how it works, let’s implement custom package hosting for your domain.

Prerequisites

Before you begin, ensure you have:

  1. Domain ownership: You must control the domain you want to use
  2. Web server or static site: You need a way to serve HTML at your domain
  3. Version control repository: Your Go package code hosted somewhere (GitHub, GitLab, etc.)

Step 1: Create the HTML Page with Meta Tags

The simplest approach is to create an HTML page that serves the required meta tags. If you’re using a static site generator like Astro (as in this site), you can create a dedicated page.

Looking at the implementation in this site, I have a GoLayout component that handles the meta tags:

// src/layouts/GoLayout.astro
---
import BaseHead from "@/components/BaseHead.astro";

import "@/styles/global.css";

const { title, description, tags, goImport, goSource } = Astro.props;
---

<!doctype html>
<html lang="en">
  <head>
    <BaseHead title={title} description={description} tags={tags} />
    <meta name="go-import" content={goImport} />
    <meta name="go-source" content={goSource} />
  </head>
  <body>
    <slot />
  </body>
</html>

And a page that uses this layout where I can set the meta tags:

// src/pages/pkg.astro
---
import GoLayout from "@/layouts/GoLayout.astro";
---

<GoLayout
  title="ariefrahmansyah.com/pkg"
  description="ariefrahmansyah.com/pkg"
  goImport="ariefrahmansyah.com/pkg git https://github.com/ariefrahmansyah/pkg"
  goSource="ariefrahmansyah.com/pkg https://github.com/ariefrahmansyah/pkg https://github.com/ariefrahmansyah/pkg/tree/main{/dir} https://github.com/ariefrahmansyah/pkg/blob/main{/dir}/{file}#L{line}"
>
  <p>ariefrahmansyah.com/pkg</p>
</GoLayout>

Key Points:

Step 2: Alternative Implementation Methods

If you’re not using a static site generator, here another approach:

Using a Simple HTML File

Create a static HTML file and serve it:

<!DOCTYPE html>
<html>
  <head>
    <meta
      name="go-import"
      content="example.com/pkg git https://github.com/username/pkg"
    />
    <meta
      name="go-source"
      content="example.com/pkg https://github.com/username/pkg https://github.com/username/pkg/tree/main{/dir} https://github.com/username/pkg/blob/main{/dir}/{file}#L{line}"
    />
    <meta
      http-equiv="refresh"
      content="0; url=https://github.com/username/pkg"
    />
  </head>
  <body>
    <p>
      Redirecting to
      <a href="https://github.com/username/pkg">package repository</a>...
    </p>
  </body>
</html>

Using Nginx

If you’re using Nginx, you can serve the meta tags directly:

location /pkg {
    add_header Content-Type text/html;
    return 200 '<html><head><meta name="go-import" content="example.com/pkg git https://github.com/username/pkg"></head></html>';
}

Using a Go Server

You can also create a simple Go HTTP server:

package main

import (
    "fmt"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    html := `<!DOCTYPE html>
<html>
<head>
    <meta name="go-import" content="example.com/pkg git https://github.com/username/pkg">
    <meta name="go-source" content="example.com/pkg https://github.com/username/pkg https://github.com/username/pkg/tree/main{/dir} https://github.com/username/pkg/blob/main{/dir}/{file}#L{line}">
</head>
<body>
    <p>example.com/pkg</p>
</body>
</html>`
    w.Header().Set("Content-Type", "text/html")
    fmt.Fprint(w, html)
}

func main() {
    http.HandleFunc("/pkg", handler)
    http.ListenAndServe(":80", nil)
}

Step 3: Testing the Setup

After deploying your page, test the configuration:

# Test that the meta tags are served correctly
curl "https://ariefrahmansyah.com/pkg?go-get=1" | grep go-import

# Test that Go can discover and download the package
go get ariefrahmansyah.com/pkg

# Verify the package is in your module cache
go list -m -versions ariefrahmansyah.com/pkg

If everything is configured correctly, go get should successfully download your package.

Step 4: Using the Package

Once set up, others can use your package just like any other Go package:

package main

import (
    "fmt"

    "ariefrahmansyah.com/pkg/cache"
)

func main() {
    c := cache.NewSimpleCache[string, string]()
    c.Set("key", "value")

    value, err := c.Get("key")
    if err != nil {
        log.Fatalf("Failed to get value: %v", err)
    }
    fmt.Println(value)
}

Advanced Topics

Versioning Strategies

Go modules support semantic versioning. When using custom import paths, versioning works the same way:

# Get a specific version
go get ariefrahmansyah.com/[email protected]

# Get the latest version
go get ariefrahmansyah.com/pkg@latest

# Update to latest
go get -u ariefrahmansyah.com/pkg

Your repository should follow semantic versioning with Git tags (e.g., v0.1.0, v1.0.0, v1.2.3).

Hosting Multiple Packages

You can host multiple packages under the same domain by creating separate pages/routes for each:

Each package needs its own page with the appropriate go-import meta tag.

Subpath Packages

You can also host packages in subpaths:

<!-- For example.com/utils/strings -->
<meta
  name="go-import"
  content="example.com/utils git https://github.com/username/utils"
/>

When someone imports example.com/utils/strings, Go will:

  1. Query example.com/utils?go-get=1
  2. Get the meta tag pointing to the repository
  3. Look for the strings subdirectory in that repository

Security Considerations

  1. HTTPS Required: Always use HTTPS. Go will warn users if they try to fetch packages over HTTP.

  2. Domain Control: Only set up custom import paths for domains you control. Never use someone else’s domain without permission.

  3. Repository Integrity: Go modules use checksums to verify package integrity. The Go checksum database (sum.golang.org) will automatically index your package once it’s publicly available.

  4. Redirects: Be careful with redirects. Go follows redirects, but complex redirect chains can cause issues.

Best Practices

  1. Always include go-source: This improves the developer experience on pkg.go.dev and makes your package more discoverable.

  2. Use semantic versioning: Tag your releases properly so users can pin to specific versions.

  3. Keep it simple: The HTML page doesn’t need to be fancy. A simple page with the meta tags is sufficient.

  4. Document your package: Write good Go documentation. It will appear on pkg.go.dev automatically.

  5. Test thoroughly: Test your setup with go get before announcing your package.

  6. Consider automation: For multiple packages, consider using tools like govanityurls to manage vanity URLs automatically.

Examples and Use Cases

Real-World Examples

Many popular Go projects use custom import paths:

When to Use Custom Hosting

Use custom hosting when:

Stick with standard hosting when:

Conclusion

Hosting custom Go packages on your own domain provides flexibility, branding control, and independence from specific hosting providers. The mechanism is elegant. It only requires a few HTML meta tags to enable Go’s tooling to discover and fetch your packages seamlessly.

The key takeaways:

Whether you’re building a personal library or a suite of enterprise packages, custom import paths give you the control you need while maintaining compatibility with Go’s ecosystem.

Further Reading

#go #golang #package-management #web