Hosting Custom Go Packages

Introduction

When you want to distribute Go packages, the standard approach is to host them on platforms like GitHub, GitLab, or Bitbucket. Alongside that, Go also provides a powerful mechanism that allows you to host packages under your own domain while keeping the actual code in any version control system. This approach, known as ✨vanity import paths✨, gives you control over your package's import path and branding.

For example, instead of importing a package as github.com/ariefrahmansyah/pkg, I can use ariefrahmansyah.com/pkg. This provides flexibility where I can change the underlying repository location without breaking existing code that imports my package.

In this deep dive, we'll explore how Go's import path discovery mechanism works, how to implement custom package hosting, and best practices for maintaining your own Go package domain.

How Go Discovers Packages

When you run go get ariefrahmansyah.com/pkg, Go doesn't immediately know where to find the code. Instead, it follows a discovery process that involves querying the domain and looking for specific HTML meta tags.

The implementation of this discovery mechanism is part of the Go toolchain itself and you can find the relevant code in the Go source repository:

• Import path resolution: cmd/go/internal/vcs - Contains the logic for parsing go-import and go-source meta tags
• HTTP fetching: cmd/go/internal/vcs/vcs.go - Handles HTTP requests with the ?go-get=1 query parameter
• VCS detection: The Go toolchain queries domains and parses HTML meta tags to determine the version control system and repository location

This mechanism is built into the go get command, so no additional configuration is needed on the client side where Go automatically handles the discovery process.

The Discovery Flow

Here's what happens when Go encounters a custom import path:

The key insight is that Go makes an HTTP request to the domain with a special query parameter ?go-get=1. The server must respond with an HTML page containing specific meta tags that tell Go where to find the actual source code.

The go-import Meta Tag

The go-import meta tag is the heart of custom package hosting. It has the following format:

<meta name="go-import" content="import-prefix vcs repo-root" />

The content attribute contains three space-separated parts:

1. Import Path Prefix: The prefix of the import path (e.g., ariefrahmansyah.com/pkg)
2. Version Control System: The VCS type (git, svn, hg, or bzr)
3. Repository Root URL: The URL where the repository can be cloned

For example:

<meta
  name="go-import"
  content="ariefrahmansyah.com/pkg git https://github.com/ariefrahmansyah/pkg"
/>

This tells Go that:

• The import path prefix is ariefrahmansyah.com/pkg
• The VCS is Git
• The repository is at https://github.com/ariefrahmansyah/pkg

The go-source Meta Tag

While go-import is required for package discovery, go-source is optional but highly recommended. It provides links to view the source code, directory listings, and individual files. This enhances the developer experience when browsing your package on pkg.go.dev.

The format is:

<meta name="go-source" content="import-prefix home directory file" />

The content attribute contains four space-separated parts:

1. Import Path Prefix: Same as in go-import
2. Home URL: Link to the project's home page
3. Directory URL Template: Template for viewing directories (supports {/dir} placeholder)
4. File URL Template: Template for viewing files (supports {/dir}, {file}, and {line} placeholders)

For example:

<meta
  name="go-source"
  content="ariefrahmansyah.com/pkg https://github.com/ariefrahmansyah/pkg https://github.com/ariefrahmansyah/pkg/tree/main{/dir} https://github.com/ariefrahmansyah/pkg/blob/main{/dir}/{file}#L{line}"
/>

This enables:

• Clickable links to source code on pkg.go.dev
• Direct navigation to specific files and lines
• Better integration with Go's documentation tools

Implementation Tutorial

Now that we understand how it works, let's implement custom package hosting for your domain.

Prerequisites

Before you begin, ensure you have:

1. Domain ownership: You must control the domain you want to use
2. Web server or static site: You need a way to serve HTML at your domain
3. Version control repository: Your Go package code hosted somewhere (GitHub, GitLab, etc.)

Step 1: Create the HTML Page with Meta Tags

The simplest approach is to create an HTML page that serves the required meta tags. If you're using a static site generator like Astro (as in this site), you can create a dedicated page.

Looking at the implementation in this site, I have a GoLayout component that handles the meta tags:

// src/layouts/GoLayout.astro
---
import BaseHead from "@/components/BaseHead.astro";
 
import "@/styles/global.css";
 
const { title, description, tags, goImport, goSource } = Astro.props;
---
 
<!doctype html>
<html lang="en">
  <head>
    <BaseHead title={title} description={description} tags={tags} />
    <meta name="go-import" content={goImport} />
    <meta name="go-source" content={goSource} />
  </head>
  <body>
    <slot />
  </body>
</html>

And a page that uses this layout where I can set the meta tags:

// src/pages/pkg.astro
---
import GoLayout from "@/layouts/GoLayout.astro";
---
 
<GoLayout
  title="ariefrahmansyah.com/pkg"
  description="ariefrahmansyah.com/pkg"
  goImport="ariefrahmansyah.com/pkg git https://github.com/ariefrahmansyah/pkg"
  goSource="ariefrahmansyah.com/pkg https://github.com/ariefrahmansyah/pkg https://github.com/ariefrahmansyah/pkg/tree/main{/dir} https://github.com/ariefrahmansyah/pkg/blob/main{/dir}/{file}#L{line}"
>
  <p>ariefrahmansyah.com/pkg</p>
</GoLayout>

Key Points:

• The page must be accessible at the exact import path. For ariefrahmansyah.com/pkg, the page should be at /pkg (or /pkg/ with redirect)
• Go will append ?go-get=1 to the URL, but your page should work without it
• The HTML doesn't need to be complex or pretty. It's only need the meta tags

Step 2: Alternative Implementation Methods

If you're not using a static site generator, here another approach:

Using a Simple HTML File

Create a static HTML file and serve it:

<!DOCTYPE html>
<html>
  <head>
    <meta
      name="go-import"
      content="example.com/pkg git https://github.com/username/pkg"
    />
    <meta
      name="go-source"
      content="example.com/pkg https://github.com/username/pkg https://github.com/username/pkg/tree/main{/dir} https://github.com/username/pkg/blob/main{/dir}/{file}#L{line}"
    />
    <meta
      http-equiv="refresh"
      content="0; url=https://github.com/username/pkg"
    />
  </head>
  <body>
    <p>
      Redirecting to
      <a href="https://github.com/username/pkg">package repository</a>...
    </p>
  </body>
</html>

Using Nginx

If you're using Nginx, you can serve the meta tags directly:

location /pkg {
    add_header Content-Type text/html;
    return 200 '<html><head><meta name="go-import" content="example.com/pkg git https://github.com/username/pkg"></head></html>';
}

Using a Go Server

You can also create a simple Go HTTP server:

package main
 
import (
    "fmt"
    "net/http"
)
 
func handler(w http.ResponseWriter, r *http.Request) {
    html := `<!DOCTYPE html>
<html>
<head>
    <meta name="go-import" content="example.com/pkg git https://github.com/username/pkg">
    <meta name="go-source" content="example.com/pkg https://github.com/username/pkg https://github.com/username/pkg/tree/main{/dir} https://github.com/username/pkg/blob/main{/dir}/{file}#L{line}">
</head>
<body>
    <p>example.com/pkg</p>
</body>
</html>`
    w.Header().Set("Content-Type", "text/html")
    fmt.Fprint(w, html)
}
 
func main() {
    http.HandleFunc("/pkg", handler)
    http.ListenAndServe(":80", nil)
}

Step 3: Testing the Setup

After deploying your page, test the configuration:

# Test that the meta tags are served correctly
curl "https://ariefrahmansyah.com/pkg?go-get=1" | grep go-import
 
# Test that Go can discover and download the package
go get ariefrahmansyah.com/pkg
 
# Verify the package is in your module cache
go list -m -versions ariefrahmansyah.com/pkg

If everything is configured correctly, go get should successfully download your package.

Step 4: Using the Package

Once set up, others can use your package just like any other Go package:

package main
 
import (
    "fmt"
 
    "ariefrahmansyah.com/pkg/cache"
)
 
func main() {
    c := cache.NewSimpleCache[string, string]()
    c.Set("key", "value")
 
    value, err := c.Get("key")
    if err != nil {
        log.Fatalf("Failed to get value: %v", err)
    }
    fmt.Println(value)
}

Advanced Topics

Versioning Strategies

Go modules support semantic versioning. When using custom import paths, versioning works the same way:

# Get a specific version
go get ariefrahmansyah.com/[email protected]
 
# Get the latest version
go get ariefrahmansyah.com/pkg@latest
 
# Update to latest
go get -u ariefrahmansyah.com/pkg

Your repository should follow semantic versioning with Git tags (e.g., v0.1.0, v1.0.0, v1.2.3).

Hosting Multiple Packages

You can host multiple packages under the same domain by creating separate pages/routes for each:

• example.com/pkg1 → Points to github.com/username/pkg1
• example.com/pkg2 → Points to github.com/username/pkg2
• example.com/utils/strings → Points to github.com/username/utils/strings

Each package needs its own page with the appropriate go-import meta tag.

Subpath Packages

You can also host packages in subpaths:

<!-- For example.com/utils/strings -->
<meta
  name="go-import"
  content="example.com/utils git https://github.com/username/utils"
/>

When someone imports example.com/utils/strings, Go will:

1. Query example.com/utils?go-get=1
2. Get the meta tag pointing to the repository
3. Look for the strings subdirectory in that repository

Security Considerations

1. HTTPS Required: Always use HTTPS. Go will warn users if they try to fetch packages over HTTP.

2. Domain Control: Only set up custom import paths for domains you control. Never use someone else's domain without permission.

3. Repository Integrity: Go modules use checksums to verify package integrity. The Go checksum database (sum.golang.org) will automatically index your package once it's publicly available.

4. Redirects: Be careful with redirects. Go follows redirects, but complex redirect chains can cause issues.

Best Practices

1. Always include go-source: This improves the developer experience on pkg.go.dev and makes your package more discoverable.

2. Use semantic versioning: Tag your releases properly so users can pin to specific versions.

3. Keep it simple: The HTML page doesn't need to be fancy. A simple page with the meta tags is sufficient.

4. Document your package: Write good Go documentation. It will appear on pkg.go.dev automatically.

5. Test thoroughly: Test your setup with go get before announcing your package.

6. Consider automation: For multiple packages, consider using tools like govanityurls to manage vanity URLs automatically.

Examples and Use Cases

Real-World Examples

Many popular Go projects use custom import paths:

• golang.org/x/tools → Points to go.googlesource.com/tools
• k8s.io/client-go → Points to github.com/kubernetes/client-go
• gopkg.in/yaml.v2 → Points to github.com/go-yaml/yaml

When to Use Custom Hosting

Use custom hosting when:

• You want consistent branding with your domain
• You might migrate between hosting providers
• You're building a suite of related packages
• You want a shorter, more memorable import path

Stick with standard hosting when:

• You're just starting out and want simplicity
• You don't own a domain
• You're publishing a one-off package
• Your team prefers the standard GitHub/GitLab URLs

Conclusion

Hosting custom Go packages on your own domain provides flexibility, branding control, and independence from specific hosting providers. The mechanism is elegant. It only requires a few HTML meta tags to enable Go's tooling to discover and fetch your packages seamlessly.

The key takeaways:

• Go uses HTTP requests with ?go-get=1 to discover packages
• The go-import meta tag tells Go where to find your repository
• The go-source meta tag enhances documentation and browsing
• Implementation is straightforward. You just need to serve HTML with the required meta tags
• Follow best practices for versioning, security, and documentation

Whether you're building a personal library or a suite of enterprise packages, custom import paths give you the control you need while maintaining compatibility with Go's ecosystem.

Further Reading

• Go Documentation: Remote Import Paths
• Go Modules Documentation
• pkg.go.dev - Browse and discover Go packages
• govanityurls - Tool for managing vanity URLs

This post is written/assisted by AI and reviewed by human. Read more about it here.