Safigo

Social networks for non-social posting in 2026

Anton Ohorodnyk — Thu, 27 Aug 2026 02:17:00 GMT

Open Table of contents

Goal

I don’t like social networks, and I’ve never used them in my life. However, I have some need to post my thoughts and contact with people.

Since Software Development, and Cybersecurity, and databases are my the main interests, it limits the selection circle.

I want to write some thoughts, speak with other people and avoid AI generated content, marketing, and clickbait.

Social networks in 2026

While the obvious choice would be X for developers, the amount of AI generated content, marketing, and clickbait is overwhelming.

Also, I understand the idea of paying subscriptions, but not in a pay-to-win model. You pay, and you get a better reach, identical to Pay-To-Win in games, that is not for me.

Amount of advertisements is also overwhelming, I’d say it’s much more than I’d tolerate.

Am I ready to pay? Maybe. But the price should be reasonable and with no pay-to-win model.

Not options for me

Instagram, Facebook, TikTok, Threads, and similar platforms are not for me.

The reasons are mine, subjective, and personal:

I don’t want to have a doom-scrolling across unlimited short videos, memes, and AI generated content.
Similar point for overwhelming amount of images
I want to have some place where technical people communicate, ideally not “influencers” and “content creators”, but people who are interested in the same topics as I am.

Mastodon

I’ve tried Mastodon a couple of times, it’s really nice. It gives reasonably good experience, but there are some catches.

Fediverse is not ideally implemented
- History of posts from other instances is not available, I need to go to another instance to see them
- To be visible on other instances, I need to follow them, and they need to follow me back
- Sometimes caching makes posts inconsistent between instances
- Instances may have different configurations that leads to different user experience, and inconsistent experience across instances
Clients are much laggier that for other services, web interface is only reliable option
Experience of reading posts is different between clients and web interface

After experiencing significant amount of issues and limitations for searching the right instance, it did not feel comfortable for me.

And of the most interesting limitations I couldn’t explain, is that Mastodon claims to be a decentralized social network, but I did not own my own profile. If I decide to switch to another instance, I will lose my ID.

It means, I don’t choose a decentralized instance, I choose a centralized instance that is part of a decentralized network.

It could be simply fixed by supporting DNS records for user profiles, and assigning an “email-like” ID to each user. Users would have a chance to migrate between instances without losing their ID.

Bluesky

Bluesky looks promising, and an initial idea was nice, but the reaching part is tricky. After X and Mastodon in feels silent, maybe not entirely, but not as warm as Mastodon.

And tricky, not clear at first, Feeds feature and decentralization.

Anyway, I chose to use Bluesky, and I will see how it goes. I hope to see you there 👋

Updates

Update 2026-08-31

I have tried to use BlueSky for a couple of days. It was surprising how concentrated it was with politics and weirdly enough, screenshots from X.

Since the most of the people use X, it looks like BlueSky contains reposts from X.

I’ll try to use X in parallel with BlueSky, and give a chance to win only by one of them.

UX and ML-Driven Functionality in Modern Music Streaming Services

Anton Ohorodnyk — Tue, 02 Jun 2026 03:15:08 GMT

Open Table of contents

A Mental Shift

I’ve never really been into social networks. It’s not that I was inherently against them from day one, it was more about my lack of personal involvement.

I remember my the first social network, I used it primarily as a chat program. It was simply a place where we could send text messages because everyone around us was there. Sharing things publicly, opinions, photos, jokes, just wasn’t my thing. Since I was never deeply affected by social networks, I remained largely untouched when they began impacting our social lives too heavily and the digital dangers became real.

Addiction to YouTube

However, one platform did manage to make me dependent, and even addicted: YouTube.

When Apple shipped the Screen Time feature, it became clear that I was spending far too much time with YouTube active on my phone. I was constantly either:

Listening to videos in the background
Watching videos actively

It became a non-stop process. Whether it was cognitive overload or something else, I realized I could mindlessly scroll through YouTube for minutes without actually selecting a video to watch.

When YouTube released the “Shorts” feature, it introduced a new problem. Now, I wasn’t just scrolling warching long videos, I was literally spending hours watching Shorts. That was the moment I truly understood how TikTok affects people. Lately, YouTube has become less interesting to me (mostly due to broken recommendations), but the underlying habit remains an issue.

Switching to Music

I’m not entirely sure how sound this decision is, but I thought I would try moving away from addictive videos and transition to music.

Due to my “unusual” personality I had almost never listened to music in my life. That left me in a situation where I had almsot no preferences. It doesn’t mean I can’t distinguish what I like from what I don’t, it just means I don’t know what exactly I’d like. I needed use “recommendations” to build my own list.

It meant I need to choose a streaming service where I can feel comfortable.

Choosing a Streaming Service

it was obvious, I needed a service where I could choose a few initial songs and artists, and use it’s recommendations to build a library.

In my mind, I had only one core requirement: I wanted a “magic playlist” that would combine my randomly shuffled saved music (songs I enjoy and have liked/favorited) with a shuffled in recommendations. To me, this felt like the best possible user experience to get started. At least during my the first few months.

YouTube Music

YouTube Music was the most obvious solution, and I almost liked it. However, I cannot understand who made the genius decision to limit it “wearable” client to work ONLY with connected phone.

When I ride a bike or go to the gym, I don’t want to bring my phone with me. I prefer to connect my smartwatch directly to my headphones and listen music completely without phone. It’s an incredibly comfortable setup.

Because of this limitation, YouTube Music was removed from my list automatically, without any second chances.

Amazon Music

I don’t remember exactly why or how, but Amazon Music offered me a 3-month free trial right around this time. I couldn’t resist taking it for a try.

Amazon Music works reasonably well on a smartwatch… almost. In general, it works, but it’s quality is similar across all Amazon’s applications: the UI feels unresponsive and laggy.

While Amazon Music has made steps forward, it only works well, until it doesn’t. Sometimes, when skipping to the next track or going back to a previous one, the synchronization between album art, title, and audio breaks entirely. It can display the image from one track, the title from another, and play the audio from a third.

The most frustrating part? When it break on my smartwatch during a ride.

Apple Music

Since I am deeply invested into Apple’s ecosystem, it made sense to try Apple Music, expecting a highly consistent experience. While for Linux, it provides a functional Web UI.

But after spending some time with it, I realized their recommendation engine simply doesn’t work for me. I read all their playlists are “hand-picked.”, but those curated selections never quite aligned with my taste.

Apple Music features a personalized station called “User’s Station,” but whenever I put it on, I wanted to skip every single song. The tracks felt completely disconnected from what I’d expect.

Moreover, one interesting feature that frustrated me, I did not expect from Apple.. When I was playing music via the Web UI and tried to switch to my mobile phone, instead of seamlessly moving the active playback between devices, it showed me an error that I need a “Family” subscription to listen on multiple devices. I haven’t similar issues on other services.

Spotify

And here we are!

Everyone recommends Spotify for its legendary recommendation algorithm. I expected this algorithmic magic to make me happy right from the start.

To be fair, Spotify boasts the highest UI quality. It just works. From my perspective, there are only two streaming services consistently building top-tier user interfaces: Netflix and Spotify.

Regardless, I started the app, completed an onboarding experience, and started listening. To my surprise, I couldn’t find an obvious “User’s Station” equivalent on the home screen like Apple Music had. It features many automated playlists, but all of them were standard Artist and Similar radios.

It wasn’t quite what I was hunting for.

After many attempts and our lovely chat bots, who help us search in internet I’ve found a hidden feature: Smart Shuffle inside the Liked Songs playlist. This was exactly what I had been searching for.

Over the last few days, I’ve quite enjoyed Spotify, and I plan to stick with it for now.

Closing Thoughts

I understand that my specific use case might be an outlier, but it feels clear to me that Discovery and Shuffled In¹ playlists should be on the main screen.

Do most people really listen to music by looking for recommendations based on an artist or a song?

From a good UX, I’d expect those playlists on the main screen:

Discovery: A dedicated space to find new suggestions based on the current intests.
Shuffled In¹: The hybrid mix of known favorites and new algorithmic suggestions.
Mood-Based Playlists
Time-and-Activity Categories (e.g., morning focus, workout, evening wind-down)

I’d love to hear from others on this. How disconnected are my UX expectations from an average streaming user?

Footnotes

A dynamic playlist with a randomly shuffled mix of my liked/favorite songs with randome suggestions of new, recommended tracks. ↩ ↩²

Phishing through referral links: A case study with DigitalOcean

Anton Ohorodnyk — Sun, 17 May 2026 05:58:32 GMT

Open Table of contents

Rise of phishing through “official” emails

There used to be nice times when we received emails from randomemail@gmail.com with an impersonation to some service or person. It was simple to filter out those emails, but just looking at the sender’s email address.

Not anymore. Recent years there was a switch to utilizing ways to send emails with custom text from an official service.

The most popular example of emails:

Referral links
Invitation to a community/group

Since all of those emails put some sender’s information (sometimes with a custom text), that utilized to be used for phishing emails.

Case study: DigitalOcean referral links

Recently, I received an email from DigitalOcean containing a referral invitation. It looked like this:

Key parts of the email were:

Subject: Join the ‘Your Bank account will be charged 499 USD_Call 656-556-2282 if unauthorized’ team on DigitalOcean
Sender’s name: DigitalOcean via Your Bank account will be charged 499 USD_Call 656-556-2282 if unauthorized
Email address: no-reply@referrals.digitalocean.com
Custom text from body: Your bank/PayPal account will be charged $499.99 for your Norton-360 Deluxe subscription. Please call +1 (656) 556 - 2282 immediately if unauthorized; otherwise, the charge will process in 12 hours. Invoice ID: N7R47 (arjunb.hamre.96@gmail.com)

Even though the email does not look consistent, by inviting to join a team, and only CTA is to Accept Invitation, text about a significant charge and direct communication methods may confuse users and make them to call/write to the scammers.

Reminder

While we used to rely on “technical” indicators of phishing emails, such as the sender’s email address, it is now equally important to inspect the content and intent of the message.

Many companies provide anti-phishing training and checklists. Use them regularly, and pause before acting on any urgent or emotional request.

What can happen if you engage

Let’s assume someone falls into the trap. What could happen next?

At least three scenarios are likely.

Clicking “Accept Invitation” link

The link may lead to accepting an empty project. This DigitalOcean org/project does not provide benefits, but it confirms the email is active and that you engage with messages. This may lead to more phishing emails in the future.

It may also expose additional account context that helps attackers research you using OSINT techniques, making future phishing attempts more personalized.

Also, they have a bond with you, since you are registered in DigitalOcean and the future emails will be more personalized, which may increase the chances of falling for the scam. More targeted attacks.

Replying to the email

Replying to the email will lead to a slow communication with scammers. They won’t try to scam in the first message, but they will try to build trust, scare you, and make you to make required actions.

Email is dangerous not only because scammers can request to pay something or steal data, but also because it can become a channel for technical attacks (malicious attachments, credential-harvesting links, and similar payloads).

Calling the phone number

Calling is often the most dangerous step, because scammers can pressure you in real time. During the call, it’s very hard to take a break and process the information.

They want immediate action without giving you time to think or verify. Typical scare tactics include fake charges, account lockouts, or threats about stolen personal information.

Reporting this to DigitalOcean

I tried to report this email to DigitalOcean, but they made it harder than it should be.

It’s not as simple as forwarding the email to abuse@digitalocean.com, but you need to fill out a form with various information about the email. The form currently has no way to attach the original email, PDF, or screenshot.

Windows after macOS

Anton Ohorodnyk — Fri, 20 Mar 2026 04:36:36 GMT

Open Table of contents

My Desktop OS Journey

The Early Days

I started my desktop OS journey with Windows 95, when my father brought home an old laptop. It wasn’t stable or performant enough to render cartoons using Flash, but it was the first device I could use.

I was a kid, fascinated by the idea of a computer, and I wanted to learn how to use it. I spent hours experimenting and reading about different software and capabilities of the OS.

The Floppy Disk Hack

At that time, we used floppy disks, and they could not fit a game on 1.44 MB. As a kid, I had a “brilliant” idea to go to my friend’s house and copy shortcuts (symlinks) from his computer. It felt like a way to hack the system by copying all of his games onto a single floppy disk.

It was a huge disappointment when it didn’t work, but that is where I started to learn why and what I could do instead. Fortunately, there were books I could read to learn more about Windows 95 and related programs.

Moving to XP and Programming

After some time, I managed to convince my parents to pay a fortune for a new computer. It was a tower PC with Windows XP, and it was a huge step forward in performance and stability. I was able to play games and try different programs available at that time.

This was the PC where I started to learn programming. My first motivation was to automate an Ultima Online client, a popular MMORPG. My character could move by itself, chop wood, and train skills without me being there.

The Shift to Linux and macOS

Unfortunately, hardware kept moving forward, and my PC struggled with new Windows versions and games. That’s when I first tried Linux.

I didn’t want to learn Linux. I just wanted to install it and use it like Windows. That didn’t go as planned. After some trial and error, I found myself with Gentoo Linux and a good understanding of how it worked.

When I got my first job, I used Linux for work and kept Windows only for gaming. Later, companies started providing MacBooks. After some initial conflicts (especially with the Magic Mouse), I switched to macOS as my main OS and was really happy with it.

For the last 10 years, I haven’t used Linux or Windows daily. I was locked into Apple’s ecosystem. To be fair, it works great, except for gaming and my nostalgia for Linux.

My New PC

Recently, I decided to buy a new PC. A real tower PC, like the ones from the past.

It allows upgrades and lets you see what is inside.
It doesn’t require battery optimizations or sleep mode management.

As expected, the device came with Windows 11, and I thought I would use it as my main OS. I remembered Microsoft doing a great job with WSL and WSL2. I thought that after all these years, Windows would provide a better development experience than macOS. Plus, gaming is a huge bonus.

Oh man, I was so wrong.

What exactly shocked me? I’ll share that in a moment. But first, the Linux part. I installed different Linux distributions and was surprised by how limited some package managers are (different PPAs, untrustworthy repositories). After some hopes and warm memories of Gentoo, I settled on Arch Linux, which is now the main OS on my new PC.

OS Comparison

macOS: The Ecosystem

I was really happy with macOS. It was stable, fast, and had a great software ecosystem:

Safari: A good browser. It works, is energy efficient, and has almost no compatibility issues.
Terminal: Good enough. I recently moved to Ghostty (a really good terminal emulator), but the default Terminal has all the required features.
Homebrew: Works like a charm. It has all the packages I’ve ever needed. It is the package manager I’d expect in Linux (especially Ubuntu).
Native Apps (Mail, Calendar, Notes, Reminders, Maps, Photos, Music): These apps are amazing. They are minimalistic but work well. I rarely looked for replacements.

Apple managed to build most apps that people need (outside of professional exceptions). There is no need to pay for or download extras. Many users stay inside this default set and are happy.

Windows 11: The Reality Check

Windows? Windows is different. I heard many negative things about Windows over the years, but people always complain, right? However, from my experience:

1. Setup and Package Management

Windows loaded quite fast. I had an issue with the PIN code setup, but in general, the experience was okay. Then I learned about winget. I was really surprised—having a native package manager with many apps is a huge plus.

But winget was also the first disappointment.

Some apps installed via winget did not work.
I had to download installers from official websites and install them manually anyway.

2. The Ad Experience

After installation, I saw many news widgets and unexpected pop-ups. It felt like some of them were promotions. Who expects to see ads on their desktop OS?

3. Missing Core Apps

The next step was opening Calendar, Mail, or Notes. I was surprised to see that real native programs are effectively missing. They are there, but they feel like web views or just stripped-down versions of Outlook with limited functionality unless you subscribe.

Pre-installed Copilot is forced as much as possible. And Edge… Edge is a decent browser, but it has too many “features” on top. I prefer a minimalistic browser.

4. Gaming Stability

Windows is the GOAT of gaming, right? At least I thought so. In general, games work with fewer compatibility issues (after the initial shader compilation).

But why do games crash so often?

I installed Fallout from Xbox (via the Game Pass subscription that came with the PC). During the first hour, it crashed about 10 times. I’m not sure why I was so motivated to keep playing, but is this how people play on PC? Did I do something wrong? I don’t know, but the experience was unpleasant.

5. WSL (Windows Subsystem for Linux)

And last but not least: WSL. It’s the feature everyone praised for Linux integration.

To be fair, it works. It’s not bad at all. But it is not native, and it does not feel as good as the development experience in macOS¹.

Bottom Line

To finish positively, I believe Windows is not terrible. Microsoft just needs to invest a little less into Copilot and a little more into quality.

Outlook is great, but without a subscription, the default apps feel lacking compared to Google or Apple.
If Microsoft looked at the best parts of the Apple ecosystem, they could catch up fast.
They have everything needed to make this OS great, they just need to focus on users, UX, and quality.

I will still load Windows from time to time to see if things change. But for this PC, Arch Linux has definitely won.

Footnotes

Linux is outside of this comparison. It’s just 10x better for backend development, especially since all my servers run Linux. ↩

Project-based local environment with mise-en-place

Anton Ohorodnyk — Sat, 14 Mar 2026 01:17:36 GMT

Open Table of contents

Development environment

Do you work on open source, commercial projects, or several projects at once?

Many of us do, usually with a mix of all of them. For example, this blog is built with Astro, and it can become outdated if I do not touch it for a while. How can we make sure builds stay reproducible and still work years later?

Modern development relies on a few layers of version management:

Dependency management, like package.json, go.mod, Cargo.toml, etc.
Deployment environments like Docker/Podman, cloud VM images, etc.
OS package managers like pacman, apt, yum, etc.

All of this works well when we target isolated environments. We build one cloud instance per service, one Dockerfile per app, and one dependency manifest per project.

But in my local development environment, I still struggle.

Most of the time I use macOS, but recently I also started using Arch Linux. Both operating systems have great package managers. On macOS I use Homebrew, on Arch Linux, pacman and AUR are more than enough for me.

But when I work on different projects, they have different requirements:

Different compiler versions
Extra tools, such as dbmate, pre-commit, prek, biome, golangci-lint, etc.
Project-specific configuration requirements, like environment variables and bootstrap scripts

And the most frustrating part: versions must be exact. Many tools are not backward compatible (especially linters and language toolchains), so one update can break expected behavior.

Testing solutions

Over the years, different teams I worked with tried many ways to solve this and build a reproducible environment.

Virtual machine with file sync

In my experience, the most obvious solution was a preconfigured project VM. I had to configure source synchronization and run everything remotely.

This works in many cases, but it is not very comfortable, and everyone depends on a good internet connection. If you need to upload something large (I can only imagine uploading node_modules. I would run pnpm install directly on the server), it is painful. It takes time, and sometimes it fails entirely.

Remote debugging is a separate topic, but it is rarely a pleasant experience.

It also does not solve outdated projects well, where you need an older version of Python. That can mean one VM per project if it falls outside the team’s default stack.

But most importantly, it’s not what I personally prefer.

I want to have a native local experience. If I want to debug something, it just works. I can work without internet, or with an unstable connection. I want to get the best experience from my IDE.

It is hard to avoid LLMs now. If I use them (and many companies expect that), I still want a native local workflow.

In short, I have never had a truly pleasant experience using a remote instance as my main development environment.

Virtual machine with IDE

This is a relatively new idea for many developers, but not for people used to working in Vim or Emacs over terminals.

Right now we have many options:

Gitpod - I used it when they had just started
GitHub Codespaces - I tried it, but it feels expensive for personal use
Visual Studio Code Server - also an option, but not all extensions are available
[N]Vim or Emacs through SSH with TMUX
Google internal corporate development experience

All of these are great, but they work best for companies ready to manage developer environments with templates.

They are often too expensive for personal use, and they still require internet access. Maybe I am old-fashioned, but sometimes I want to work offline.

I’m sure many people will go for it, but it’s not for me, at least for today.

Development Containers

Another obvious solution is containers.

The first time I tried this approach, a colleague suggested Vagrant as a dev environment. To be fair, it was workable, but that was more than 10 years ago, and it was less usable than most options today.

Since Docker was released, deployment workflows changed significantly. We can now build a container and run it much more easily and cheaply across projects.

Not Docker itself, but Dockerfile was a revolution!

Development Containers can work well, but they still do not feel as smooth as a truly local setup.

I still use Development Containers from time to time, but lately I almost completely migrated to a native experience.

💡 Tip: use isolated environments like virtual machines, containers, or devcontainers when running untrusted code.

Local and native experience

I used to just install all required tools using Homebrew. It just works, no quirks.

Until it does not.

A recent example: I had a website built with [Go]Hugo that I had not touched for years, except for small content updates. Some articles were added, edited, or removed. But [Go]Hugo is not strongly backward compatible. When I come back to that site, it often does not build locally until I fix things. I cannot compile something that used to work.

I had the same experience with Python, PHP, Ruby, and others. It is not just Hugo.

So the problem is clear. I can pin app dependencies to keep runtime behavior stable. But that does not fully solve the local development environment itself.

At one point, I wanted to try monorepo tooling. I thought a monorepo might fit one of my projects. I wired many separate Taskfile files together by including them in each other.

After some research for something that worked for me (outside the JS/TS world), I found moonrepo with proto, and surprisingly it could pull various dependencies automatically. So, I did not need to configure an environment for my app.

It downloaded and installed required languages and additional tools with specified versions.

I was happy with it at first, but it was not enough. Some tools I needed were not available, so I still had to install them separately, which brought back the same problem.

For me, it was a partial solution, not a complete one. It is definitely nice, but proto still feels unfinished.

mise-en-place

Then I discovered mise. It felt like a better fit, with even more potential as I learned it.

Initially I started with a simple [tools] block, like this:

node = "24.14"
pnpm = "10.32"
go = "1.25"
golangci-lint = "2.11"
rust = "1.94"

After configuring my Fish shell (you can check my dotfiles repo), it just worked:

#!/usr/bin/env fish

brew install mise
echo 'mise activate fish | source' > ~/.config/fish/conf.d/mise.fish

You cd into a directory and get all required tools with the right versions.

> prek --version
fish: Unknown command: prek
> cd safigo.dev
> prek --version
prek 0.3.5

Plugins for JetBrains IDEs and Visual Studio Code significantly improve the experience.

You simply create a mise.toml configuration (or another available way) and everyone who has configured mise gets a reproducible local development environment and build results.

You can install tools from many sources, including npm and GitHub. This is one of its most important features. So far, I have not found a tool I need that I cannot install with mise.

Simplification by mise-en-place

Before mise I used to have pyenv, pyautoenv, nvm, goenv, rustup, etc. All of that needed to be configured and kept working.

Later I discovered tools like uv and fnm that simplified my environment significantly, but they are still far from mise.

mise is simply the next level, somewhere between a Brewfile and a Dockerfile.

Security

As I mentioned earlier, never run untrusted repositories, and DO NOT use mise with them.

By default, mise does not allow dependency installation or environment switching until you trust the directory/project by running mise trust.

Be careful, since mise can execute code during mise install and maybe even when you cd there, though I have not fully explored all such hooks yet.

My use cases

I want a reproducible environment when I return to my own projects or start working on someone else’s project, old or new.

I use various tools that can be specific to my preferences.

Here is an example from my blog setup (this is the first post here):

[tools]
# https://github.com/nodejs/node
node = "24.14"
# https://github.com/pnpm/pnpm
pnpm = "10.32"
# https://github.com/j178/prek
prek = { version = "0.3", postinstall = "prek install" }
# https://github.com/biomejs/biome
"github:biomejs/biome" = { version = "2.4.6", version_prefix = "@biomejs/biome@", bin = "biome" }

[tasks.install]
run = "pnpm install --frozen-lockfile"
alias = ["i"]

[tasks.dev]
run = "pnpm run dev"
alias = ["d"]

[tasks.lint]
run = "biome check"

[tasks.lint-fix]
run = "biome check --write"

[env]
ENV = "local" # Just an example

As you can see, a contributor only needs to run mise install, and everything I expect will be installed. It also includes a post-install script that sets up the pre-commit hook automatically for anyone who contributes.

Another interesting line in this config is:

"github:biomejs/biome" = { version = "2.4.6", version_prefix = "@biomejs/biome@", bin = "biome" }

It does not use a default biome plugin and instead downloads from GitHub, even though Biome releases use unusual tags like @biomejs/biome@2.4.6. Even when a repository uses unusual version tags, mise can strip prefixes and rename the binary.

Why don’t I use npm to fetch biome?

I used to. It worked well, but it only covers a subset of tools. mise feels native and consistent across my projects: go, rust, nodejs. I see no reason why biome would be so unique to install it differently from other tools.

Global tools

Mise also supports to install tools on a global level, similar to npm or go install.

But why? We already have our package manager and npm and other tools!

There are few reasons:

Your tool is not available in your particular package manager
You want to have dotfiles that will install something independently on your OS and Linux distro
You’d prefer to have the latest version of tools pulling directly from their direct repo.

Given all of that, Mise can be handy for global (user) level tool management as wel las project-level.

Bottom line

I know everyone is focused on AI/LLM right now. But foundational tools are still here: libc, gcc, go, python, makefile, taskfile, curl, brew, pacman, apt, mise, and more.

AI may replace some tools, some programs, maybe even some professions, but there will always be a foundation. Something still has to run the infrastructure where AI operates.

I see mise as closer to those fundamental tools and services that power development infrastructure, rather than just another hype tool.

I encourage my colleagues to integrate it into their workflows. So far, I have received only positive feedback.

Vibe Coding Initiation

Anton Ohorodnyk — Sat, 11 Oct 2025 02:32:53 GMT

My experience with vibe coding

Everyone talks about vibe coding. There are many tools that promote it, and many people claim to practice it.

I’ve tried it multiple times with various tools, but it never really worked. Actually, it’s not a complete truth; it worked until it did not.

The actual issue was always the amount of generated code and the quality of the results.

But I’ve found an ideal task for vibe coding - redesign this blog. I wanted to change the theme, but since I’m not a designer, I needed some help.

Firstly, I asked to review the codebase and generate a README for it. Afterwards, I asked to make the website more modern and appealing.

Starting from that, it took multiple iterations to get to the final result. In some cases, it generated a lot of redundant CSS; in some cases, it was quite good. Usually, the model could fix issues and solve small tasks I asked it to do in one or a few iterations.

Apart from CSS quality, the main task was done. It redesigned the blog to a more modern look. But some links did not contain a hover effect, even though all others had.

So, I asked to fix it, describing the issue and showing the example (the page and the text that was affected). It took five or even more iterations until I gave up, since the LLM had generated Exceptions in CSS for the links and it never really worked.

OK, I thought to myself, let’s try to fix it manually. I opened two separate windows and started to compare; all styles were the same, except the line that did not work was multi-line, while the others were single line. I found a width: 100% that did not work in this case.

When I asked to fix it, the LLM fixed it with no issues.

Every iteration it tried to fix the issue, it generated more and more CSS that made an effect on other places.

Meanwhile, the final result was quite good; I decided to go to the next steps and clean it up.

Cleaning CSS

The first step was to ask the LLM to clean CSS from unused styles and make it more compact and efficient. It did some job. Some CSS was deleted, and everything works well.

Why not try to make it even better? Let’s ask again.

The second iteration started to break the website, so I reverted to the previous version.

But even when it was done, the website required some cleaning, where I decided to jump in and fix it manually. It took me around 30 minutes to remove all styles I did not like in a design and keep it more minimalistic.

Final thoughts

When I look at the final CSS, I can clearly state that it’s not perfect. Maybe it’s even not efficient, but it works well and looks good enough.

I think I’ve started to understand the Vibe Coding concept, and I can say that it’s not for everyone. Until LLMs won’t be able to understand the context and the task fully, it will require a lot of iterations and human intervention to do the job. And we did not start speaking about complex tasks.

I’ve tried LLMs with complex tasks very often, and I can say that it’s not working well. For me, it has never worked well yet.

But for simple tasks, specifically from an area you are not familiar with, it can be a great help. It’s not a hired developer, but it feels like an assistant that can do some job for you. It can review your code, generate documentation, fix small issues, and brainstorm ideas.

Dotfiles with Chezmoi

Anton Ohorodnyk — Wed, 08 Oct 2025 04:54:17 GMT

Dotfiles Management

I’m not a fan of complex workflows for personal use, simpler is better. Over many years I’ve seen countless posts and repositories about dotfiles, and often they felt highly customized. Frequently, people provide their own bash scripts to install and manage dotfiles.

That’s not my approach. After many years in software engineering, I appreciate the power of open source and well-designed, well-tested solutions. Even a small library should be thoughtfully designed and carefully tested — so why reinvent the wheel?

I don’t remember how I found chezmoi¹, but it was simple and straightforward. It has good documentation, is actively maintained, and offers many features.

Although I felt comfortable with chezmoi¹, I forgot about it several times. Each time I returned, I was pleasantly surprised at how useful and well-maintained it still was.

So I use it, and I recommend it to others — everyone is happy. Since it has proven itself many times, I can confidently recommend it.

Quick walkthrough

Chezmoi¹ has many features, and most importantly, excellent documentation. I won’t repeat that documentation here. Instead, here’s a quick overview of how I suggest getting started.

As an example, we’ll install my dotfiles from my dotfiles repo and assume you’re using macOS.

Install through Homebrew: brew install chezmoi
Initialize chezmoi with my repo: chezmoi init git@github.com:aohorodnyk/dotfiles.git
Apply dotfiles to your home directory: chezmoi apply, and write your password if prompted.

That’s it — you’ll have my dotfiles in your home directory.

To open the cloned repo, run chezmoi cd — it will open the repo in your terminal.

Read docs, it’s really powerful!

Good luck with your dotfiles management!

This note isn’t just about chezmoi — it’s about the value of exploring and learning new technologies, and of continual self-improvement. Even after 10, 20, or 30 years of experience, you can still learn and grow.

Feel free to reach out if you have any questions or need assistance.

Footnotes

Repo and website ↩ ↩² ↩³

KISS for websites below 14KB

Anton Ohorodnyk — Sat, 20 Sep 2025 01:42:47 GMT

And here we go again

How many times have you heard that your website should be lightweight and fast? How many times have “geeks” asked to remove unnecessary JavaScript and CSS?

And here we go again, one more reason to keep your website simple and lightweight by referring to the Why your website should be under 14kB in size article.

When I read about it, it just flashed in my mind that some websites just feel blazing fast. For example, a blog you read this article on. It does not feel like it is loaded, it just appears.

OK, I thought to myself. Let’s check the actual size of the page. It’s 5-7KB. Hm… Let’s check some other blogs.

But when I went through other websites, the difference felt (especially on slower connections). OK then, how hard is it to keep everything below 14KB?

Actually, 14KB is a lot of text. If we keep it simple, the design would be much more readable and the size would be small. My blog is a theme I found on the internet and slightly modified it. But investing a little bit of time may make the size even smaller.

My bottom line would be “geeky” and very predictable. Keep it simple, keep it small, keep it fast. I want to see more simple sites and blogs. Use static site generators, not WordPress or other CMS.

If you don’t earn money from your website, why make it complex?

Capturing traffic with Golang

Anton Ohorodnyk — Sat, 31 Aug 2024 20:43:16 GMT

Introduction

Most of the people from Software Engineering world heard about TCPDump, WireShark, etc. And most likely you have heard about libpcap¹ library developed by TCPDump team to capture traffic and used by Wireshark as well.

This library¹ provides a flexible interface to capture a traffic from a network interface and process it in a code. It provides a flexibility to write performant code and include as much business logic as needed to collect only required data.

Recently I’ve identified a task where it would be a good idea to collect required packets, analyze them and save for the future manual review. To review manually WireShak can be used to load pcap file and manually review the collected packets using nice UI.

And the most important part, why Go? Go is a nice simple enough language to write code and support it by multiple team members. It’s much more safe than C and C++ and requires much less experience for other people to support it without unexpected surpises. Until we don’t have requirements when we HAVE TO go with much performant language (C, C++, rust, etc.), I prefer to choose Golang.

Task definition

Before we will do something, it would be great to understand, what do we want to achieve in a result? Let’s define a short list of requirements.

Requirements

To keep implementation as simple as possible, let’s define only few points:

We want to collect outcoming traffic
Let’s collect IPv4 traffic
Ignore private networks traffic
And let’s keep UDP packets

These simple few points will be enough to get an idea on how to use libpcap¹ from Golang. After that it’s only matter of imagination on what logic will be added on top of that.

Implementation

Before we will start, let’s define that code is not suppose to be a production ready. Our goal is to see the minimum example and test that it works well.

We will use these libraries:

slog for loggin
github.com/google/gopacket for capturing packets and formatting pcap file format

An interface in the future code will be eth0, even though most likely your system will have a different interaface name.

Working code

This is the code with comments you can copy and try in your own environment. Since we use libpcap¹, our application will use CGO and we will need to run the app from root user.

package main

import (
	"bytes"
	"log/slog"
	"net"
	"os"

	"github.com/google/gopacket"
	"github.com/google/gopacket/layers"
	"github.com/google/gopacket/pcap"
	"github.com/google/gopacket/pcapgo"
)

const (
	interfaceName = "eth0"
	snaplen       = 1500
)

func main() {
	slog.Info("Running our applicaiton...")

	// Get handler attached to an interface.
	handle, err := pcap.OpenLive(interfaceName, snaplen, true, pcap.BlockForever)
	if err != nil {
		slog.Error("Could not OpenLive", slog.String("err", err.Error()))
		os.Exit(1)
	}

	iface, err := net.InterfaceByName(interfaceName)
	if err != nil {
		slog.Error("Could not OpenLive", slog.String("err", err.Error()))
		os.Exit(1)
	}

	// Start new Source reader.
	source := gopacket.NewPacketSource(handle, handle.LinkType())

	// This is suppose to be a file writer, but we will use memory, just for simplification.
	fileWriter := bytes.NewBuffer(nil)
	pcapWriter := pcapgo.NewWriterNanos(fileWriter)
	err = pcapWriter.WriteFileHeader(snaplen, handle.LinkType())
	if err != nil {
		slog.Error("Could not write pcap header", slog.String("err", err.Error()))
		os.Exit(1)
	}

	// Reading packages.
	for packet := range source.Packets() {
		// Filter by outcoming traffic only.
		// To filter it, we need to compare MAC addresses from out interface and source MAC.
		// To access a mac Address we need to get an Ethernet layer.
		layer := packet.Layer(layers.LayerTypeEthernet)

		ethernet, ok := layer.(*layers.Ethernet)
		if !ok {
			slog.Error("Could not get Ethernet layer")
			continue
		}

		if !bytes.Equal(ethernet.SrcMAC, iface.HardwareAddr) {
			// Our interface did not send this packet. It's not outcoming.
			continue
		}

		// Now we need to identify IPv4 layer.
		layer = packet.Layer(layers.LayerTypeIPv4)

		ipv4, ok := layer.(*layers.IPv4)
		if !ok {
			// It's not IPv4 traffic.
			continue
		}

		if ipv4.DstIP.IsPrivate() {
			// Do not collect private traffic.
			continue
		}

		if ipv4.Protocol != layers.IPProtocolUDP {
			// Ignore not UDP protocol.
			continue
		}

		err = pcapWriter.WritePacket(packet.Metadata().CaptureInfo, packet.Data())
		if err != nil {
			slog.Error("Could not write a packet to a pcap writer", slog.String("err", err.Error()))

			continue
		}

		slog.Info("Stored packet", slog.Any("packet", packet))

		// Let's collect ONLY 100K bytes, just for example perposes.
		if fileWriter.Len() > 100000 {
			break
		}
	}

	slog.Info("We have successfuly collected bytes", slog.Int("bytes", fileWriter.Len()))
}

And after running there the truncated output looks like that:

2024/08/31 13:35:36 INFO Running our applicaiton...
2024/08/31 13:37:48 INFO Stored packet packet="PACKET: 105 bytes, wire length 105 cap length 105 ..."
...
2024/08/31 13:37:48 INFO Stored packet packet="PACKET: 1291 bytes, wire length 1291 cap length 1291 ..."
2024/08/31 13:37:48 INFO We have successfuly collected bytes bytes=101018

I hope this minimum example will help someone to start their jorney in this area using Go.

Afterword

If you meet any new requirements in your everyday tasks that you have no idea how to do. Don’t hasitate to ask people around, write me in a private and do your own research.

Research and communication is a key to all problems solving.

Footnotes

libpcap is a portable C/C++ library for network traffic capture ↩ ↩² ↩³ ↩⁴

Customer metrics availability

Anton Ohorodnyk — Thu, 25 Apr 2024 02:23:16 GMT

Overengineering or not?

Often enough I share an idea that all new serverless platforms as well as simple cloud providers like Digital Ocean do not support customer metrics. All the time I hear it’s not needed for non-enterprise customers and I ask for the feature due my overengineering mindset.

But today I saw a blogpost in my RSS reader from Internal Tech Emails:

From: Bill Gates Sent: Thursday, October 19, 2000 3:02 PM To: Will Poole Cc: Jim Allchin; Eric Rudder Subject: Quality experience and streaming media

I don’t know how unique I am but my experience in clicking on streaming media stuff and seeing if it works is still less than 50%.

For example the MSN home page talk about a golf match going on called the President’s cup. I click on that and then I click on this link:

http://stream1.[…].com/[…]/101700192418001.rm

I get a message saying: “This document is not a RealAudio or RealVideo Document” “This is error 11. To learn more go to a real URL.

This is so typical. I always get error messages when I click on Audio and Video.

Just for fun I go to the home page and grab an MSNBC video clip:

<http://go.msn.com/CM/10002/default.asp?target=http://www.msnbc.com/[…]\>

I click and bizarrely I get a browser page and a video page - the lack of integration of the browser and the player is crazy.

The browser page is fine but the video says buffering and 5 minutes later nothing has happened.

We should go on a quality campaign. We should really collect error message reports from our servers.

Something is very hard to set up or hard to track in this area. The quality is giving us a bad name.

I guess I can keep sending all my failures along - the key point is that I haven’t seen any improvement.

I have yet to have a good video experience.

Where Bill Gates asks for a solution to collect logs from different servers to monitor them. It was ~24 years ago, Microsoft that is already big company with many customers and many servers, and they still don’t have a solution for this.

The CEO experiences an issue with a production product, but there is not simple solution to look at metrics/logs/etc. to identify the reason of the issue. I’m sure it’s not the case in current Microsoft’s infrastructure.

Moreover, there is no such cases even in small per projects or startups that just deployed a single serverless function to any serverless solution. But why?

Available solutions

The main reason we have logs in any cloud provider is believement in this feature as essential part of any system. In current environment we just expect to have it and the most of us will be surprised to find out that it’s not available.

But what about metrics?

Suprisingly, the most providers provide system metrics along with logs. We expect to have some high-level system metrics along with logs for every component of our system. But, there is no such case for custom metrics. The most of the providers do not provide a buildin solution for custom metrics in a service.

We can try to explain it as an expensive solution, or not popular request from customers. However, in my opinion, it’s because there is no the first provider who started to provide it.

It’s only matter of time after the first big player who will provide it, other would copy and standardize the solution. I’m waiting for the first serverless provider (Supabase? Firebase? Vercel?) to introduce it as a free (or cheap) additional service and it will be spread to all small and big cloud proviers at no time.

Conclusion

Why have I decided to write down my thoughts?

I hope to engage someone to add this small, but really important feature to their platform. I believe it will be a killer feature for the platform (at least for short period of time while it’s uniq their feature) as well as a great investment to our bright future of cloud services.

Signin with Apple with Expo's app.json

Anton Ohorodnyk — Sat, 13 Apr 2024 17:37:12 GMT

Motivation

Apple’s Sign in with Apple feature is a convenient way for users to sign in to your app without creating a new account. It provides a secure and private way to authenticate users and helps you comply with privacy regulations. If you are building an application to distribute on the App Store and you offer third-party sign-in options, you must include Sign in with Apple as an option.

Expo team has made their own implementation of Sign in with Apple library expo-apple-authentication that may work for you. I have never tried this library, but consider using itm before going through this article’s recommendations.

Firebase React Native library

In my application I’ve decided to use Firebase to authenticate users. Firebase provides a way to authenticate users using Apple’s Sign in with Apple feature. To enable this feature, you need to configure your app in the Apple Developer Console and Firebase Console.

For React Native there is a cool implementation for Firebase integrations called react-native-firebase. It provides instructions and examples on how to integrate Firebase Authencation with Social Providers like Google, Facebook, Twitter, and Apple, etc.

The library provides detailed instructions on how to configure all social networks authentication providers, but Sign In with Apple require some additional steps to be configured in Xcode. But since I play with Expo, I generate ios and android folders automatically, and I don’t want to configure anything manually. Expo should provide a way to configure this automatically as all other parts of the app and the project.

Expo iOS app configuration

After a couple of ours of research, I’ve found a way to configure Sign in with Apple for Expo iOS app through app configuration (app.json/app.config.js/app.cofngi.ts). Expo’s configuration paramerter has name entitilements.

Here is an example on how to enable Sign In with Apple in Xcode project for auto-generated Expo iOS app:

{
  "ios": {
    "entitlements": {
      "com.apple.developer.applesignin": ["Default"]
    }
  }
}

Funny part:

I tried to use different AI chats to get help with this issue, but they all were not able to help me. Regular old-style search engines and GitHub/StackOverflow/Reddit/etc. are still the best way to get help with issues.

List all ignored files in git

Anton Ohorodnyk — Wed, 29 Nov 2023 06:00:00 GMT

Motivation

Recently, I’ve been working on a project where I, as usual, vendored my Golang dependencies by go mod vendor command. I then added these files to git and pushed them to the remote repository.

However, I noticed that the CI/CD pipeline was failing due to the presence of these files. I was puzzled as to why this was happening. I then realized that some files or directories were being ignored by git, and I had to find out which ones.

The Method

I was aware of the git status --ignored command, but it didn’t provide the desired output. I wanted to see the list of ignored files and directories, but the command only showed the ignored files that were already tracked by git. I needed to see all ignored files, including the untracked ones.

I then discovered the git ls-files --others --ignored --exclude-standard command, which did exactly what I wanted. It listed all ignored files and directories, including the untracked ones. Where I could then see the files and directories that were being ignored by a .gitignore file generated by gitignore.io.

Epilogue

In troubleshooting a CI/CD hiccup during Golang dependency vendoring, a routine process led to unexpected complexities. Conventional Git commands fell short, prompting the discovery of git ls-files —others —ignored —exclude-standard. The culprit? A dynamically generated .gitignore from gitignore.io. This incident is a reminder of the dynamic nature of development workflows, where routine tasks can unearth unforeseen challenges, paving the way for continual learning.

Golang panics in libraries

Anton Ohorodnyk — Tue, 01 Aug 2023 04:32:14 GMT

Introduction

When I read about Golang for the first time, I was an active user of languages with exceptions, like Java, PHP, Python, Ruby, etc.

It was obvious that big projects handle their behaviour through exceptions and we can control flow through try/catch blocks in parent call stack.

Outside languages with exceptions I had some experience with other languages like C or codebases that prohibit exceptions.

Experience without exceptions, usually was less pleasant, because it applied many restrictions and limitations to the codebase. Additionally, many of these codebases did not support return for multiple values, so the flow control had to be built through returning unsupported values (like -1 for ints and null for references). Because of these limitations, there were popular ways to handle these restrictions through reference parameters and returning results through them.

Golang

There we are! We have met Golang for the first time. The documentation and articles and bool suggest to use error return value, instead of exceptions. More over, exceptions are not supported in Golang at all (or almost).

By going through Golang documentation and the codebase, many people notice that there is a panic with recover, that can be used as exceptions. These people write many articles, posts, and comments about how panics in Go. But Golang community always criticizes these thoughts and suggests to use error returned value instead of panic. Moreover, those people share an opinion that error handling in Golang is much better than exceptions in other languages.

Let’s try to understand why it is so.

Error handling

Before we will compare exceptions and error handling from Golang, let’s try to go through the main ways to handle errors.

There are three of them:

Return error from the function.
Throw error through current stack of calls until someone will catch it.
Stop execution of the program.

All these ways are used in many different languages in parallel and non of them is better than others.

Return error from the function

This way is the first way I’ve seen when I started to learn programming in C and Assembly. It is the most obvious way to handle errors, because we can just agree that some special value will be returned in case of error. In case of C it is -1 for ints and NULL for references. Sometimes it cam be some error parameter that will be filled with error value by reference.

The best thing about this way is that it is very simple and easy to understand. It increases readability of the code and makes it easy to debug by printing a value or putting debugger on the line to see the scope of variables. In case of C it was kinda confusing, since we did not have predictable way to work with errors and we had to check the documentation of each function to understand how it handles errors.

In case of modern languages like Golang, Rust, etc. we have a predefined way to work with error handling by specific error types and rules on how to use them. It improves readability and simplifies interfaces of functions.

Basically all of the changes around it are just cosmetic that make huge difference, but concept is still the same.

In the diagram above we can see the flow of the program that uses this approach to handle errors. Short explanation of the diagram:

main function calls func1 function.
func1 function calls func2 function.
func2 function returns an error.
func1 function checks return values from func2 function and returns an error based on error from func1.
main function checks return values from func1 function and handles the error.
main function can continue execution of the program.

The main idea of this approach a software developer should handle errors in each function in the call stack. And they can handle errors on any level of the call stack. If we do not know what to do on a specific level, we can just return an error to the caller and let them handle it.

Pros

Easy to understand.
Easy to debug.
Easy to use.

Cons

In some cases this approach adds a boilerplate code.
We need to thing about error handling in each call of a function.
Every function in the call stack should handle errors and return them back to the caller, if needed.
It can be simple to ignore an error without adding any handling for it.

This handle of errors is used to guarantee that an error won’t be ignored and developer will always aware about an error.

Throw error through current stack of calls until someone will catch it

The main idea of this approach is to throw an error from a function up to callers stack until someone will catch it. This way is the most popular way to handle errors in languages with exceptions like Java, PHP, Python, Ruby, etc.

By using this approach we can send a error through call stack without adding any code and knowledge to intermediate levels. It simplifies development and removes requirements to thing about errors on the most levels of the call stack.

Languages that support exceptions have a special keyword to throw an exception and to catch it.

In case of Java and PHP it is throw and try/catch blocks.
In case of Python it is raise and try/except blocks.
In case of Ruby it is raise and begin/rescue blocks.

But all of them have the same idea to control the execution flow: throw an exception and catch it on the level where we want handle it.

With Go it is a bit different. We do not have exceptions, but we have panic and recover functions. Panics by themself are not control-flow statements, they are closer to Java Error type or PHP ErrorException type. They are used to stop execution of the program in case of critical errors that are not related to business flow of the program. In case of Go we can use panic to throw an error and recover to catch it, but we can catch it only in defer functions that are not linear execution code block and applies some limitations on top of it.

In the diagram above we can find the flow of the program that uses this approach to handle errors. Short explanation of the diagram:

main function calls func1 function.
func1 function calls func2 function.
func2 function throws an error.
main function catches an error and handles it.
main function can continue execution of the program.

This way to handle errors is used to simplify code writing and to reduce boilerplate code on intermediate levels of the call stack.

Pros

Easy to write new code.
Zero boilerplate code on intermediate levels of the call stack.

Cons

It is hard to understand where an error was thrown.
It is hard to debug.
It is hard to understand where an error will be caught, if it will be caught at all.
If error was not caught, the program will be stopped during an execution.

Stop execution of the program

The most radical way to handle errors is to stop execution of the program in case of any error. In the most languages it is done by using exit function that stops execution of the program and returns an error code to the caller. To handle errors there assert function that checks a condition and stops execution of the program if it is not met.

There is no way to recover the execution of the program after exit or assert functions were called. The one way error handling is used to stop execution in case of some critical errors that are not related to business flow of the program, but they also cannot be handled by the program.

The simplest example of such error is a unsupported value of a parameter in command line interface. In this case we can just stop execution of the program and print an error message to the user.

Let’s review the flow of the program that uses this approach to handle errors. The diagram above shows the flow. In this diagram we added OS level that represents the operating system that runs the program from an entry point¹.

Short explanation of the diagram:

os calls an entry point of a program, usually by calling the main function.
main function calls func1 function.
func1 function calls func2 function.
func2 function checks a condition and stops execution of the program.
os receives an error code from the program and handles it and stops execution.

This approach is used to stop execution of the program in case of critical errors that are not related to business flow of the program.

Pros

Easy to stop an execution of the program.
Guarantees that the program will not continue execution in case of critical errors.
Application will stop very fast.

Cons

Since the program stops execution with a code, it is hard to debug without logs.
The program stops execution, that is not what we usually want.

Panic and recover in Go

We went through different ways to handle errors in the program. We have successfully reviewed the flows and pros and cons of each approach. During the review we also mentioned that Go supports all of them, but it’s important to jump a little bit deeper into it.

Error type in Go

Go has a special interface to represent an error. It is called error and it is defined in the builtin package.

type error interface {
	Error() string
}

We can use this interface to define our own error types and to use them in our programs.

// NewMyError factory method that creates a new MyError.
func NewMyError(message string) error {
  return &MyError{message: message}
}

type MyError struct {
  message string
}

func (e *MyError) Error() string {
  return e.message
}

The error type is expected to be the latest return value of a function.

func func1() error {
  return NewMyError("error message")
}

func func2() (int, string, bool, error) {
  return 0, "", false, NewMyError("error message")
}

And we also expect that the caller will check the error and handle it. There is a popular linter that checks it for us: errcheck.

Also, errors can be stacked in a chain, wuth two interfaces:

interface {
  Unwrap() error
}

interface {
  Unwrap() []error
}

With errors package that contains useful helpers to work with errors.

Errors is the main way to handle errors in Go and it is used in most of the cases and suggested to be used when possible.

Note: If you can use error type, use it. Do not even thing to use any other error handling approaches.

Panic and recover

We went through throw error handling flow. Although exceptions are very popular in other languages, Go does not support them. In Go we have panic and recover functions that are used to handle critical errors.

The code that uses panic usually means that something went wrong, very wrong. When we throw panic we stop execution of current code and start to unwind the stack of the program, with no call stack code execution. But all defers are still executed, even for functions that are not expecting to handle panics. We will see it in the following example.

package main

import "fmt"

func main() {
	defer func() { fmt.Println("Defer main") }()
	fmt.Println("Hello world")

	func1()

	fmt.Println("Not executed code main")
}

func func1() {
	defer func() { fmt.Println("Defer func1") }()

	func2()

	fmt.Println("Not executed code func1")
}

func func2() {
	defer func() { fmt.Println("Defer func2") }()

	panic("Test panic")

	fmt.Println("Not executed code func2")
}

// Output:
// Hello world
// Defer func2
// Defer func1
// Defer main
// panic: Test panic
//
// goroutine 1 [running]:
// main.func2()
// 	/tmp/sandbox493636343/prog.go:26 +0x49
// main.func1()
// 	/tmp/sandbox493636343/prog.go:19 +0x3f
// main.main()
// 	/tmp/sandbox493636343/prog.go:11 +0x7d
//
// Program exited.

Since we are using panic in func2 function, we stop execution of the program and start to unwind the stack. All defers are executed from func2, func1 and main. Because we have not recovered from the panic, the program stops execution and prints a panic message with stack trace.

We can use recover to handle panics and sometimes we will, but as good practice let’s agree that we will not use panics as a control flow of the program.

Stop execution

Sometomes we want to just stop execution of the program. For example, we have a program that is a CLI tool and we want to stop execution of the program if the user provided wrong parameters.

We can use os.Exit to stop execution of the program.

package main

import (
  "fmt"
  "os"
)

func main() {
  if len(os.Args) < 2 {
    fmt.Println("Please provide a name")
    os.Exit(1)
  }

  fmt.Println("Hello", os.Args[1])
}

// > ./main test
// Hello test
// > ./main
// Please provide a name
// exit status 1

os.Exit will close program emidiately and will not execute any defers. This way is even more dangerous than panic and should be used only in cases when we know what do we do.

Note: Do not use os.Exit unless you know what you are doing.

Panic in libraries

Finally, let’s talk about panics in libraries. We have already mentioned that we should not use panic as a control flow of the program. But, why are libraries so unique?

Libraries are created to be used by other programs and people. When people use libraries, they expect that the library will not stop execution of the program. If there are any errors, library will return an error and the caller will know about it based on the interface and can handle it.

This approach helps to be sure that even without careful reading of the documentation, the caller will not be surprised by the behavior of the library. And it is a good practice to follow. Error handling through return value helps to be Go code more predictable and stable. This simple rule helps to make programs to be unfailable on production.

One exception and informal agreement

Although we have a rule that we should not use panic in libraries, there is one exception. Sometimes libraries return error value and we know that caller in the most cases will not handle it and just panic.

There are some examples:

regexp.Compile - it is expected that the caller will provide a valid regular expression.
- If the regexp is set in constant code regexp.Compile("invalid regexp"), we will just panic and fix the code.
- If the regexp is provided as an input from the user, we should handle the error ro notify user and move forward with execution.
uuid.NewRandom (source code) Where we create a random UUID generator. If we have a problem with the random generator, it will return an error.
- In the most cases we will panic, because we expect that random generator will work.
- In some cases we will handle the error and use some fallback generator (for example from an external service).

As you can see, there are some cases when we expect from the caller to panic in the most cases of using a library. We ALWAYS provide the interface with error returned as a value, but since we know that in the most cases the caller will not handle the error, we can provide a helper function that will panic in case of error.

Let’s see an example:

package main

import (
	"errors"
	"fmt"
)

func main() {
	fmt.Println(Compile("test"))     // Will return "test".
	fmt.Println(Compile(""))         // Will return the error.
	fmt.Println(MustCompile("test")) // Will return "test".
	fmt.Println(MustCompile(""))     // Will panic
}

func Compile(regexp string) (string, error) {
	if regexp == "" {
		return "", errors.New("critical error, empty regexp")
	}

	return regexp, nil
}

func MustCompile(regexp string) string {
	res, err := Compile(regexp)
	if err != nil {
		panic(err)
	}

	return res
}

// Output:
// test <nil>
//  critical error, empty regexp
// test
// panic: critical error, empty regexp
//
// goroutine 1 [running]:
// main.MustCompile(...)
// 	/tmp/sandbox2091129551/prog.go:26
// main.main()
// 	/tmp/sandbox2091129551/prog.go:12 +0x178
//
// Program exited.

There is a good practice or/and agreement that if we have a helper-function that throws a panic, we should name it with Must prefix. For example: MustCompile, MustNew, MustOpen and so on.

Note: Do not use panic in libraries except you provide a helper function with Must prefix.

Conclusion

In Go we have all tools to handle errors. But the preferrable way to use errors is to return them as a value and handle them in the caller code.

If a developer of a library expects that in the most cases the caller will not handle the error, they can provide a helper function with Must prefix that will panic in case of error. But the main interface of the library MUST still return an error.

Let’s follow good practices and make our code more stable and predictable.

Footnotes

In computer programming, an entry point is the place in a program where the execution of a program begins, and where the program has access to command line arguments. ↩

Multiple mastodon aliases to domain using Cloudflare Pages

Anton Ohorodnyk — Sun, 09 Jul 2023 05:14:23 GMT

Introduction

Recently, I’ve begun utilizing Mastodon as my go-to social media platform for technical queries. I’ve found that while Mastodon’s decentralized nature – a core aspect of the Fediverse – is attractive, it also brings about some complexities. Similar to how email addresses work, a user’s profile on Mastodon is tied to their instance’s domain.

In my journey, I discovered Fosstodon, a server with a community that aligns with my perspectives. I wished to tie my Fosstodon’s account to my personal domain and username, essentially making it me@aohorodnyk.com. For my website, I use the superb Hugo static site generator and deploy it to CloudFlare Pages, but it doesn’t have any server-side logic to resolve the name. Following, this goal came with a challenge: I had to find a solution without any server-side logic to resolve the name.

The Problem

I came across a service named WebFinger in my search for a solution. WebFinger utilizes the fediverse specification to resolve names. When I input my Fosstodon’s account aohorodnyk@fosstodon.org into a search field, it demonstrates how the name gets resolved.

It does these two things:

04:15:24 Looking up WebFinger data for acct:aohorodnyk@fosstodon.org
04:15:24 GET https://fosstodon.org/.well-known/webfinger?resource=acct%3Aaohorodnyk%40fosstodon.org

From my analysis, I concluded that for an alias to my Fosstodon’s account, the same JSON should be resolved by this path: /.well-known/webfinger?resource=acct%3Ame%40aohorodnyk.com.

The simplest approach would be to create a file /.well-known/webfinger with the content from Fosstodon. The downside is that all users in my domain would be resolved with the same account. So, I needed to find a way to resolve different files by different GET parameters to the same path.

The solution

My search for a way to apply minimal logic to resolve different files by different GET parameters to the same path was initially fruitless. The only available solutions online involved using CloudFlare Workers/CloudFlare Functions.

Reluctantly, I was considering using functions. However, the unpredictable pricing of serverless solutions was a deterrent. I didn’t want to incur costs for my minimalist needs.

After several failed attempts, including trying to create a file with the full name /.well-known/webfinger?resource=acct%3Ame%40aohorodnyk.com, I found a ray of hope in CloudFlare Transform Rules. This feature lets us create rules to transform requests to our server.

CloudFlare Transform Rules

The concept was to develop a rule that could parse /.well-known/webfinger?resource=acct%3Ame%40aohorodnyk.com and transform it into a unique file in CloudFlare Pages. The main file /.well-known/webfinger wouldn’t exist at all, ensuring all other requests returned a 404 as expected.

Rule Requirements

While it might not be overly critical, it’s beneficial to lay down requirements for our solution:

The request /.well-known/webfinger?resource=acct%3Ame%40aohorodnyk.com must resolve our JSON
The same JSON should be resolved on the request /.well-known/webfinger?resource=acct%3Ame%40aohorodnyk.com&format=json
For ALL other requests, a 404 error should be returned

Successful Implementation

In CloudFlare’s admin page, I formulated a rule with specific settings and the action to transform the request to the file /.well-known/webfinger-aohorodnyk.json

Expression preview: (http.request.uri.path eq "/.well-known/webfinger" and http.request.uri.query contains "resource=acct:me@aohorodnyk.com") or (http.request.uri.path eq "/.well-known/webfinger" and http.request.uri.query eq "resource=acct%3Ame%40aohorodnyk.com").

And the action is to transform the request to the file /.well-known/webfinger-aohorodnyk.json.

Evaluation

Upon saving the rules, I ran various positive and negative test cases using curlie and found the results to be successful:

# Positive test cases
$ curlie "https://aohorodnyk.com/.well-known/webfinger?resource=acct%3Ame%40aohorodnyk.com"
HTTP/2 200
...
{
  ...
}
$ curlie "https://aohorodnyk.com/.well-known/webfinger?resource=acct:me@aohorodnyk.com"
HTTP/2 200
...
{
  ...
}
# Negative test cases
$ curlie "https://aohorodnyk.com/.well-known/webfinger?resource=acct%3Aame%40aohorodnyk.com" # wrong username with prefix `a`
HTTP/2 404
...
$ curlie "https://aohorodnyk.com/.well-known/webfinger?resource=acct%3Amea%40aohorodnyk.com" # wrong username with suffix `a`
HTTP/2 404
...
$ curlie "https://aohorodnyk.com/.well-known/webfinger?resource=acct%3Ame%40ohorodnyk.com" # wrong domain
HTTP/2 404
...
$ curlie "https://aohorodnyk.com/.well-known/webfinger?resource=acct%3Ame%40aohorodnyk.co" # wrong domain
HTTP/2 404
...

Conclusion

Overall, the solution proved to be straightforward, efficient, cost-free, and operational. Moreover, it demonstrated excellent scalability for multiple users on the same domain.

Effortless Node.js Version Management with the Fish Shell

Anton Ohorodnyk — Sat, 10 Jun 2023 22:00:53 GMT

Motivation

Continuing our exploration of the Fish shell and its synergy with prevalent tools, this post follows the prior piece focused on Python’s virtualenv.

This write-up provides a straightforward strategy to incorporate automatic Node.js version control with the Fish shell, whilst avoiding resource-hungry frameworks like oh-my-fish which can often slow down the shell.

A Brief on Node.js Version Management (nvm)

Node.js is a widely embraced JavaScript runtime environment, enabling the execution of JavaScript code outside a browser. It finds utility in the creation of server-side apps, command-line tools, and even desktop and mobile apps.

nvm, renowned for managing multiple Node.js versions on a single machine, facilitates easy installation and swapping between different Node.js versions with a single command.

nvm, while built for Bash and Zsh shells, isn’t directly compatible with the Fish shell. There are several ways to mesh nvm with the Fish shell, but they are often too intricate or resource-consuming.

In my experience, attempting to use an NVM clone in Fish shell fell short of my expectations. It failed to accomplish the primary task of automatically shifting between Node.js versions when I switch the directory based on the .nvmrc file.

The Method

Objective

The goal is to have Node.js versions switch automatically when the directory is changed, based on the .nvmrc file.

Why not use .node-version? The .nvmrc file is favored due to its support for additional options like iojs and lts/*, plus being a standard for nvm.

The How

The implementation is uncomplicated as usual. It doesn’t entail multiple dependencies or complex logic. It’s essentially a few lines of code that get the job done.

Setting It Up

The plugin is dependent on another plugin edc/bass, which is a utility for executing Bash commands from Fish shell. It’s a simple plugin that can be installed through fisher or manually.

fisher install edc/bass

I suggest using fisher as a simplistic plugin manager for Fish. I’ve authored a plugin that can be easily installed through fisher and amalgamated into any Fish environment. Should you have reasons not to use Fisher, simply copy-paste nvm.fish files from all directories to your ~/.config/fish/ in their respective ones.

To set up the plugin with fisher, execute the following command:

fisher install aohorodnyk/fish-nvm

How to Use

Upon changing the directory based on the .nvmrc file, the plugin will automatically switch between Node.js versions. If the required version of Node.js isn’t installed, it will be installed automatically.

Configurations

No particular configurations are required for this plugin. However, you can configure it by setting the below environment variables:

nvmrc_enable (default: yes): Toggles Fish NVM on or off. Set to no to disable.
nvmrc_announce (default: yes): Toggles on or off the announcement of the Node.js version change. Set to no to disable.

Epilogue

Don’t hesitate to utilize the Fish shell. It’s a fast, versatile, and user-friendly shell that stands as a strong contender to bash and zsh.

Contributing to open-source is a great way to learn, assist others, and help make the technological world simpler and better for all.

Integrating Python's Virtualenv with Fish shell without overcomplicated frameworks

Anton Ohorodnyk — Sat, 10 Jun 2023 20:12:16 GMT

Introduction

The Fish shell is renowned for its user-friendly nature, making it an ideal command-line interface for macOS, Linux, and more. The shell stands out with its unique features, including auto-completion, syntax highlighting, and tabbed completion. Additionally, its learning curve is gentle enough for beginners to grasp quickly.

Despite these attractive attributes, many developers I’ve come across don’t prefer Fish shell, primarily due to integration gaps with tools like Python’s virtualenv. So, in this article, I’m offering a simple solution for automatic virtualenv activation for Fish shell, steering clear of resource-intensive frameworks like oh-my-fish that often slow down the shell.

Understanding Python’s Virtualenv

Python’s virtualenv is a tool that creates isolated Python environments, making it an invaluable resource for managing project dependencies and testing Python’s new versions without affecting the system’s main installation.

It is perplexing and perhaps not ideal to have Python’s dependencies installed globally instead of project-specific folders. This issue necessitates workarounds such as virtualenv.

The Objective

Before diving into the solution, let’s clearly define our goal.

Task

We aim to automate the activation of virtualenv when we navigate into the project folder or any of its sub-folders. Additionally, we want to deactivate virtualenv when exiting the project’s scope.

A bottom-up search for the virtualenv folder is crucial as multiple virtualenvs can reside within the project folder, and we need to use the nearest one.

The ultimate goal is to devise a minimalist solution to minimize performance hindrances and ensure maximum compatibility with any Fish environment.

Solution: An Adapted Plugin

I adapted the code from timothybrown/fish-autovenv, originally created by Timothy Brown, and published the modified version on GitHub.

The existing solution had a few shortcomings:

It was incompatible with newer versions of Fish, and my goal was to retain simplicity.
The plugin for created for oh-my-fish, which is a resource-intensive framework that slows down the shell.
The plugin could only detect virtual environments in the current directory, not in specified sub-directories. For instance, I store my virtual environments in a .venv sub-directory.
The plugin didn’t apply the virtual environment when opening a terminal in a directory containing a custom virtual environment.
It searched for virtualenvs top-down (/home, /home/user, /home/user/projects, /home/user/projects/pytest), whereas a bottom-up approach would have been more efficient and user-friendly.

As a minimalist plugin manager for Fish, I recommend fisher. I’ve created a plugin that can be installed via fisher and integrated into any Fish environment. If there are any reasons why you don’t want to use Fisher, you can just copy-paste conf.d/autoenv.fish file to your ~/.config/fish/conf.d directory.

The project can be found here: aohorodnyk/fish-autovenv.

Installation

To install aohorodnyk/fish-autovenv, by using fisher:

fisher install aohorodnyk/fish-autovenv

The plugin offers several configurable settings for a tailored user experience:

set -U autovenv_enable yes|no - to enable or disable the plugin (enabled by default).
set -U autovenv_announce yes|no - to enable or disable announcements when virtualenv is activated or deactivated (enabled by default).
set -U autovenv_dir '.venv' - to specify the name of the directory where virtualenv is located (default is .venv).

Conclusion

Don’t be apprehensive about using the Fish shell. It’s a versatile, fast, and intuitive shell, making it a worthy alternative to bash and zsh.

Contributing to open-source is an excellent way to learn, help others, and contribute to making the technological world better and simpler for everyone.

Dart should not be ignored

Anton Ohorodnyk — Fri, 09 Jun 2023 05:32:23 GMT

Introduction

Introduced in 2011, Dart is a modern language developed by Google, designed for crafting web, server, and mobile applications. As a general-purpose language, Dart is versatile and flexible, supporting a wide array of developments. It’s a statically typed, garbage-collected language which combines object-oriented and functional programming paradigms. Dart can be compiled to JavaScript and it presents a user-friendly C-style syntax, making it relatively straightforward to learn and apply.

However, despite these features, Dart is not the favored language among the developers I encounter, and positive feedback is quite scarce. Personally, my preference leans towards Go, but I acknowledge Dart has certain distinctive elements we can glean insights from. Let’s explore some unique attributes.

Dart as a Statically Typed Language

Dart’s syntax closely resembles those of Java, C#, and Kotlin. This may be advantageous as these languages are widely recognized, although I confess I’m not particularly enamored with them.

Dart, like many languages, has introduced a concise syntax for variable definition, and it seems to elevate this feature. Not only does it support conventional type declarations such as var, final, and const, Dart also integrates unusual variable types for static languages like late and dynamic. This surprising blend of characteristics imparts Dart with a uniqueness that can broaden our understanding of varying concepts.

Mixins in Dart

I first encountered mixins in Ruby, a feature that although could complicate the understanding of the code, I recognized its potential. Mixins provide a means to incorporate new functionality into classes without necessitating inheritance, enhancing flexibility and promoting the object-oriented programming approach.

Dart embraces mixins but in a more declarative way. When we need to use a function from a mixin, we must import the class’s extension. This enhances the readability and comprehension of the code, which is a commendable improvement.

Null Safety Feature

While null safety is not an innovative feature, being used in numerous languages, Dart has integrated it into its core. It feels inherently a part of the language, which is a refreshing experience.

Conclusion

The objective of this succinct article is to present a few key points aimed at fostering a less toxic environment in our world.

To summarize, Dart may not be my go-to language, nor the most favored among my developer peers. However, its unique features like unusual static type declarations, an improved version of mixins, and native null safety are enlightening and can significantly contribute to our programming knowledge. Despite my preference for Go, I respect and appreciate the diverse learning opportunities Dart presents.

I urge individuals not to despise any programming language, but instead contribute their knowledge and experimentation for its advancement. Dart may not be perfect, but its strengths make it too promising to be dismissed outright.

Is Redis merely caching system?

Anton Ohorodnyk — Sat, 18 Mar 2023 04:00:00 GMT

Background

Improving the performance of distributed applications is a common task for backend software engineers, especially when multiple servers use a database to retrieve data and generate responses. Communication among engineers occurs frequently and often revolves around work-related topics and technology.

Redis and Memcache are commonly referred to as caching systems or cache services, even by experienced individuals, but they should be recognized as full-fledged databases due to their maturity and functionality. This article will explore the reasons why it is important to recognize Redis and Memcache as databases.

Defintions

In order to review the topic, it is necessary to establish clear definitions.

Cache

In my understanding, cache refers to a software layer that stores precalculated results from previous computations, which can be utilized to avoid repeated calculations. Although some people view it as a quick storage solution, I believe this is debatable, as the primary objective of caching is to enhance performance by using a tool that provides adequate speed to achieve significant improvements.

For instance, consider Full Page Cache¹ as an example. Some websites may take a considerable amount of time² to load a page. To enhance performance, developers may include an extra table in MySQL³, where they can save fully rendered pages. This strategy reduced the load time from 15 seconds to 100ms, which is a significant improvement that negates the need for high-speed storage solutions.

Cache definitions from Wiki and AI

There is a definition from Wikipedia.

In computing, a cache (/kæʃ/ (listen) KASH) is a hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere. A cache hit occurs when the requested data can be found in a cache, while a cache miss occurs when it cannot. Cache hits are served by reading data from the cache, which is faster than recomputing a result or reading from a slower data store; thus, the more requests that can be served from the cache, the faster the system performs.

And let’s follow trends and ask Chat-GPT what is cache.

A cache is a temporary storage area that stores frequently accessed or recently accessed data, which helps to reduce the time it takes to access that data in the future. Caches can be found at various levels in a computer system, such as within a CPU, on a hard disk, or in a web browser. They function by keeping a copy of frequently used data or instructions closer to the processor or application, which decreases the time it takes to retrieve them from their original location. Caches can be hardware or software-based and can be designed to work with various types of data. By using a cache, a system can minimize the time it takes to retrieve data from slower storage devices or networks, leading to faster overall performance and a better user experience.

Database

A software program, known as a database⁴ or a DBMS⁴, is capable of storing data and allowing users to access it for reading and writing purposes through an API⁵.

Database definitions from Wiki and AI

There is a definition from Wikipedia.

In computing, a database is an organized collection of data stored and accessed electronically. Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage. The design of databases spans formal techniques and practical considerations, including data modeling, efficient data representation and storage, query languages, security and privacy of sensitive data, and distributed computing issues, including supporting concurrent access and fault tolerance.

And let’s follow trends and ask Chat-GPT what is database.

A database is a digital collection of organized data that is stored and accessed electronically. It is designed to efficiently store and manage large amounts of information. Typically, a database consists of one or more tables, each containing rows and columns of data. A row represents a single record, while each column represents a different attribute or field of the record. Databases have a wide range of applications, including business operations, scientific research, e-commerce, and social media. To create, modify, and query the data, specialized software called database management systems (DBMS) is used.

Redis and Memcache

As stated in our definitions, caching is a methodology rather than a particular service. This means that we have the flexibility to utilize any storage solution that suits our requirements in order to construct a cache. Whether it’s MySQL, MSSQL, MongoDB, Cassandra, CockroachDB, Redis, Memcached, or simply a hash map in memory within the application, the choice is not significant as long as it meets our needs and assists us in accomplishing our goals.

Redis and Memcache are databases, as they are capable of storing data (primarily in memory) and offering an API for data access. Although they are limited to a predefined storage capacity, they may be suitable as the primary storage for our applications in specific scenarios that align with our requirements. In addition, Redis offers a range of features beyond storage, making it useful for a wide range of tasks.

Imprortance of naming

Although it may seem that naming is not crucial and we can assign any name we desire, using inaccurate names and definitions for tools may often create biases in our minds.

By enhancing our understanding and naming conventions for various technologies, we can eliminate biases from our perspective, leading to more efficient problem-solving.

Conclusion

In conclusion, it is crucial to communicate more and ask questions when in doubt. Reading documentation and technical resources can also be helpful in gaining a better understanding of a topic. By doing so, we can avoid biases that can complicate our lives. It’s essential to create a culture where asking questions is encouraged, and everyone feels comfortable enough to seek clarity. In this way, we can increase our knowledge, improve our work, and work towards a more inclusive and unbiased environment. So, let’s be curious and open to learning, and not let biases hinder our progress.

Footnotes

Full page cache is a technique when full rendered response of some service (event full website page) are saved after the first render to not rerender it for every request. ↩
I’ve seen websites that spent more than 15 seconds to render a page. ↩
MySQL is an open-source relational database management system. Link to Wikipedia. ↩
Within our topic database, there is a DBMS (Database Management System) present. However, since the term “database” is too granular and not relevant to the user experience aspect, it is not of our interest. ↩ ↩²
When we speak about DB API we mean any interface that can be used to access data, like Golang interfaces, Java interfaces, SQL, CQL, MQL, etc. ↩

First release of gotimeparser

Anton Ohorodnyk — Tue, 07 Mar 2023 06:34:35 GMT

Background

Many companies use service to service integration through different API porotocols. I had the same situation in my production environment. At some point, clients of my service started to send me dates in different formats.

Some of them sent timestamps in int, some of them used ISO 8601, etc. But what was clear, these dates were valid and we could not request to change it in one day.

As a result of this issue, I implemented some solution that solved the issue and wrote a couple of articles that shared the issue and the approach:

But I did not want to create an Open Source package, because of I believe that Open Source cannot be just few lines of code. It’s a right formatting of the code, some code coverage by tests, examples and the most important - is commitment to support it.

First release

Today is a day when I decided to invest time and commit to a simplification of software development in Go. Time parsing is a regular issue, especially in event-driven architecture and microservice arachitectures.

Feel free to use the zero-dependency library to solve all the main issues you could have with time parsing in Go.

Preallocate dynamic structures

Anton Ohorodnyk — Sat, 04 Mar 2023 03:30:35 GMT

Introduction

We live in a beautiful software development time, where every language contains dynamic structures. Therefore, we can think about only some memory allocation. Differences between development performance are significantly decreasing between statically and dynamically typed languages.

We can write code on Go, Kotlin, Java, etc., almost like we write on statically typed languages like Python, JavaScript, PHP, etc. Of course, it’s impressive, but let’s discuss some things that can help us improve performance in a small, easy step.

This short article will explain why preallocation is beneficial, even for dynamic structures.

Simplification in statically typed languages

The significant difference between statically typed languages and dynamically typed languages is mainly based on three main differences:

Memory management
Verbosity
Type conversion

Verbosity and type conversion parts will discuss in future articles. However, memory management is the topic we will discuss right now.

Memory management

When dynamic languages were involved, all static languages required co-manage memory manually. Every developer had to consider an array size, resizing, memory cleaning, etc.

Currently, every language (including C++) provides a way how to almost or entirely avoid memory management in these ways:

Reference counters
Garbage collectors

So, if we are doing right, we can use some slice¹, ArrayList, vector (similar structures from different languages), etc., to work with dynamic arrays. So, in the current world, we can append whatever amount of data we want to build a slice¹.

Example with an array:

package main

import "fmt"

func main() {
	const size = 10

	arr := [size]int{}
	fmt.Println(arr) // [0 0 0 0 0 0 0 0 0 0]

	for i := 0; i < size; i++ {
		arr[i] = 333
	}

	fmt.Println(arr) // [333 333 333 333 333 333 333 333 333 333]
}

The above example shows that we had a fixed array size where we could not add the eleventh element without some manipulations.

Example with a dynamic array data structure (slice¹ from Go):

package main

import "fmt"

func main() {
	const size = 10

	arr := []int{}
	fmt.Println(arr) // []

	for i := 0; i < size; i++ {
		arr = append(arr, 333)
	}

	fmt.Println(arr) // [333 333 333 333 333 333 333 333 333 333]
}

As we can see, we created a slice with zero size. After that, we extended it from zero elements up to 10. After that, we had no issues adding one or hundreds of more pieces.

Performance improvements

As we saw above, dynamic structures like slice¹ improve our life quality and help us write code comparably fast as on dynamic languages.

The example above provided an example where we created a slice with zero size and allocated it up to ten elements. However, there is a minor performance issue here. However, we knew the expected size (in our use case, it was ten elements), so we could preallocate the slice and improve performance.

Let’s create a small benchmark to see the difference between preallocated and non-allocated slices in Go:

package slices_test

import (
	"testing"
)

func BenchmarkNonAllocated(b *testing.B) {
	slice := []int{}
	// run the allocate variable b.N times
	for n := 0; n < b.N; n++ {
		slice = append(slice, 333)
	}
}

func BenchmarkPreAllocated(b *testing.B) {
	slice := make([]int, 0, b.N)
	// run the allocate variable b.N times
	for n := 0; n < b.N; n++ {
		slice = append(slice, 333)
	}
}

Running result:

$ go test -benchmem -bench . github.com/safigo/gox/slices -benchtime=20s
goos: darwin
goarch: arm64
pkg: github.com/safigo/gox/slices
BenchmarkNonAllocated-8         1000000000              14.52 ns/op           42 B/op          0 allocs/op
BenchmarkPreAllocated-8         1000000000               3.918 ns/op           8 B/op          0 allocs/op
PASS
ok      github.com/safigo/gox/slices        19.872s

In the result, we can see that performance improvement is ~3.7 times.

Conclusion

This article needs to describe how slices, ArrayList, or similar data structures work. Then, it’s simple to find in your favorite search engine.

Meanwhile, this article imperatively shows that preallocation significantly improves performance. In the article, we touched only dynamic arrays, but it can be applied to every type like slice¹, map (hash map), etc.

Our applications can handle millions of requests. So if we can improve performance for every request ONLY by this improvement, it will be a massive step for us.

It’s a small part of the improvement. We will discuss other options in future articles as well.

Footnotes

Some high-level internals about slices can be read here. ↩ ↩² ↩³ ↩⁴ ↩⁵

Open questions about performance

Anton Ohorodnyk — Wed, 01 Mar 2023 01:20:35 GMT

Introduction

We live in a fantastic time with cheap resources and advanced chipsets in almost all devices. Moore’s law¹ predicted a bright future with continued chip performance growth. This prediction offered us the future where we can meet our performance expectations by doing nothing in our code by making affordable investments in our hardware infrastructure.

While this approach works well for most projects, after seeing tens or even hundreds of them, I can assume that these performance-neutral decisions postpone architectural complexity.

How do we measure performance?

When we speak about the performance of any application, we can mean many different parts that affect our business flow. The specific topic we can discuss will depend on the use case we want to review and the business problem(s) that solves our application. It could be any mix of:

Network
Memory (RAM²)
Disk
CPU

There is a couple of examples of performance measurement:

CPU utilization per task³
Time execution per task
Number of tasks executed per second⁴

If you measure CPU utilization per task³, this introductory article will likely not help you because you apply all possible optimizations. But we will focus on two other types of measurement.

Time execution per task

This is a straightforward way to measure the task.

We find a couple of paths in the task and identify payloads to trigger them.
We trigger tasks by identified parameters.
We calculate the execution time of our action (without transport time):
- We can collect start and end times in nanoseconds (micro or milliseconds, depending on the performance of the task).
- We collect the difference somewhere.
Or we can calculate an execution time for an external spectator:
- For example, we can run curl with the time function.
- When the function finishes, it will show us time calculations.

$ time sleep 1

________________________________________________________
Executed in 1.01 secs fish external
 usr time 1.18 millis 0.16 millis 1.02 millis
 sys time 4.25 millis 1.88 millis 2.38 millis

Number of tasks executed per second

We will not dive deep into this topic, but we must mention some tools to help you dive into the question.

To calculate a number an RPS⁴ number, we can use one of the various available tools for that, like k6, JMeter, Apache Bench (ab), Siege, etc.

After running the chosen application with the required configuration, we will get a result with some numbers, like RPS, average request execution time, total time, number of performed requests, etc.

On-going metrics

It’s not a type of synthetic measurement but accurate data generated by real users.

One of the most critical parts that barely related to the topic, but I couldn’t avoid it. Always collect at least high-level metrics for the real processing of your applications. Use public services or Open Source solutions, but always collect high-level metrics per endpoint.

Ideally, we want to collect more detailed metrics for all incoming and outcoming IO⁵.

Why is it important to measure performance?

There are many reasons to know the performance of the application:

Predict scaling hardware investments in case of glow
Understand the limitations of an application
Find the technical issues that could be reasons to leave your service customers
- For example, response time is more than 1 second
Find places for improvement to solve
- Technical issues
- Scaling issues
- Save on hardware by decreasing the number of required resources

Complex question

When we already have a system where we want to improve performance significantly, it requires:

Deep understanding in performance analyzing: collecting metrics, profiling an application, reviewing code, etc.
Good knowledge of the product (good to have).
Understanding of how to do refactoring with minimal effect on business processes.

All applications sooner or later require these complicated steps due to the history of the application evolution. Usually, projects at this stage people name as legacy⁶. Unfortunately, it’s the unavoidable result of every project.

At the same time, we can keep writing more effective code to keep performance as good as possible to avoid premature performance issues.

The best practices

Following the best practices during everyday work could be challenging. But in reality, everything repeated many times will lead us to habits that we will apply without mental overwhelming.

We will not speak here about the actual best practices because it’s a big question for a book or a couple of books. So, all these snippets and suggestions will be highlighted in the later articles.

However, the basic knowledge of:

Computer science
Performance testing
Code simplicity
Benchmarking

It can help to postpone the tech debt. Follow the blog for the next articles for real snippets and examples.

Conclusion

This article was written as a reference for the following snippets and articles to avoid repetition. It will be a good starting point for someone. And this article could be eventually updated to be more effective. Any feedback is welcome!

Footnotes

Moore’s law is not a law. It’s an observation. And it looks like it has already stopped working. Anyway, it’s too important a part of the computer’s history to forget the name. ↩
RAM (Random-access memory) is a type of memory usually used to store temporary data related to currently running applications. ↩
By task, I mean any work to be done in the scope of the process - for example, HTTP request processing, RabbitMQ message processing, cron-job scheduled tasks, etc. ↩ ↩²
RPS or QPS (queries per second) - measurement where we calculate how many requests we can process per second in the given environment, like a node in the cloud, laptop, dedicated server, etc. ↩ ↩²
IO - input/output is the communication between an information processing system. ↩
In this article legacy word does not involve any negative connotations. It’s used here to highlight that project on the life-cycle stage where refactoring and improvements are required for future support and feature growth. ↩

Improving security in HTTPS communication

Anton Ohorodnyk — Sat, 18 Feb 2023 06:40:42 GMT

Introduction

We are living in a world where almost not left self-sufficient products. Almost every product has at least one integration into other services.

Different protocols can be used for communication between them, like REST, gRPC, GraphQL, etc. However, all of them use some network-level protocol, and usually, as a network protocol, they use HTTP¹.

HTTP¹ is a great multi-level² communication protocol. It also provides secure options such as HTTP over SSL/TLS³.

This article will touch on some high-level concepts with links as starting points to read about them. Something can be used with simplification to avoid detailed explanations, but provided links can be used to learn more details. I will use Golang as a great language for backend services.

HTTPS pros and cons

HTTPS³ provides good security that makes our internet safe and with a much better level of privacy. Some details on how HTTPS’s security works can be found here.

The protocol itself contains a couple of steps that help encryption in HTTPS be as secure as it is:

TLS handshake
Certificate verification
Encrypted communication

We will focus on step number two - Certificate Verification.

Certificate verification

Certificate verification is an important step that helps us to guarantee that we did a TLS handshake with a trusted actor without MITM⁴ attack.

When the client finishes with a TLS handshake, it still has a question, is the opponents public key belongs to an expected actor?

It’s not a simple question. Security invented certificates for public keys⁵. By having a trusted certificate⁶ we can sign the public key and verify that the certificate’s owner generated the public key.

But how to deliver a certificate, since it’s the same as a public key?

Public and private keys are secure parts, so we want to regenerate them as often as possible and keep the ability to reissue them, in case of potential compromises.

On the other hand, certificates can be distributed to a limited amount of trusted issuers that will guarantee us the security and truthfulness of all signed keys. So, the decision was to build a database of public parts of root certificates⁷.

A list of trusted certificates (root certificates) is agreed upon across the organizations and companies⁸. So they are delivered and updated from time to time. However, they are not updated so frequently, so companies and organizations can provide them to all customers in some acceptable gap during OS updates or browser updates.

The idea with the list of certificates looks fantastic. What could go wrong?

Root certificates authorities verified, and we can assume that we can trust them, but something can go wrong. There are a couple of examples from Wikipedia about Root Certificates:

DigiNotar hack of 2011 - In 2011, the Dutch certificate authority DigiNotar suffered a security breach.
China Internet Network Information Center (CNNIC) Issuance of Fake Certificates.
WoSign and StartCom: Issuing fake and backdating certificates.

Also, we can add new simplified ways to get a certificate like Let’s Encrypt, AWS, Cloudflare, GCP, etc. I do not want to say they are not secure, but they simplified how we generate certificates.

We can believe that one certificate issuer, like Cloudflare will generate exclusive certificates per domain because they can control certificates for how many domains they issue. Meanwhile, there is a small chance that a hacker can generate the certificate for the same domain from another authority.

Specific root certificate verification

The initial goal was to secure our backend<->backend integration. Let’s assume that we want to integrate with some financial institution’s API, where we will send some SPII⁹ data.

There are a few agreements:

We know that our API uses AWS.
Also, we believe that AWS will not issue two certificates for the same domain for different accounts.
We want to provide more guarantees than HTTPS for our customers.

Based on everything we have discussed and reviewed, we can assume that the solution would be used strictly ONE root certificate that belongs to our partner’s provider. In our use case, it will be AWS.

Test system’s CA certificates

First of all, we need to test. Can we download a page, https://aohorodnyk.com/? It uses Cloudflare’s certificate.

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	resp, err := http.Get("https://aohorodnyk.com/")
	if err != nil {
		log.Fatalln("Could not read the URL with error: ", err.Error())
	}

	fmt.Println("Status code:", resp.StatusCode) // Status code: 200
}

The code above works well because it uses ALL system’s CA certificates.

In the next test, let’s try to do the same for https://aws.amazon.com/. We will use the system’s CA certificates as well.

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	resp, err := http.Get("https://aws.amazon.com/")
	if err != nil {
		log.Fatalln("Could not read the URL with error: ", err.Error())
	}

	fmt.Println("Status code:", resp.StatusCode) // Status code: 200
}

Use only the AWS ROOT certificate

Before we will do that, we need to download the AmazonRootCA. For Amazon, it can be found in their repository, but for every website, it simply can be done through OpenSSL app or firefox.

We downloaded Amazon’s certificate, and it’s content:

We need to modify the previous code that will ignore ALL system’s CA certs but only use our custom’s one.

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"log"
	"net/http"
)

func main() {
	// For the certificate, formatting is important. \n can be parsed as in the example, but
	// tabs or spaces that can be added before every line by IDE will break the cert.
	caCert := `-----BEGIN CERTIFICATE-----
MIIDQTCCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG9w0BAQsF
ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6
b24gUm9vdCBDQSAxMB4XDTE1MDUyNjAwMDAwMFoXDTM4MDExNzAwMDAwMFowOTEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv
b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj
ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM
9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw
IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6
VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L
93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm
jgSubJrIqg0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0OBBYEFIQYzIU07LwMlJQuCFmcx7IQTgoIMA0GCSqGSIb3DQEBCwUA
A4IBAQCY8jdaQZChGsV2USggNiMOruYou6r4lK5IpDB/G/wkjUu0yKGX9rbxenDI
U5PMCCjjmCXPI6T53iHTfIUJrU6adTrCC2qJeHZERxhlbI1Bjjt/msv0tadQ1wUs
N+gDS63pYaACbvXy8MWy7Vu33PqUXHeeE6V/Uq2V8viTO96LXFvKWlJbYK8U90vv
o/ufQJVtMVT8QtPHRh8jrdkPSHCa2XV4cdFyQzR1bldZwgJcJmApzyMZFo6IQ6XU
5MsI+yMRQ+hDKXJioaldXgjUkK642M4UwtBV8ob2xJNDd2ZhwLnoQdeXeGADbkpy
rqXRfboQnoZsG4q5WTP468SQvvG5
-----END CERTIFICATE-----`

	rootCAs := x509.NewCertPool() // Create a new cert pool and ignore system's root certificates.

	ok := rootCAs.AppendCertsFromPEM([]byte(caCert)) // Add our custom CA cert.
	if !ok {
		log.Fatalln("Cannot append our root certificate")
	}

	// This custom client is not for production use.
	// For production, please, configure the client properly.
	httpClient := http.Client{
		Transport: &http.Transport{
			TLSClientConfig: &tls.Config{
				RootCAs: rootCAs, // Set custom root CAs.
			},
		},
	}

	resp, err := httpClient.Get("https://aws.amazon.com/")
	if err != nil {
		log.Fatalln("Could not read the URL with error: ", err.Error())
	}

	fmt.Println("Status code:", resp.StatusCode) // Status code: 200
}

And now, let’s try to fetch our domain https://aohorodnyk.com/ that uses Cloudflare’s certificate.

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"log"
	"net/http"
)

func main() {
	// For the certificate, formatting is important. \n can be parsed as in the example, but
	// tabs or spaces that can be added before every line by IDE will break the cert.
	caCert := `-----BEGIN CERTIFICATE-----
MIIDQTCCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG9w0BAQsF
ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6
b24gUm9vdCBDQSAxMB4XDTE1MDUyNjAwMDAwMFoXDTM4MDExNzAwMDAwMFowOTEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv
b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj
ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM
9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw
IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6
VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L
93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm
jgSubJrIqg0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0OBBYEFIQYzIU07LwMlJQuCFmcx7IQTgoIMA0GCSqGSIb3DQEBCwUA
A4IBAQCY8jdaQZChGsV2USggNiMOruYou6r4lK5IpDB/G/wkjUu0yKGX9rbxenDI
U5PMCCjjmCXPI6T53iHTfIUJrU6adTrCC2qJeHZERxhlbI1Bjjt/msv0tadQ1wUs
N+gDS63pYaACbvXy8MWy7Vu33PqUXHeeE6V/Uq2V8viTO96LXFvKWlJbYK8U90vv
o/ufQJVtMVT8QtPHRh8jrdkPSHCa2XV4cdFyQzR1bldZwgJcJmApzyMZFo6IQ6XU
5MsI+yMRQ+hDKXJioaldXgjUkK642M4UwtBV8ob2xJNDd2ZhwLnoQdeXeGADbkpy
rqXRfboQnoZsG4q5WTP468SQvvG5
-----END CERTIFICATE-----`

	rootCAs := x509.NewCertPool() // Create a new cert pool and ignore system's root certificates.

	ok := rootCAs.AppendCertsFromPEM([]byte(caCert)) // Add our custom CA cert.
	if !ok {
		log.Fatalln("Cannot append our root certificate")
	}

	// This custom client is not for production use.
	// For production, please, configure the client properly.
	httpClient := http.Client{
		Transport: &http.Transport{
			TLSClientConfig: &tls.Config{
				RootCAs: rootCAs, // Set custom root CAs.
			},
		},
	}

	resp, err := httpClient.Get("https://aohorodnyk.com/")
	if err != nil {
		log.Fatalln("Could not read the URL with error: ", err.Error()) // Could not read the URL with error: Get "https://aohorodnyk.com/": x509: certificate signed by unknown authority.
	}

	fmt.Println("Status code:", resp.StatusCode)
}

If will change the domain to google.com, the error will be the same Could not read the URL with error: Get "https://google.com/": x509: certificate signed by unknown authority.

Conclusion

Small steps can help us to improve the security of services. Most crucial is that these improvements do not require vast time investments. So, let’s make the web more secure!

I hope this article will motivate people to pay attention to these small details.

Footnotes

Hypertext Transfer Protocol (HTTP). ↩ ↩²
HTTP supports headers in requests and responses that help to keep an additional level of abstraction of internal protocols, for example Accept mime headers. ↩
HyperText Transfer Protocol Secure (HTTPS). ↩ ↩²
MITM (man-in-the-middle) is the type of attack where someone can read and/or change traffic transferred between two parties. ↩
Certificate for public keys provides a way to verify that a specific private certificate signed public key. Details are here. ↩
Public information is the same as a public key. It can be published anywhere without any risks to the owner. ↩
The first-level certificate is secured as best as possible and used only by an authority to sign domain certificates. Details are here. ↩
As an example, we can see the list of certificates trusted by Apple for all their products, like iOS, MacOS, TvOS, etc. Other organizations and companies can be simply found through your favorite search engine. ↩
SPII (Sensitive Personally Identifiable Information) like credit card data, SSN, passport data, etc. ↩

Universal time UnmarshalJSON implementation

Anton Ohorodnyk — Sun, 08 May 2022 20:01:24 GMT

Introduction

We have done with two complecated steps by writing the parsers for different date time formats:

The next and final step will be to write a real world function that solves the real and actual problem. We need to parse the JSON field from different formats with different data types.

I will go though couple of investigation steps before we will write the real code.

I’ll use Golang for all code examples, but you can use any other languages base on the provided algorithm.

Task overview

Current task is to write a Golang type that can be used in JSON parsing pipeline. This field should support couple of use cases that we already reviewed in previous articles.

We received a JSON field with:

Numeric types and we expect to parse it as a timestamp.
String type and we expect to parse it as a timestamp.
String type and we expect to parse it as a some layout from the provided list.

Expectations

Based on the above use cases, we expect to see the following list of JSON fields:

{
  "num_seconds": 1651808102,
  "num_milliseconds": 1651808102363,
  "num_microseconds": 1651808102363368,
  "num_nanoseconds": 1651808102363368423,

  "hex_seconds": "0x62749766",

  "str_seconds": "1651808102",
  "str_milliseconds": "1651808102363",
  "str_microseconds": "1651808102363368",
  "str_nanoseconds": "1651808102363368423",

  "str_rfc3339": "2022-05-06T03:35:02Z",
  "str_rfc3339_nano": "2022-05-06T03:35:02.363368423Z",
  "str_rfc1123": "Fri, 06 May 2022 03:35:02 UTC",
  "str_rfc850": "Friday, 06-May-22 03:35:02 UTC",
  "str_rfc822": "06 May 22 03:35 UTC"
}

Obviously, it’s not full list of types, but as we already tested comprehensive list of cases before, we can simplify it right now.

We added hex seconds field, just in to see it works as well, because of the magic of strconv.ParseInt function.

UnmarshalJSON interface overview

Before the implementation we need to consider the Golang way of custom JSON data types implementation.

Golang contains the interface Unmarshaler with one method UnmarshalJSON([]byte) error. It means that we need to implement only one method to parse the JSON field.

There is an example of the simple implementation of the interface where we just need to separate the string with a value separated by underscore _ in a JSON field to 2 variables:

package main

import (
  "encoding/json"
  "errors"
  "strings"
)

type unmarshaled struct {
  part1 string
  part2 string
}

func (u *unmarshaled) UnmarshalJSON(text []byte) error {
  str := string(text)
  parts := strings.Split(str, "_")
  if len(parts) != 2 {
    return errors.New("invalid format")
  }

  u.part1 = parts[0]
  u.part2 = parts[1]

  return nil
}

type jsonType struct {
  T unmarshaled `json:"t"`
}

func main() {
  var jt jsonType

  json1 := `{"t": "one_two"}`

  err := json.Unmarshal([]byte(json1), &jt)
  if err != nil {
    panic(err)
  }

  println(jt.T.part1, jt.T.part2)

  json2 := `{"t": "onetwo"}`

  err = json.Unmarshal([]byte(json2), &jt)
  if err != nil {
    panic(err)
  }

  println(jt.T.part1, jt.T.part2)
}

Output:

"one two"
panic: invalid format

goroutine 1 [running]:
main.main()
        /Users/aohorodnyk/projects/anton.ohorodnyk.name/main.go:47 +0x13c
exit status 2

The first input parsed correctly, but the second one failed.

Implementation

The final implementation will contain the following steps:

Check if the JSON field is a numeric, because of it’s the fastest condition.
1. If it’s a numeric, we can parse it as a timestamp.
2. Use the strconv.ParseInt function to parse it.
3. The strconv.ParseInt function will parse the type based on the prefix of the string. If the prefix is 0x, it will be parsed as a hexadecimal number. 0b as a binary number, etc. By default it will be parsed as a decimal number.
4. If we can’t prase the numeric value as an integer, then return the error, since we do not know waht to do with the value.
5. Otherwise set the time and finish the processing.
Otherwise the value is a string. For strings we need to trim quotes to make the value a plain string.
Try to parse the string with all supported layouts.
If we successfuly parsea the string with a some listed layouts, then set the time and finish the processing.
If we did not find the needed layout, try to parse the string as a timestamp.
If we successfuly parse the string as a timestamp, then set the time and finish the processing.
Otherwise return the error.

The following code example implements the algorithm described above.

package main

import (
  "encoding/json"
  "errors"
  "fmt"
  "math"
  "strconv"
  "strings"
  "time"
)

// List of supported time layouts.
var formats = []string{
  time.RFC3339Nano,
  time.RFC3339,
  time.RFC1123Z,
  time.RFC1123,
  time.RFC850,
  time.RFC822Z,
  time.RFC822,
  time.Layout,
  time.RubyDate,
  time.UnixDate,
  time.ANSIC,
  time.StampNano,
  time.StampMicro,
  time.StampMilli,
  time.Stamp,
  time.Kitchen,
}

const (
  maxNanoseconds  = int64(math.MaxInt64)
  maxMicroseconds = int64(maxNanoseconds / 1000)
  maxMilliseconds = int64(maxMicroseconds / 1000)
  maxSeconds      = int64(maxMilliseconds / 1000)

  minNanoseconds  = int64(math.MinInt64)
  minMicroseconds = int64(minNanoseconds / 1000)
  minMilliseconds = int64(minMicroseconds / 1000)
  minSeconds      = int64(minMilliseconds / 1000)
)

type InternalTime struct {
  time.Time
}

func (it *InternalTime) UnmarshalJSON(data []byte) error {
  // Make sure that the input is not empty
  if len(data) == 0 {
    return errors.New("empty value is not supported")
  }

  // If the input is not a string, try to parse it as a number, otherwise return an error.
  if data[0] != '"' {
    timeInt64, err := strconv.ParseInt(string(data), 0, 64)
    if err != nil {
      return err
    }

    it.Time = parseTimestamp(timeInt64)
  }

  // If the input is a string, trim quotes.
  str := strings.Trim(string(data), `"`)

  // Parse the string as a time using the supported layouts.
  parsed, err := parseTime(formats, str)
  if err == nil {
    it.Time = parsed

    return nil
  }

  // As the final attempt, try to parse the string as a timestamp.
  timeInt64, err := strconv.ParseInt(str, 0, 64)
  if err == nil {
    it.Time = parseTimestamp(timeInt64)

    return nil
  }

  return errors.New("Unsupported time format")
}

type jsonType struct {
  NumSeconds      InternalTime `json:"num_seconds"`
  NumMilliseconds InternalTime `json:"num_milliseconds"`
  NumMicroseconds InternalTime `json:"num_microseconds"`
  NumNanoseconds  InternalTime `json:"num_nanoseconds"`

  HexSeconds InternalTime `json:"hex_seconds"`

  StrSeconds      InternalTime `json:"str_seconds"`
  StrMilliseconds InternalTime `json:"str_milliseconds"`
  StrMicroseconds InternalTime `json:"str_microseconds"`
  StrNanoseconds  InternalTime `json:"str_nanoseconds"`

  StrRFC3339     InternalTime `json:"str_rfc3339"`
  StrRFC3339Nano InternalTime `json:"str_rfc3339_nano"`
  StrRFC1123     InternalTime `json:"str_rfc1123"`
  StrRFC850      InternalTime `json:"str_rfc850"`
  StrRFC822      InternalTime `json:"str_rfc822"`
}

func main() {
  var jt jsonType

  json1 := `{
    "num_seconds": 1651808102,
    "num_milliseconds": 1651808102363,
    "num_microseconds": 1651808102363368,
    "num_nanoseconds": 1651808102363368423,

    "hex_seconds": "0x62749766",

    "str_seconds": "1651808102",
    "str_milliseconds": "1651808102363",
    "str_microseconds": "1651808102363368",
    "str_nanoseconds": "1651808102363368423",

    "str_rfc3339": "2022-05-06T03:35:02Z",
    "str_rfc3339_nano": "2022-05-06T03:35:02.363368423Z",
    "str_rfc1123": "Fri, 06 May 2022 03:35:02 UTC",
    "str_rfc850": "Friday, 06-May-22 03:35:02 UTC",
    "str_rfc822": "06 May 22 03:35 UTC"
  }`

  err := json.Unmarshal([]byte(json1), &jt)
  if err != nil {
    panic(err)
  }

  fmt.Println(jt) // {2022-05-05 20:35:02 -0700 PDT 2022-05-05 20:35:02.363 -0700 PDT 2022-05-05 20:35:02.363368 -0700 PDT 2022-05-05 20:35:02.363368423 -0700 PDT 2022-05-05 20:35:02 -0700 PDT 2022-05-05 20:35:02 -0700 PDT 2022-05-05 20:35:02.363 -0700 PDT 2022-05-05 20:35:02.363368 -0700 PDT 2022-05-05 20:35:02.363368423 -0700 PDT 2022-05-06 03:35:02 +0000 UTC 2022-05-06 03:35:02.363368423 +0000 UTC 2022-05-06 03:35:02 +0000 UTC 2022-05-06 03:35:02 +0000 UTC 2022-05-06 03:35:00 +0000 UTC}
}

func parseTimestamp(timestamp int64) time.Time {
  switch {
  case timestamp < minMicroseconds:
    return time.Unix(0, timestamp) // Before 1970 in nanoseconds.
  case timestamp < minMilliseconds:
    return time.Unix(0, timestamp*int64(time.Microsecond)) // Before 1970 in microseconds.
  case timestamp < minSeconds:
    return time.Unix(0, timestamp*int64(time.Millisecond)) // Before 1970 in milliseconds.
  case timestamp < 0:
    return time.Unix(timestamp, 0) // Before 1970 in seconds.
  case timestamp < maxSeconds:
    return time.Unix(timestamp, 0) // After 1970 in seconds.
  case timestamp < maxMilliseconds:
    return time.Unix(0, timestamp*int64(time.Millisecond)) // After 1970 in milliseconds.
  case timestamp < maxMicroseconds:
    return time.Unix(0, timestamp*int64(time.Microsecond)) // After 1970 in microseconds.
  }

  return time.Unix(0, timestamp) // After 1970 in nanoseconds.
}

func parseTime(formats []string, dt string) (time.Time, error) {
  for _, format := range formats {
    parsedTime, err := time.Parse(format, dt)
    if err == nil {
      return parsedTime, nil
    }
  }

  return time.Time{}, fmt.Errorf("could not parse time: %s", dt)
}

The above code covers positive test cases and proves the algorithm works without any issues and implementation is correct.

Conclusion

In the latest three articles we implemented the universal time parser that can be used to improve clients’ developer experience. It can help to decrease amount of production bugs and errors.

I hope these articles will be helpful to you and your team.

Fill free to ask any question you might have.

Parse time from different non timestamp formats

Anton Ohorodnyk — Sun, 08 May 2022 03:48:20 GMT

Introduction

In previous article, we implemented the efficient solution to parse timestamps in different timestamp formats (seconds, milliseconds, microseconds, nanoseconds).

As I mentioned in the previous article, we can receive a string formatted date time from a client, when we expected string timestamp in seconds.

Obviously we hardcoded some solution to support the specific format on a server (as we are expecting to find this issue on production with a production client), but we still would like to make it more universan and forget about the problem for a while.

Let’s implement the solution!

I’ll use Golang for all code examples, but you can use any other languages base on the provided algorithm.

Issue overview

Before we will start writing code, we need to understand the issue.

Expectations

To simplify current task, let’s assume that we have a string field and we need to write a function that will support a multiple time formats.

There are our expectations:

We have a string field where we expect to receive a RFC339 date time information.
Our expectation is to see the timestamp in a range of dates and times between current time - 3 days and current time + 3 days.
We use a language with strict type checking. We accept ONLY string value in our JSON field.

Issue identification

What do we unexpectedly see in our logs:

In the log we have found an error like 0001-01-01 00:00:00 +0000 UTC parsing time "Sat, 07 May 2022 19:22:10 PDT" as "2006-01-02 15:04:05": cannot parse "Sat, 07 May 2022 19:22:10 PDT" as "2006"
And also we have found this type of errors 0001-01-01 00:00:00 +0000 UTC parsing time "05/07 07:22:54PM '22 -0700" as "2006-01-02 15:04:05": cannot parse "7 07:22:54PM '22 -0700" as "2006"

Possible wrong formats

After some investigation we found the following formats used by clients:

time.Layout from Golang -> 01/02 03:04:05PM '06 -0700
RFC1123 with a format Mon, 02 Jan 2006 15:04:05 MST

It means that some our clients uses wrong standardized formats, instead of expected RFC3339. As well with identified formats we assumed that clients could use some more different formats and we need to communicate with all our client teams or partners to identify all possible formats.

Issue solving steps

Based on our investigation to fix the issue for the most future cases we need to do these fixes:

Support the most popular formats.
Design the extensible API to support for additional formats.
Try do not overcomplicate the solution, for this article.

Known limitations

Surely this function will not support all possible formats, so, we will need to extend it with more formats, when needed.
We will use only formats from the predefined list in standard Golang library.
There are possible conflicts between different formats. The order of provided formats is important.

Solution

Before we start writing code, we need to do some preparations.

Understand on how to parse the date time in Golang.
Find the list of supported formats in standard Golang library.
Implement the function.

How to parse the date time in Golang

Fortunately, Golang supports pretty simple API to parse the date time from a string in different formats. However, It has unusual layout’s format.

To parse the date time, we need to use time.Parse function.

package main

import (
  "fmt"
  "time"
)

func main() {
  now := time.Now()                                                 // Get current date time.
  parsed, err := time.Parse(time.RFC3339, now.Format(time.RFC3339)) // Parse current date time in RFC3339 format.
  fmt.Println(parsed, err)                                          // 2022-05-07 20:07:11 -0700 PDT <nil>
}

In the example above, we parse the current date time in RFC3339 format that contains the layout 2006-01-02T15:04:05Z07:00. As we can see, in golang we use the specific time date as a layout to specify the format that we will parse. It’s unusual, but since we are not going to write our own formats, the task will not require detailed knowledge of these internal implementations.

As we can see, there are two different parameters returned from time.Parse function: time and error. If error is nil, then we sucessfully parsed the date time, otherwise something went wrong.

Let’s see an example with different formats:

package main

import (
  "fmt"
  "time"
)

func main() {
  now := time.Now()                                                 // Get current date time.
  parsed, err := time.Parse(time.RFC3339, now.Format(time.RFC1123)) // Parse current date time in RFC3339 format.
  fmt.Println(parsed, err)                                          // 0001-01-01 00:00:00 +0000 UTC parsing time "Sat, 07 May 2022 20:09:12 PDT" as "2006-01-02T15:04:05Z07:00": cannot parse "Sat, 07 May 2022 20:09:12 PDT" as "2006".
}

As we can see in the output, we got an empty date time 0001-01-01 00:00:00 +0000 UTC and an error parsing time "Sat, 07 May 2022 20:09:12 PDT" as "2006-01-02T15:04:05Z07:00": cannot parse "Sat, 07 May 2022 20:09:12 PDT" as "2006" about wrong format.

We will use this behavior to parse the date time in different formats.

List of supported formats

As I mentioned before, we need to support all formats from the standard Golang library.

All supported formats are listed in the time package of the standard library. We can find it in the documentation.

I’ll list all supported formats in the current latest Golang version:

package time

const (
  Layout      = "01/02 03:04:05PM '06 -0700" // The reference time, in numerical order.
  ANSIC       = "Mon Jan _2 15:04:05 2006"
  UnixDate    = "Mon Jan _2 15:04:05 MST 2006"
  RubyDate    = "Mon Jan 02 15:04:05 -0700 2006"
  RFC822      = "02 Jan 06 15:04 MST"
  RFC822Z     = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
  RFC850      = "Monday, 02-Jan-06 15:04:05 MST"
  RFC1123     = "Mon, 02 Jan 2006 15:04:05 MST"
  RFC1123Z    = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
  RFC3339     = "2006-01-02T15:04:05Z07:00"
  RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
  Kitchen     = "3:04PM"
  // Handy time stamps.
  Stamp      = "Jan _2 15:04:05"
  StampMilli = "Jan _2 15:04:05.000"
  StampMicro = "Jan _2 15:04:05.000000"
  StampNano  = "Jan _2 15:04:05.000000000"
)

Efficient solution

For implementation we will use the simplest possible algorithm:

Iterate over all supported formats.
Parse the receved string with the current format.
If the returned error is equal to nil, then we found the correct format. We can return from the function.
Otherwise, we need to continue the iteration.

The intermediate solution:

package main

import (
  "fmt"
  "time"
)

func main() {
  t := time.Date(2022, time.May, 23, 7, 3, 5, 734423, time.UTC)

  fmt.Println(parseTime(t.Format(time.RFC3339))) // 2022-05-23 07:03:05 +0000 UTC <nil>
  fmt.Println(parseTime(t.Format(time.RFC1123))) // 2022-05-23 07:03:05 +0000 UTC <nil>
  fmt.Println(parseTime(t.Format(time.Layout)))  // 0001-01-01 00:00:00 +0000 UTC could not parse time: 05/23 07:03:05AM '22 +0000
}

func parseTime(dt string) (time.Time, error) {
  var formats = []string{
    time.RFC3339,
    time.RFC1123,
  }

  for _, format := range formats {
    parsedTime, err := time.Parse(format, dt)
    if err == nil {
      return parsedTime, nil
    }
  }

  return time.Time{}, fmt.Errorf("could not parse time: %s", dt)
}

In the code with the example we iterated over two formats: RFC3339 and RFC1123. We wrote the test where we tried to parse three different formats with the same date time string. As we can see both supported formats are parsed correctly.

There is the limitation that we can recognize based on the above example: we can’t parse sequantly formats that are sub-formats of the next format. So, we need to be sure that we provide time layouts in the list formats from the most specific to the most general.

Implementing the function that will cover all requirements we already specified:

package main

import (
  "fmt"
  "time"
)

func main() {
  // Specify all formats in the specific order.
  formats := []string{
    time.RFC3339Nano,
    time.RFC3339,
    time.RFC1123Z,
    time.RFC1123,
    time.RFC850,
    time.RFC822Z,
    time.RFC822,
    time.Layout,
    time.RubyDate,
    time.UnixDate,
    time.ANSIC,
    time.StampNano,
    time.StampMicro,
    time.StampMilli,
    time.Stamp,
    time.Kitchen,
  }

  t := time.Date(2022, time.May, 23, 7, 3, 5, 234734423, time.UTC)

  fmt.Println(parseTime(formats, t.Format(time.RFC3339Nano))) // 2022-05-23 07:03:05.234734423 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.RFC3339)))     // 2022-05-23 07:03:05 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.RFC1123Z)))    // 2022-05-23 07:03:05 +0000 +0000 <nil>
  fmt.Println(parseTime(formats, t.Format(time.RFC1123)))     // 2022-05-23 07:03:05 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.RFC850)))      // 2022-05-23 07:03:05 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.RFC822Z)))     // 2022-05-23 07:03:00 +0000 +0000 <nil>
  fmt.Println(parseTime(formats, t.Format(time.RFC822)))      // 2022-05-23 07:03:00 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.Layout)))      // 2022-05-23 07:03:05 +0000 +0000 <nil>
  fmt.Println(parseTime(formats, t.Format(time.RubyDate)))    // 2022-05-23 07:03:05 +0000 +0000 <nil>
  fmt.Println(parseTime(formats, t.Format(time.UnixDate)))    // 2022-05-23 07:03:00 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.ANSIC)))       // 2022-05-23 07:03:00 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.StampNano)))   // 2022-05-23 07:03:05.234734423 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.StampMicro)))  // 0000-05-23 07:03:05.234734 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.StampMilli)))  // 0000-05-23 07:03:05.234 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.Stamp)))       // 0000-05-23 07:03:05 +0000 UTC <nil>
  fmt.Println(parseTime(formats, t.Format(time.Kitchen)))     // 0000-01-01 07:03:00 +0000 UTC <nil>
}

func parseTime(formats []string, dt string) (time.Time, error) {
  for _, format := range formats {
    parsedTime, err := time.Parse(format, dt)
    if err == nil {
      return parsedTime, nil
    }
  }

  return time.Time{}, fmt.Errorf("could not parse time: %s", dt)
}

Usually we see the implementation with all possible test cases to be sure that our algorithm is correct and it supports all required formats.

Looks like all formats were parsed without any issues and we will assume that this algorithm is correct and usable for our application.

Conclusion

In this article we solved one more issue related to parsing date time string. Now we can assume that our clients are covered and safe with all predefined formats.

Current implementation can be used without any restrictions in all your projects. But, pay more attention that you covered all required formats by your business. Probably, you will need to add more formats to the list.

I hope this article will save some time for you.

Parse timestamp formats

Anton Ohorodnyk — Fri, 06 May 2022 03:46:23 GMT

Introduction

Have you ever had a situation when different clients send milliseconds in requests to a back-end that expects timestamp in seconds?

If not, you are lucky person. Unfortunately, it’s so popular issue in my experience. I even cannot count a number of bugs related to the specific issue.

Every time we find this issue on a client, we need to find the best on how to solve it. If a front-end is web-site, it’s littery not an issue and can be simply fixed and redeliver to all users in short period of time. But, if the front-end is a client application, it could be much more complicated to redeliver the fix to all clients in some predictable period of time.

In my point of view the best way to fix the issue in this specific situation is to fix everything on backend and support two different types of payload. We have predictable time expectations when we can fix the issue, we know how can we deliver the fix to ALL users, etc.

In the current example when we expect to receive the unix timestamp in seconds, but receive it in milliseconds. We can fix it by adding support for both possible values to our parser.

Moreover, we know about the situations when a client sets the RFC3339 as a string field type, instead of int with a timestamp.

So, we have fixed the issues for our production, let’s relax. But, as we discussed, this issue have happened multiple time, let’s try to think about the solution to prevent this issue in future?

I’ll use Golang for all code examples, but you can use any other language base on the provided algorithm.

Issue overview

Before we will start writing code, we need to understand the issue.

Expectations

There are our expectations:

We have an int64 field where we expect to receive the unix timestamp in seconds.
We expect to see always possitive int numbers in this field.
Our expectation is to see the timestamp in a range of dates and times between current time - 3 days and current time + 3 days.
We use a language with strict type checking. We accept ONLY int value in our JSON field.

Issue identification

What do we unexpectedly see in our stored data or logs:

Instead of 2022-05-05T10:43:27-07:00 we see 54312-08-04T05:24:35-07:00 in our storage.
Instead of processed requests we see some errors in logs that JSON could not be parsed, because of string provided in a field, instead of expected int.

Possible wrong formats

After some investigation we found that the both issues are related to the following wrong values:

We see in the storage the future date 54313-06-15T01:03:45Z, instead of the current date 2022-05-06T01:16:23Z, because of the client used milliseconds instead of seconds during timestamp generation.
We see in logs an error that JSON could not be parsed, because of string provided in a field, instead of expected int, because of the client sent us some string value in the field.
1. The client set the timestamp in a string field, instead of int, during JSON generation.
2. The client sent us the date time in an RFC3339 format, instead of unix timestamp in seconds.
3. The client used some other date time format to send us the date time.

Issue solving steps

Based on our investigation to fix the issue for the most future cases we need to do these fixes:

Add support of all possible values for our timestamp field: seconds, milliseconds, microseconds, nanoseconds.
Add support of the timestamp field in a string type (we will not solve it in this article).
Add support of the date time formats like: RFC3339, RFC3339Nano, etc (we will not solve it in this article).

Known limitations

In this document we will use Golang language for all code examples.
We will solve the issue with write the functions that will parse .
As we are fixing the specific use case with a timestamp, we are not going to support all range of possible date times.
1. We will surely support the range between 1970-04-17T18:02:52Z and 2262-04-11T23:47:16Z years.
2. If you need dates after 2262-04-11T23:47:16Z year or before 1970-04-17T18:02:52Z, you will need to adjust the solution for your specific use case.
3. Negative values are supported for the timestamp. But they have the same restrictions as positive values. The supported range for the negative values betwee 1969-09-15T22:57:07.963145225-07:00 and 1677-09-20T16:19:45.145224192-07:52.
We will solve only timestamp parsing issue, without string dates parsing.

Looks like we are done with the issue formalization, let’s move to the implementation side.

Solution

To write the code we need to understand on how the timestamp works with different time formats. What can we use and how to identify the edges of the time range?

First of all let’s try to understand on how timestamp for different time formats works.

Timestamp

Timestamp is a number of seconds since January 1, 1970 UTC. To use more granular time we just add more digits to the timestamp. For example, to store a timestamp in milliseconds we add 3 digits to the timestamp (multiple seconds to 1000 to get milliseconds).

All modern implementation for the timestamp format uses int64 type, to solve the issue of 2038 year’s problem¹. Tht meximum date for int64 is 292277026596-12-04T15:30:07Z. We can assume that we will never met this limit.

To collect data in milliseconds, microseconds and nanoseconds we uses int64 as well in many cases. The max date time for int64 is 2262-04-11T23:47:16Z. It’s too far in the future to be a problem as well.

Golang introduction

So, now we know how the timestamp works, but what the next?

Before we will start, let’s see on how to parse dates in Golang, if we are going to work with this language in the further steps.

In Golang we use the time package to parse dates and times.

To convert int64 timestamp in seconds to time.Time we use the time.Unix(int64, 0) function. Let’s convert int64 timestamp in seconds to time.Time and print it in RFC3339 format.

package main

import "time"

func main() {
  t := time.Unix(1651804700, 0) // Convert inte64 timestamp in seconds to time.Time
  str := t.Format(time.RFC3339) // Convert time.Time to RFC3339 format in string.
  println(str)                  // 2022-05-05T19:38:20-07:00

Now let’s see how to print the same timestamp in nanoseconds.

package main

import "time"

func main() {
  nanosecondMultiplier := int64(time.Second)
  t := time.Unix(0, 1651804700*nanosecondMultiplier) // Convert inte64 timestamp in seconds to time.Time
  str := t.Format(time.RFC3339)                      // Convert time.Time to RFC3339 format in string.
  println(str)                                       // 2022-05-05T19:38:20-07:00

All timestamps in microseconds and milliseconds we will convert to nanoseconds and use the sacond example to parse it.

Identify the edges of the time range

We already know how to parse the dates, but the question, how to identify the edges of the time range?

The simplest solution would be just to parse the timestamp in a loop and check the timeframes to identify the right date time. We will parse numbers as nanoseconds and if the number less than the magic year (let’s choose 1980), then divide it by 1000 and try again. Let’s see the code:

package main

import "time"

func main() {
  timestamp := int64(1651804700000) // timestamp in milliseconds.
  for timestamp > 0 { // If int will overflow, we will get 0.
    t := time.Unix(0, timestamp) // Try to parse the timestamp in nanoseconds.
    if t.Year() > 1980 && t.Year() < 2100 {
      println(t.Format(time.RFC3339)) // Print the timestamp in RFC3339 format.

      return // Exit the loop.
    }

    timestamp *= 1000 // Try to convert to the next level (seconds -> milliseconds -> microseconds -> nanoseconds) and try again.
  }
}

It will work, but this algorithm is not the most efficient, because of:

It can go overflow.
We do multiple covertions int to time.
We perform multiple calculations.
We need to choose more accurate magic years.

Efficient solution

First of all, instead of multiple calculations we can precalculate the most efficient timestamps and just verify with them. And if we find the required range, do one time calculation.

To find the most efficient timestamps, let’s try to find the maximum year we can effort with the most granular format. Our the most granular format is nanoseconds. Let’s see:

package main

import (
  "time"
  "math"
)

func main() {
  timestamp := int64(math.MaxInt64) // Maximum available timestamp.
  t := time.Unix(0, timestamp) // Parse the maximum timestamp.
  println(t.Format(time.RFC3339)) // 2262-04-11T23:47:16Z
  println(timestamp) // 9223372036854775807
}

As we can see the maximum available int64 timestamp is 9223372036854775807 and it equals to 2262-04-11T23:47:16Z date time.

Based on our previous knowledge if we will devide this number by 1000 we will get the next less granular format. Let’s see it in the example:

package main

import (
  "math"
  "time"
)

func main() {
  nano := int64(math.MaxInt64)
  micro := int64(nano / 1000)
  milli := int64(micro / 1000)
  sec := int64(milli / 1000)

  // Print all timestamps.
  println("seconds: ", sec)        // seconds:  9223372036
  println("milliseconds: ", milli) // milliseconds:  9223372036854
  println("microseconds: ", micro) // microseconds:  9223372036854775
  println("nanoseconds: ", nano)   // nanoseconds:  9223372036854775807

  // Parse seconds.
  println(time.Unix(sec, 0).UTC().Format(time.RFC3339)) // 2262-04-11T23:47:16Z
  // Parse seconds by converting it to nanoseconds.
  println(time.Unix(0, sec*int64(time.Second)).UTC().Format(time.RFC3339)) // 2262-04-11T23:47:16Z
  // Let's see what time will we see if we try to parse the seconds as milliseconds.
  println(time.Unix(0, sec*int64(time.Millisecond)).UTC().Format(time.RFC3339)) // 1970-04-17T18:02:52Z
  // Parse milliseconds.
  println(time.Unix(0, milli*int64(time.Millisecond)).UTC().Format(time.RFC3339)) // 2262-04-11T23:47:16Z
  // Let's see what time will we see if we try to parse the milliseconds as microseconds.
  println(time.Unix(0, milli*int64(time.Microsecond)).UTC().Format(time.RFC3339)) // 1970-04-17T18:02:52Z
  // Parse microseconds.
  println(time.Unix(0, micro*int64(time.Microsecond)).UTC().Format(time.RFC3339)) // 2262-04-11T23:47:16Z// Let's see what time will we see if we try to parse the microseconds as nanoseconds.
  println(time.Unix(0, micro).UTC().Format(time.RFC3339)) // 1970-04-17T18:02:52Z
  // Parse nanoseconds.
  println(time.Unix(0, nano).UTC().Format(time.RFC3339)) // 2262-04-11T23:47:16Z
}

With the negative values we will follow the same logic, but we will start from the math.MinInt64 and go down. In the next code snippet with the solution we will see the actual implementation.

As we can see the simple and pretty efficient solution is to just base max timestamps on max int64. Let’s write the function that will efficiently parse the timestamps independent of the format.

package main

import (
  "math"
  "time"
)

const (
  maxNanoseconds  = int64(math.MaxInt64)
  maxMicroseconds = int64(maxNanoseconds / 1000)
  maxMilliseconds = int64(maxMicroseconds / 1000)
  maxSeconds      = int64(maxMilliseconds / 1000)

  minNanoseconds  = int64(math.MinInt64)
  minMicroseconds = int64(minNanoseconds / 1000)
  minMilliseconds = int64(minMicroseconds / 1000)
  minSeconds      = int64(minMilliseconds / 1000)
)

func main() {
  // Positive time.
  now := time.Date(2022, time.May, 06, 03, 35, 02, 363368423, time.UTC)
  sec := now.Unix()
  rfc := parseTimestamp(sec).UTC().Format(time.RFC3339)
  println(sec) // 1651808102
  println(rfc) // 2022-05-06T03:35:02Z

  millisec := now.UnixMilli()
  rfc = parseTimestamp(millisec).UTC().Format(time.RFC3339)
  println(millisec) // 1651808102363
  println(rfc)      // 2022-05-06T03:35:02Z

  microsec := now.UnixMicro()
  rfc = parseTimestamp(microsec).UTC().Format(time.RFC3339)
  println(microsec) // 1651808102363368
  println(rfc)      // 2022-05-06T03:35:02Z

  nanosec := now.UnixNano()
  rfc = parseTimestamp(nanosec).UTC().Format(time.RFC3339)
  println(nanosec) // 1651808102363368423
  println(rfc)     // 2022-05-06T03:35:02Z

  // Negative time.
  now = time.Date(1830, time.May, 06, 03, 35, 02, 363368423, time.UTC)
  sec = now.Unix()
  rfc = parseTimestamp(sec).UTC().Format(time.RFC3339)
  println(sec) // -4407164698
  println(rfc) // 1830-05-06T03:35:02Z

  millisec = now.UnixMilli()
  rfc = parseTimestamp(millisec).UTC().Format(time.RFC3339)
  println(millisec) // -4407164697637
  println(rfc)      // 1830-05-06T03:35:02Z

  microsec = now.UnixMicro()
  rfc = parseTimestamp(microsec).UTC().Format(time.RFC3339)
  println(microsec) // -4407164697636632
  println(rfc)      // 1830-05-06T03:35:02Z

  nanosec = now.UnixNano()
  rfc = parseTimestamp(nanosec).UTC().Format(time.RFC3339)
  println(nanosec) // -4407164697636631577
  println(rfc)     // 1830-05-06T03:35:02Z
}

func parseTimestamp(timestamp int64) time.Time {
  switch {
  case timestamp < minMicroseconds:
    return time.Unix(0, timestamp) // Before 1970 in nanoseconds.
  case timestamp < minMilliseconds:
    return time.Unix(0, timestamp*int64(time.Microsecond)) // Before 1970 in microseconds.
  case timestamp < minSeconds:
    return time.Unix(0, timestamp*int64(time.Millisecond)) // Before 1970 in milliseconds.
  case timestamp < 0:
    return time.Unix(timestamp, 0) // Before 1970 in seconds.
  case timestamp < maxSeconds:
    return time.Unix(timestamp, 0) // After 1970 in seconds.
  case timestamp < maxMilliseconds:
    return time.Unix(0, timestamp*int64(time.Millisecond)) // After 1970 in milliseconds.
  case timestamp < maxMicroseconds:
    return time.Unix(0, timestamp*int64(time.Microsecond)) // After 1970 in microseconds.
  }

  return time.Unix(0, timestamp) // After 1970 in nanoseconds.
}

Conclusion

In this article we solved a real-world problem, surely affected more than one person. I hope this solution will help someone to improve their product and improve user-experience of their API.

Do not forget, that every complicated problem can be solved with a simple and efficient² solution.

Footnotes

Year 2038 is the problem when the max value of int32 in seconds will be reached on 2038-01-19T03:14:07Z. ↩
By efficient I mean that the solution has efficient cpu and memory consumption. It will help to save money, improve user and developer experience as well as saving the world by decreasing electrycity consumption. ↩

Kafka as a message queue

Anton Ohorodnyk — Tue, 27 Jul 2021 17:00:00 GMT

Introduction

Small but crucial part for people who wants to choose Kafka as a message queue

Kafka is an excellent tool for data processing in streaming mode. It can help build great tools, especially when analyzing some streaming data (like user analytics, some events, etc. Moreover, Kafka as storage with Kafka streams will help process even some data with “big data”.

However, this article will describe only one use case where Kafka can be used.

This task has the name “message queue”. It is an essential part of every project I was involved in. People try to solve this task in different ways:

Use a designed tool for that (like NSQ, RabbitMQ, Gearman, etc.)
Build message queue on the top of their DB (PostgreSQL, MySQL, Redis, etc.)
Try to delegate all tasks required message queue to external services, to avoid an implementation

Kafka pretends to be used as a designed tool for this specific task and provides incredible guarantees. Therefore, we will review only this part. I hope it will help us to understand Kafka is the right tool for the next project.

Also, we will skip the partitions¹ part.

Message queue (MSQ)

Before reviewing the implementation of the pattern in Kafka, let’s overview it is in general.

Simple message processing usually splits into two different parts:

Producer (who sends a message to a queue)
Consumer (who consumes the message from the queue)

Suppose we want to guarantee that message was processed at least once. In that case, we need to introduce some acknowledged method to our message processing pattern. Let’s add the commit method:

Committer (acknowledge message queue that message processed and we want to stop sending it to a consumer)

As a result, we will have three methods for our message queue:

Produce
Consume
Commit

Example with the simple flow with one message:

A producer sends a message to a queue
A consumer received this message and tried to process it
If the consumer fails (or after some time, depends on implementation) and a message was not committed, the consumer will receive this message one more time (go to option 2)
Commit message (do not send it to a consumer anymore)

Kafka message queue key features

First of all, let’s overview the main features you will deal with when choosing Kafka as a message queue.

We need to understand the main concepts we will work with this information. We will not only deal with event publishing and consumption. There are more things we will deal with.

Key names

Topic - something like database or collection in other databases. In a topic Kafka stores all events pushed to it. Consumers use a topic as well to read messages from it.
Broker - Kafka node processes events.
Partition - Kafka has a feature to scale between nodes and consumers. For this reason, the Kafka team introduced a partition rate (it’s just int number). This number is used in the scope of message distribution between partitions. Partitions assign to consumers and brokers uniquely.
Producer - an application that pushes new events to a topic.
Consumer - an application that reads messages from a topic, processes them and commits them.

Message stream

Message queue in Kafka builds over Kafka message stream. Actually, the message stream is just storage with all messages sequentially persisted. Data from the storage can be read by batches with a specific size in bytes.

Kafka can be used as a storage to read the data and build some decisions on top of it. This solution is usually used in Data Science. If we need to speed up calculation over the data, we can use Kafka Streams².

When we want to use Kafka as a message queue. We read a batch of messages[^read_batch_message] with some shifts, parse them and start to process.

Partitions

This article should not focus on the partitions feature in Kafka. Still, unfortunately, it’s impossible to skip the topic in a consumer/producer context.

Warn: All messages with the same key (for example, user_uuid) will be sent to the same partition.

BE CAREFUL!

When we create a topic, we can specify the number of partitions (for example, 3). Then, based on the message key (usually generated UUIDv4), the Kafka client will choose one of the partitions and send a message to this partition.

The diagram above shows how Kafka clients sort messages between partitions. Partitions from topics used for Kafka distribution (add new nodes to a cluster) and consumer distribution as well see consumer paragraph.

Producer

The producer works most simply. We just produce message³ (send it to Kafka). Nothing special we want to discuss.

Consumer

As we agreed before, we will not review the partition part. In this stage, we will work only with group readers.

Kafka provides consumer groups feature. Group ID should be provided⁴ by a consumer that reads from a topic.

Partitions used to distribute messages between consumers. The group ID is used by Kafka to identify messages that need to be delivered to a specific consumer. It automatically distributes partitions between consumers from the same group (Group ID).

Two consumers from the same group will not receive the same message. Instead, every consumer will follow their specific partition(s) and receive a message from them.

If we use user_uuid as a message key, then all messages from the same user will always be sent to the same partition and processed by the same consumer

As we can see on the diagram, consumers automatically assign between partitions while running.

Warn: If we have more consumers than partitions, some will idle.

Commit message

When choosing a message queue pattern, we expect to work with messages individually. We hope to commit messages individually.

In the diagram above, we can see the example of our expectations. Of course, if we do not commit some individual message, we will receive it soon. Still it will not stop processing the following messages.

As we discussed, Kafka does not provide an API to work with messages individually, so we will shift our pointer of reading messages. If we skip one message and do not want to commit it, we cannot move forward, and we need to wait until this message is processed or ignored to shift our pointer and move forward to the next batch.

Following the diagram, we can see that we have to process all messages in any way if we want to move forward.

Warn: Kafka will not solve our individual message processing task. We have to think about a use case when a message cannot be processed.

Conclusion

Kafka is a powerful tool for message processing, but it has some limitations as with any other tool. It can solve a task with billions of messages, and it’s a fast solution. Moreover, it’s a solution with fault-tolerance out of the box. Unfortunately, however, we have to pay for these great features by increased complexity.

When we choose Kafka for the message queue task, instead of, for example, RabbitMQ, let’s remember about limitations and architecture decisions to cover edge cases in our code itself to make our product the best.

The main point I would like to highlight one more time:

Remember about limitations with partitions
Do not forget that you cannot commit individual messages
Do not forget to commit all messages to avoid sticking in a queue

I hope this article will help someone avoid the main mistakes while working with Kafka.

Footnotes

The main concepts of Kafka. I did not find a direct link to Kafka documentation that explains only the partitions part. ↩
Kafka Streams is a solution to run code on the data side to speed up processing. I’d tell it to seem to be stored procedures from SQL. ↩
It requires a message key to specify a partition. Usually, the client will provide some UUIDv4 by default, ignoring it. ↩
A consumer can avoid group id, if it will subscribe to specific partition. But it’s not our case. ↩

Use secure random number generators, please

Anton Ohorodnyk — Sat, 17 Jul 2021 04:16:29 GMT

Introduction

The current article is an introductory article about the practical usage of different generators in standard libraries of other languages. It is the reason we do not explore different pseudo-random algorithms.

The article tries to help avoid the most common issues of developers.

If you have to use a specific random algorithm, certified algorithms, or algorithm forced by your security team or developer’s collegium (for example, some security products and products with particular regulations), just use it.

Motivation

I have seen a few real-world security issues related to the topic in projects I worked on.

Some news (I remember only one the newest of them, I heard about):

Knowing the creation date of an account, an attacker can try to brute-force the account password with a small range of passwords (~100) and gain access to it.

— donjon.ledger.com¹

What random is

On the one hand, random is a simple entity to understand. But on the other hand, a random value - is a value that someone cannot predict.

On the other hand, computer science did not solve the problem with the name “random”. All values generated by pseudo-random algorithms we name as “random”. All randomization algorithms are pseudo-random because they do not generate really random values.

Truly random number generators

“truly random” is more or less a natural random number generator. Specific devices can generate these values. These devices use several techniques for generating entropy, such as atmospheric noise, reverse bias semiconductors and beam splitting (for example - Recommendation for the Entropy Sources Used for Random Bit Generation).

There is at least one service with APIs that can be used to generate “truly random” values:

random.org

Pseudo-random number generators

In all languages I’ve seed before presented at least two different types of random number generators:

Pseudo random number generator². Known as PRNG
Cryptographically secure random number generator. Known as CSPRNG

Pseudo-random number generator (PRNG)

PRNG is a predictable number generator to specify custom “seed”.

Seed is a value that uses as a starting value for the following random values mathematically calculated from previously generated values.

It means that everyone who knows used seed can generate the same infinite sequence of numbers.

When to use

We need to have predictable random values, and we save seeds between generated values
We write a performance-sensitive application, and we do not use random functions for sensitive data

When not to use

Use this function NEVER, until you are sure you HAVE TO use it in your particular case

Small suggestion

If you decide to use this random and you need to specify your custom seed. Use, please, something less predictable than the then-current timestamp in seconds.

Possible ideas (higher is better in my opinion):

If you can use some secure random algorithm, but you decided to use a faster solution, then generate seed once by secure random and use the selected algorithm after that
If you cannot use secure random at all, use different unique values from your node, like the PID of an app, timestamp in nanoseconds, IP address, mac address, etc. Combine as many unique values as you can
Use, at least, UNIX timestamp in nanoseconds with PID

Cryptographically secure random number generator (CSPRNG)

CSPRNG is also a pseudo-random number generator and mathematical. Still it uses more complicated rules and algorithms to make it challenging to predict random values.

These algorithms are usually suggested for “real” sensitive generated values. I’m not a security expert to contradict this statement, but I have a reason to add more rules for using secure random.

Usually, software engineers are not as familiar with security (I’m as well) as we think. So when developers write some feature, they think the particular random number will not affect safety. Still, it can affect it in the future.

When not to use

Use a basic CSPRNG provided by a programming language whenever you need random bytes. You did not decide to use the faster one
Investigate more secure implementation in a language or external libraries, if you need a certified crypto-secure generator

When not to use

From my perspective, there are only two reasons to stop using this solution:

Performance issues that require to switch to something faster
Platform limitations where no way to use this kind of generators

Summary

Until developers are sure what they are doing (even better to get approval from the security team for a particular use case), I’d suggest using CSPRNG from a language.

In libraries, a good way would be to provide options between different random functions and specify crypto-random generators by default.

Footnotes

First found website, because I remember the issue from the news ↩
In some languages, this type of “random” is named as mathematical, for example, Golang math/rand, JavaScript and Java Math.random ↩

Accept header parser and matcher

Anton Ohorodnyk — Sun, 04 Jul 2021 02:09:05 GMT

Introduction

It’s not evident that the Accept header is an essential part of HTTP, especially REST communication. Usually, people do not worry about the Accept header, either I’m.

This short article will show how simple to add validation and matching to your HTTP router independently to the router and framework you use.

About Accept header

Accept header is a simple list of mime types (application/json, text/html, etc.) or wildcard rules (*/*, application/*, etc.) that a client supports. In the case of the provider Accept header, the client expects to respond with one of the matched mime types from the header.

This header also supports additional parameters that a client can provide per a mime type.

The main parameter is quality q, that defines order of rules to choose the best option for a client, for example: */*;q=0.1, application/json; q=1, application/xml; q=0.8. In this example, we prefer to receive a response in JSON. If it’s not supported, return in XML; otherwise, it does not matter the response mime type.

The second example is character encoding charset. If a client sets some charset, it expects to receive a response encoded.

Implementation notes

Following the description above. We can specify requirements for the Accept header parser

Split all mime types by , symbol;
Parse mime type and params;
Order mime types and rules by:
1. Use q parameter to sort;
2. More strict mime types have more priority than wildcards;
3. More parameters have more priority.
Match supported mime types to Accept rules from the Accept header and find the best result for the client or server.

Creating middleware

Your router/framework can support this feature out of the box. If possible, do not spend time implementing your solution. However, read the article in case of:

You disagree with an implementation in your framework;
You need to implement support of Accept header by yourself;
You want to discover more about Accept header supporting.

Background

We will use mimeheader library to parse and match mime types;
Our middleware will implement http.HandlerFunc type;
It’s ONLY for learning purposes, do not use it in real projects AS IS.

Header parsing explanation

In this block, we will review the main part of an example:

header := r.Header.Get("Accept")

// Parse Accept header to build needed rules for matching.
ah := mimeheader.ParseAcceptHeader(header)

// We do not need default mime type.
mh, mtype, m := ah.Negotiate(acceptMimeTypes, "")
if !m {
  // If not matched accept mim type, return 406.
  rw.WriteHeader(http.StatusNotAcceptable)

  return
}

// Add matched mime type to context.
ctx := context.WithValue(r.Context(), "resp_content_type", mtype)
// Add charset, if exists.
chs, ok := mh.Params["charset"]
if ok {
  ctx = context.WithValue(ctx, "resp_charset", chs)
}

Actually the main magic happenes in two lines of code:

// Parse Accept header to build needed rules for matching.
ah := mimeheader.ParseAcceptHeader(header)
// We do not need default mime type.
mh, mtype, m := ah.Negotiate(acceptMimeTypes, "")

That’s precisely the whole code needed to parse and match mime types. Other logic is related to the processing of retrieved data.

http/net middleware for http.HandleFunc

package main

import (
 "context"
 "log"
 "net/http"

 "github.com/safigo/mimeheader"
)

func main() {
 r := http.NewServeMux()

 r.HandleFunc("/", acceptHeaderMiddleware([]string{"application/json", "text/html"})(handlerTestFunc))

 err := http.ListenAndServe(":8080", r)
 if err != nil {
  log.Fatalln(err)
 }
}

func acceptHeaderMiddleware(acceptMimeTypes []string) func(http.HandlerFunc) http.HandlerFunc {
 return func(next http.HandlerFunc) http.HandlerFunc {
  return func(rw http.ResponseWriter, r *http.Request) {
   header := r.Header.Get("Accept")
   ah := mimeheader.ParseAcceptHeader(header)

   // We do not need default mime type.
   mh, mtype, m := ah.Negotiate(acceptMimeTypes, "")
   if !m {
    // If not matched accept mim type, return 406.
    rw.WriteHeader(http.StatusNotAcceptable)

    return
   }

   // Add matched mime type to context.
   ctx := context.WithValue(r.Context(), "resp_content_type", mtype)
   // Add charset, if set
   chs, ok := mh.Params["charset"]
   if ok {
    ctx = context.WithValue(ctx, "resp_charset", chs)
   }


   // New requet from new context.
   rc := r.WithContext(ctx)

   // Call next middleware or handler.
   next(rw, rc)
  }
 }
}

func handlerTestFunc(rw http.ResponseWriter, r *http.Request) {
 mtype := r.Context().Value("resp_content_type").(string)
 charset, _ := r.Context().Value("resp_charset").(string)

 rw.Write([]byte(mtype + ":" + charset))
}

Responses

GET http://localhost:8080/
Accept: text/*; q=0.9,application/json; q=1;

##HTTP/1.1 200 OK
##Date: Sat, 03 Jul 2021 23:55:41 GMT
##Content-Length: 17
##Content-Type: text/plain; charset=utf-8
##
##application/json:

####

GET http://localhost:8080/
Accept: text/*; q=1,application/json; q=1; charset=utf-8bm;

##HTTP/1.1 200 OK
##Date: Sat, 03 Jul 2021 23:56:14 GMT
##Content-Length: 24
##Content-Type: text/plain; charset=utf-8
##
##application/json:utf-8bm

####
GET http://localhost:8080/
Accept: text/html; charset=utf-8; q=1,application/*; q=1; charset=cp1251;

##HTTP/1.1 200 OK
##Date: Sat, 03 Jul 2021 23:54:20 GMT
##Content-Length: 14
##Content-Type: text/plain; charset=utf-8
##
##text/html:utf-8

####
GET http://localhost:8080/
Accept: text/*; q=1,application/*; q=0.9;

##HTTP/1.1 200 OK
##Date: Sat, 03 Jul 2021 23:56:33 GMT
##Content-Length: 10
##Content-Type: text/plain; charset=utf-8
##
##text/html:

####
GET http://localhost:8080/
Accept: text/plain; q=1,application/xml; q=1;

## HTTP/1.1 406 Not Acceptable
## Date: Sat, 03 Jul 2021 19:17:28 GMT
## Content-Length: 0
## Connection: close

Conclusion

Let’s not forget about the Accept header even if this feature is not implemented in the current framework.

If you use Golang and want to work with the Accept header or mime types in general, you could try mimeheader library. I believe it will help with the task.

Quick review of the most popular ways to implement flags

Anton Ohorodnyk — Mon, 04 Jan 2021 02:13:42 GMT

Introduction

There are many ways to store flags and use them for communication between client <-> backend service or service <-> service. In the article, we will review the most popular options and try to help to choose the correct way for the next project.

The most popular ways to work with flags are:

Store in separate column in DB or separate property in a class/struct
Store as an array or in a set
Store as binary data in a variable

Options

Separate field

It means we will create a separate column in DB for each flag.

DB Example

In an example, is_new will be a flag.

id	username	is_new
1	test1	true
2	test2	false

JSON Example

In a code or a NoSQL database representation will be:

[
	{
		"id": 1,
		"username": "test1",
		"is_new": true
	},
	{
		"id": 2,
		"username": "test2",
		"is_new": false
	}
]

Pros

Simple to read
Simple to understand
Self-documented
In a code, it can represent as typed properties
Simple to convert between different representations, like struct/class -> JSON -> protobuf -> MsgPack -> DB row -> struct/class
Simple to use in search and aggregation requests in DB (for example by SQL)
Can be updated in DB in parallel, by UPDATE query

Cons

Takes up memory/storage space/traffic
Could be expensive in development or risky to remove in case of dynamically typed languages
Could be expensive in development to add or remove or update in case of microservice architecture
Could be expensive in development to remove from DB as a column

Use Cases

In my opinion, before using this approach, we should answer “yes” for all listed options in the checklist:

I agree to add and remove columns in your database (or add fields to the NoSQL database)?
I understand it could take up resources in DB will be used
I do not have memory and traffic sensitive clients or services

Store In Array

In the current implementation, we will store all flags in plain names.

DB Example

In a DB, we are going to store values in a separate table as one to many relationships.

Table user:

id	username
1	test1
2	test2

Table user_flag

id	user_id	flag
1	1	is_new
2	1	is_test

Every user_flag row is linked to a row in user table by user_id field. It can be optimized to not store the flag as a string, but it is not so important in our current topic.

JSON Example

In a code or a NoSQL database representation will be:


[
	{
		"id": 1,
		"username": "test1",
		"flags": [
			"is_new",
			"is_test"
		]
	},
	{
		"id": 2,
		"username": "test2",
		"flags": []
	}
]

Pros

Simple to read
Simple to understand
Self-documented
Could be dynamically extended in a code (does not need to update code to add new flag)
Simple to convert between different representations, like struct/class -> JSON -> protobuf -> MsgPack -> DB row -> struct/class
Pretty simple to use in search and aggregation requests in DB (for example by SQL)
Can be updated in DB in parallel, by INSERT query and DELETE query

Cons

Takes up memory/storage space/traffic
Need to write additional code for set and unset and find functions
Could affect different limitation in case of many flags
Could affect performance, especially on a client-side

User Cases

In my opinion, before using this approach, we should answer “yes” for all listed options in the checklist:

I do not have performance and memory and traffic sensitive clients or services
I understand it could take up resources on all sides it would be used

Store In Bitmask

Bitmask is a way when we use bitwise operations to get access to specific flag. Every flag stores in binary data in memory, for example it can be stored in: byte, short, int, uint, long, []byte, []short, map[int]byte, etc.

Simply read README.md from binflags library.

DB Example

In DB, we can store bitmask in various types, like: TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT, BLOB types.

id	username	flags
1	test1	1
2	test2	0

In the current example, flags field has the type INT, and the first bit is is_new flag.

JSON Example

In a code or a NoSQL database representation will be:


[
	{
		"id": 1,
		"username": "test1",
		"flags": 1
	},
	{
		"id": 2,
		"username": "test2",
		"flags": 0
	}
]

Pros

Simple to understand
The fastest implementation for all operations with flags
Do not take up additional memory and traffic resources (always only 1 bit per flag)
Can be used in a search by DB (but cannot be efficiently used indexes)

Cons

In some cases, flags cannot be updated in parallel
Cannot search flags by index
Have to be documented (name to bit mapping)
Should be explained for some people

Use Cases

In my opinion, before using this approach, we should answer “yes” for all listed options in the checklist:

I want to save memory/traffic/processor resources
I understand all limitations

Conclusion

As we can see, as usual, we do not have a silver bullet for all use cases and systems. But the provided list of implementations can help find the best solution for a specific project.

If you want to use the most efficient option in Go, I would suggest checking Go implementation of binary flags for various types.