My name is Philipp C. Heckel and I write about nerdy things.

Posts Categorized / Programming


  • Jun 20 / 2021
  • 3
Programming

elastictl: Import, export, re-shard and performance-test Elasticsearch indices

For my work, I work a lot with Elasticsearch. Elasticsearch is pretty famous by now, so I doubt that it needs an introduction. But if you happen to not know what it is: it’s a document store with unique search capabilities, and incredible scalability.

Despite its incredible features though, it has its rough edges. And no, I don’t mean the horrific query language (honestly, who thought that was a good idea?). I mean the fact that without external tools it’s quite impossible to import, export, copy, move or re-shard an Elasticsearch index. Indices are very final, unfortunately.

This is quite often very inconvenient if you have a growing index for which each Elasticsearch shard is outgrowing its recommended size (2 billion documents). Or even if you have the opposite problem: if you have an ES cluster that has too many shards (~800 shards per host is the recommendation I think), because you have too many indices.

This is why I wrote elastictl: elastictl is a simple tool to import/export Elasticsearch indices into a file, and/or reshard an index. In this short post, I’ll show a few examples of how it can be used.

Continue Reading

  • Dec 17 / 2020
  • Comments Off on Snippet 0x0F: Recursive search/replace tool “re”
Code Snippets, Programming

Snippet 0x0F: Recursive search/replace tool “re”

Two and a half years ago, I wrote my first Go program. I wanted to learn another language, and Go looked like a ton of fun: straight forward, easy to learn, and a static binary with no runtime shenanigans. I picked a project and I started hacking. Looking back, the code I wrote is a little cringy, but not terrible. I’d surely do things differently these days, now that I have more Go experience. But we all start somewhere.

However, the tool that I wrote, a recursive search/replace tool which I intelligently dubbed re, is actually incredibly useful: to my own surprise, I use it every day. I haven’t made a single modification to it in all that time (until today for this post). And since I’m in the sharing mood today, I thought I’d share it with the millions of people (cough) that come here every day. Ha!

Continue Reading

  • Dec 13 / 2020
  • Comments Off on Go: Calculating public key hashes for public key pinning in curl
Scripting, Security

Go: Calculating public key hashes for public key pinning in curl

Something occurred to me the other day. This is my blog, and that means I can write about whatever I want. Now you may think that’s totally obvious, but it’s not. For the longest time I wouldn’t blog about anything that I didn’t deem blog-worthy. Small things, like “this is a cool function I found” or “I learned this thing today”, were not blog-worthy in my mind for some reason.

Well today I am changing that. I like writing, but not necessarily so much that I always want to write a super long post. Sometimes, things should be short. Like this one.

So in this super short post I’m gonna show you a cool thing I figured out: How to calculate the the value that curls --pinnedpubkey option needs in Go.

Continue Reading

  • Nov 19 / 2019
  • Comments Off on Providing remote access to devices via SSH tunnels
Uncategorized

Providing remote access to devices via SSH tunnels

At my work, the backup appliances are typically physically located inside the LAN of our end users — much like other appliances such as routers, NAS devices or switches. Under normal circumstances that means that they are behind a NAT and are not reachable from the public Internet without a VPN or other tunneling mechanisms. For my employer’s customers, the Managed Service Provider (MSP), only being able to access their devices with direct physical access would be a major inconvenience.

Fortunately we’ve always provided a remote management feature called “Remote Web” for our customers: Remote Web lets them remotely access the device’s web interface as well as other services (mainly RDP, VNC, SSH), even when the device is behind a NAT.

Internally we call this feature RLY (pronounced: “relay”, like the owl, get it?). In this post, I’d like to talk about how we implemented the feature, what challenges we faced and what lessons we learned.

Continue Reading

  • Jul 22 / 2019
  • Comments Off on Deduplicating NTFS file systems (fsdup)
Uncategorized

Deduplicating NTFS file systems (fsdup)

At my work, we store hundreds of thousands of block-level backups for our customers. Since our customer base is mostly Windows focused, most of these backups are copies of NTFS file systems. As of today, we’re not performing any data deduplication on these backups, which is pretty crazy considering that how well you’d think a Windows OS will probably dedup.

So I started on a journey to attempt to dedup NTFS. This blog post briefly describes my journey and thoughts, but also introduces a tool called fsdup I developed as part of a 3 week proof-of-concept. Please note that while the tool works, it’s highly experimental and should not be used in production!

Continue Reading

  • Jan 05 / 2016
  • 7
Uncategorized

How-To: PHP based JSON-RPC API, with authentication, validation and logging

At my work, we use JSON-RPC based APIs very heavily, in particular with our PHP JSON-RPC library php-json-rpc. While JSON-RPC is not as wide spread as REST, it fits our needs quite nicely. In particular, it is protocol independent and can be used over HTTP, SSH or as local CLI. With our library and its numerous extensions (HTTP, SSH, authentication, validation, request-to-class mapping and logging), development is super fast and incredibly easy.

In this post, I’d like to demonstrate how to set up a PHP based JSON-RPC API, with authentication, validation and logging.

Continue Reading

  • Aug 22 / 2015
  • 1
Code Snippets, Programming

Snippet 0x0C: Load multiple composer.json files at runtime

Remember the times when we copied PHP “libraries” into our project folder, or we copy and pasted code from some random site into our project? Those times are over. Composer and Packagist are the modern way to manage PHP dependencies. They are great. Almost as good as The Maven repos and their build tools in the Java world. However, while Composer is really good at managing the dependencies of a single project, i.e. one composer.json file, it does not play well if you want to plug different projects together at runtime. And by “does not play well” I mean it simply doesn’t work if you have two or more composer.json files. This quick post demonstrates a way around this limitation. Quick and dirty. Just like the foundations of PHP :-)

Continue Reading

  • May 04 / 2015
  • 3
Linux, Programming, Security

OpenSSH ‘AuthorizedKeysCommand’ with ‘fingerprint’ argument (Patch for OpenSSH 6.6p1)

Many of us developers or system administrators use OpenSSH’s public key authentication (aka password-less login) on a daily basis. The mechanism works based on public key cryptography: By adding a RSA/DSA public key to the authorized_keys file, the user with the matching private key can login without a password. The mechanism works great for a couple of hundred, thousands and even 100k thousand users (tested, login takes ~2sec).

But what if there are more keypairs, say, a million users, or a more flexible approach is desired? Maybe with an LDAP or a database backend? Think of GitHub and how they do their ssh git@github.com ... login! This blog post shows you how to do that by patching OpenSSH’s AuthorizedKeysCommand option to support an additional fingerprint argument.

Continue Reading

  • Mar 24 / 2015
  • 2
Code Snippets, Linux, Scripting

Snippet 0x0B: Bash completion with sub-commands and dynamic options

Every system administrator, most programmers and countless of command line surfing Linux/Mac users use it every day without thinking twice. Hitting the tab key twice, [TAB][TAB], has become the most common thing in the world. Bash completion is the magic behind the tab key. It’s easy to use, but it’s a pain to write. This tiny post demonstrates how to write scripts for bash completion, with sub-commands and dynamic parameters. A working script is embedded in my open source file sync software Syncany.

Continue Reading

Pages:1234›