Filtering large cohort VCFs sounds like a boring problem until you're staring at a job that's been running for twenty minutes on a file that shouldn't take that long. bcftools is the default tool for the job, and it's correct, but on large files, it spends a surprising amount of time fighting its own memory management rather than actually filtering variants. Every variant line triggers allocations that the garbage collector then has to clean up. At scale, that overhead becomes a real bottleneck.
That's the problem vcfilt was built to solve.
The core idea: zero heap allocation
vcfilt is a streaming VCF filter written in Go, designed around one constraint: process each variant without allocating new heap memory for it. Instead of parsing each line into fresh objects, it reuses buffers across the entire stream. Read, filter, write, repeat, with no per-variant garbage for the collector to chase.
It filters on the fields that matter most in practice, INFO/DP, INFO/AF, and QUAL, and processes in parallel batches, so the zero-allocation core scales across threads instead of running single-file.
The benchmark
On an 18GB plain VCF from 1000 Genomes chromosome 20 (1.8 million variants), vcfilt filtered the full file in 12.3 seconds on a single thread, about 147,000 variants per second. bcftools 1.18 took 148.6 seconds on the same file and filter, a 12.2x difference. On gzip-compressed input, the gap narrows to 7.9x (20.0s vs 157.8s), since decompression becomes more of the bottleneck. Output was byte-for-byte identical across both tools.
Thread count barely matters here, and that's revealing on its own. At 1, 8, and 48 threads, vcfilt's speedup over bcftools stays at roughly 12x the whole way. That's because even at a single thread, vcfilt is I/O-bound: reading 18GB off disk is slower than the CPU work needed to parse and filter it, so adding threads doesn't change the wall-clock time much.
Why it's actually faster
Three things do the work here. First, zero-copy record scanning: the parser finds field boundaries using byte offsets instead of allocating a new string for every field it touches. Second, that's not a rough claim, Go's benchmark tooling confirms it directly: go test -bench -benchmem reports 0 B/op and 0 allocs/op for both parsing and filtering. Third, early-exit evaluation: the cheapest check (the FILTER field, a single byte comparison) runs first, so records that fail early never pay the cost of parsing QUAL or DP at all.
bcftools, by contrast, fully parses every field into htslib structs for each record, even when the filter only needs two or three of them.
Shipping it
vcfilt is distributed as a static binary, a Docker image, and a Singularity container, under the MIT license. The goal was reproducible deployment across whatever environment someone's pipeline already runs in, not a tool that demands its own infrastructure.
The full writeup, methodology, and benchmark details are in the preprint on bioRxiv. Source is on GitHub.