13 posts tagged with "bioinformatics"

In Part 1, we built a complete 16-step GATK variant calling pipeline in bash—perfect for academic research and 1-10 samples. But what happens when you need to scale to 100+ samples? This is where Nextflow becomes essential.

📁 Repository: All code from this tutorial is organized in the variant-calling-gatk-pipeline-best-practice-from-scratch repository. The structure follows best practices with separate directories for bash (workflows/bash/) and Nextflow (workflows/nextflow/) implementations.

Before you can transform a bash workflow to Nextflow, you need a solid, reproducible bash baseline. This hands-on guide walks through building a complete 16-step GATK variant calling workflow using bash scripts and Pixi for environment management—following GATK best practices with GVCF mode and hard filtering. While this traditional approach works for academic research and proof-of-concept work, scaling to thousands of samples in industry requires the reproducibility and reliability of workflow managers like Nextflow, which we'll cover in Part 2.

HTSlib-based tools like bcftools and samtools provide powerful capabilities for working with genomic data stored on remote servers. Whether your data is in AWS S3, accessible via FTP, or hosted on HTTPS endpoints, these tools allow you to efficiently query and subset remote files without downloading entire datasets. This guide covers authentication, remote file access patterns, and practical workflows.

Running interactive web applications like RStudio, JupyterLab, and Code Server in containers is a powerful way to provide reproducible analysis environments. However, users often need to spawn additional containerized tools from within these applications. Docker out of Docker (DooD) elegantly solves this by allowing containers to access the host's Docker daemon. This post explains how to set up DooD for interactive web applications and why it's the right approach for bioinformatics workflows.

Unix pipes (|) are one of the most powerful yet underutilized features in bioinformatics. They allow you to chain multiple commands together, processing data in a streaming fashion that dramatically reduces memory usage and disk I/O. This post explores why pipes are essential for bioinformatics work and shows how they work under the hood.

Docker containers are revolutionizing bioinformatics by automating reproducibility and portability across platforms. But what problems can they actually solve? This post shows real-world applications of containers in bioinformatics workflows, then guides you through the simplest possible ways to use, build and debug them.

Docker containers are revolutionizing bioinformatics by automating reproducibility and portability across platforms. But what problems can they actually solve? This post shows real-world applications of containers in bioinformatics workflows, then guides you through the simplest possible ways to use, build and debug them.

Building reproducible bioinformatics pipelines is hard. Every project starts from scratch with its own testing, CI/CD, and deployment strategy. What if you could clone a template, add your analysis tools, and be ready to go?

This post introduces a standardized bioinformatics workflow template featuring consistent testing, CI/CD, and project structure. Developed from real production experience with bioinfor-wf-template, this template reduces setup time from days to minutes, ensures research reproducibility, and promotes modular, reusable code. It is Python-based and ideal for proof-of-concept projects. Support for more advanced and widely adopted bioinformatics frameworks (such as Snakemake and Nextflow) is planned, applying the same core principles while leveraging their native testing systems.

GitHub Actions are powerful for automating bioinformatics pipelines, but waiting 5-10 minutes for each cloud run is painful during development. act lets you run GitHub Actions workflows locally on your machine in seconds, slashing feedback time by 5x.

In this post, we'll explore act, a command-line tool that runs GitHub Actions locally using Docker. Perfect for testing ML pipelines, gene expression analysis, and CI/CD workflows before pushing to GitHub.

Machine learning is transforming bioinformatics, enabling us to discover patterns in biological data. In this first part, we'll build a K-Nearest Neighbors (KNN) classifier from scratch using only Python, then apply it to simulated gene expression data. This post is designed for anyone who knows basic Python and biology—no advanced ML experience required!

Machine learning is transforming bioinformatics by automating pattern discovery from biological data. But what problems can it actually solve? This post shows real-world applications of classification models, then builds the simplest possible classifiers to understand how they work and how to evaluate them. This is Part 0—the practical foundation before diving into complex algorithms like KNN.

Welcome to Part 2 of our series on version control in bioinformatics. In Part 1, we introduced Git fundamentals, branching strategies, and collaborative workflows. In this post, we'll dive into how Continuous Integration and Continuous Deployment (CI/CD) can transform your bioinformatics projects. If these concepts are new to you, don't worry—this guide will walk you through managing your bioinformatics repository to ensure your work is easily reproducible on any machine. Whether your server is wiped or you need to spin up a new virtual machine, you'll be able to quickly rerun your pipeline. With CI/CD, every code update can automatically trigger tests on a small dataset to verify everything works before scaling up, ensuring that new changes don't break your results or workflows.

In Part 1 (this post), we explore the history of Git, its integration with GitHub, and basic hands-on tutorials. Part 2 (coming soon) will cover real-world bioinformatics examples and advanced workflows with best practices.

This part focuses on practical applications, including NGS quality control using multiqc and fastqc.