Using Shaders For GPU Computation

Shaders are not limited to drawing pixels. They can also be used to solve mathematical problems on the GPU. This style of programming is often called GPGPU: general-purpose computing on graphics processing units.

The basic idea is simple:

  • The CPU creates input buffers and uploads data to GPU memory.
  • A shader runs many parallel invocations on the GPU.
  • Each invocation computes part of the result.
  • The GPU writes results into an output buffer.
  • The CPU waits for the GPU work to finish and reads the output buffer back.

No image has to be presented to the screen.

Rendering Shaders vs Compute Shaders

Vertex and fragment shaders are part of a graphics pipeline. They are designed around drawing primitives such as triangles.

Compute shaders are different: they are designed for general parallel work. A compute shader is dispatched directly with a grid size, not by drawing geometry.

flowchart TD A[CPU application] --> B{Type of GPU work} B -->|Draw command| C[Graphics pipeline] C --> D[Vertex shader] D --> E[Rasterization] E --> F[Fragment shader] F --> G[Framebuffer] B -->|Dispatch command| H[Compute pipeline] H --> I[Compute shader] I --> J[Storage buffer / image] J --> K[CPU readback]

Graphics shaders can be forced into compute-like work by rendering into textures or buffers, but compute shaders are usually the cleaner tool when the goal is math rather than pixels.

Data Flow Between CPU And GPU

GPU memory is separate from normal CPU memory on many systems. Even on integrated GPUs, graphics APIs still make the ownership and synchronization explicit.

The normal flow is:

sequenceDiagram participant CPU as CPU application participant Upload as Upload/staging buffer participant GPU as GPU memory participant Shader as Compute shader participant Readback as Readback buffer CPU->>Upload: Write input arrays CPU->>GPU: Copy input data to GPU buffers CPU->>Shader: Dispatch compute work Shader->>GPU: Read inputs and write outputs CPU->>GPU: Insert memory barrier / wait for completion GPU->>Readback: Copy output data CPU->>Readback: Map and read results

At a high level, the CPU needs to do five things:

  1. Create GPU buffers for the input and output data.
  2. Upload input values to the GPU.
  3. Bind those buffers so the shader can access them.
  4. Dispatch the shader with enough invocations to cover the problem.
  5. Synchronize and copy or map the output buffer back to CPU memory.

Storage Buffers

For mathematical workloads, data is commonly passed through storage buffers.

In GLSL, these are declared with buffer blocks. In Vulkan and modern OpenGL, they are often called SSBOs: shader storage buffer objects.

Example GLSL declarations:

layout(std430, binding = 0) readonly buffer MatrixA {
    float a[];
};

layout(std430, binding = 1) readonly buffer MatrixB {
    float b[];
};

layout(std430, binding = 2) writeonly buffer MatrixC {
    float c[];
};

This means:

  • Binding 0 contains input matrix A.
  • Binding 1 contains input matrix B.
  • Binding 2 contains output matrix C.
  • readonly tells the shader not to write to the input buffers.
  • writeonly tells the shader not to read from the output buffer.
  • std430 defines a predictable memory layout for buffer data.

Matrix Multiplication Example

For matrix multiplication, the goal is:

C = A * B

If A is M x K and B is K x N, then C is M x N.

Each output element is independent:

C[row, col] = sum(A[row, i] * B[i, col]) for i from 0 to K - 1

That independence maps well to the GPU. One shader invocation can compute one output element of C.

flowchart LR A[Matrix A] --> S[Compute shader invocation row,col] B[Matrix B] --> S D[Dimensions M,K,N] --> S S --> C[One element of Matrix C]

Example Compute Shader

This GLSL compute shader computes one matrix element per invocation:

#version 450

layout(local_size_x = 16, local_size_y = 16, local_size_z = 1) in;

layout(std430, binding = 0) readonly buffer MatrixA {
    float a[];
};

layout(std430, binding = 1) readonly buffer MatrixB {
    float b[];
};

layout(std430, binding = 2) writeonly buffer MatrixC {
    float c[];
};

layout(push_constant) uniform Dimensions {
    uint m;
    uint k;
    uint n;
} dims;

void main() {
    uint col = gl_GlobalInvocationID.x;
    uint row = gl_GlobalInvocationID.y;

    if (row >= dims.m || col >= dims.n) {
        return;
    }

    float sum = 0.0;
    for (uint i = 0; i < dims.k; ++i) {
        float av = a[row * dims.k + i];
        float bv = b[i * dims.n + col];
        sum += av * bv;
    }

    c[row * dims.n + col] = sum;
}

Important details:

  • local_size_x and local_size_y define the size of each workgroup.
  • gl_GlobalInvocationID.x selects the output column.
  • gl_GlobalInvocationID.y selects the output row.
  • The bounds check handles matrices whose sizes are not exact multiples of the workgroup size.
  • Each invocation writes exactly one value in C, avoiding write conflicts.

Dispatch Size

The CPU dispatches enough workgroups to cover the output matrix.

For a 16 x 16 local size:

groups_x = ceil(N / 16)
groups_y = ceil(M / 16)
groups_z = 1

The 16 is a practical default rather than a requirement. For the reasoning behind it, the hardware limits it must respect, and how different workgroup shapes trade off, see Compute Workgroup Sizing.

flowchart TD O[Output matrix C has M rows and N columns] --> G[Split into 16 x 16 tiles] G --> W[Dispatch workgroups] W --> I[Each invocation computes one C row,col]

In C or C++ code, the calculation often looks like:

uint32_t groups_x = (n + 15) / 16;
uint32_t groups_y = (m + 15) / 16;

Then the API-specific dispatch call runs the shader:

// OpenGL style
glDispatchCompute(groups_x, groups_y, 1);

// Vulkan style
vkCmdDispatch(command_buffer, groups_x, groups_y, 1);

Passing Data To The GPU

The exact API calls differ between OpenGL and Vulkan, but the shape of the operation is the same.

flowchart LR CPUData[CPU arrays A and B] --> Create[Create GPU buffers] Create --> Upload[Upload/copy data] Upload --> Bind[Bind buffers to shader bindings] Bind --> Dispatch[Dispatch compute shader]

In OpenGL, this usually means:

  • Create buffers with glGenBuffers.
  • Allocate and upload data with glBufferData or glNamedBufferData.
  • Bind buffers to shader storage bindings with glBindBufferBase.
  • Set uniforms or push-like values for dimensions.
  • Call glDispatchCompute.

Conceptual OpenGL flow:

glBindBufferBase(GL_SHADER_STORAGE_BUFFER, 0, matrix_a_buffer);
glBindBufferBase(GL_SHADER_STORAGE_BUFFER, 1, matrix_b_buffer);
glBindBufferBase(GL_SHADER_STORAGE_BUFFER, 2, matrix_c_buffer);

glUseProgram(compute_program);
glDispatchCompute(groups_x, groups_y, 1);

In Vulkan, this usually means:

  • Create VkBuffer objects for inputs and outputs.
  • Allocate and bind VkDeviceMemory.
  • Use a host-visible staging buffer or mapped memory to upload data.
  • Bind buffers through descriptor sets.
  • Bind a compute pipeline.
  • Call vkCmdDispatch inside a command buffer.

Conceptual Vulkan flow:

vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, compute_pipeline);
vkCmdBindDescriptorSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, pipeline_layout,
                        0, 1, &descriptor_set, 0, nullptr);
vkCmdPushConstants(cmd, pipeline_layout, VK_SHADER_STAGE_COMPUTE_BIT,
                   0, sizeof(Dimensions), &dimensions);
vkCmdDispatch(cmd, groups_x, groups_y, 1);

Getting Data Back To The CPU

After dispatching the shader, the CPU cannot immediately assume the result is ready. GPU work is asynchronous.

The CPU must ensure two things:

  • The shader writes are complete and visible to later copy/read operations.
  • The GPU has finished copying or exposing the output data to CPU-visible memory.
flowchart LR Dispatch[Dispatch compute shader] --> Barrier[Memory barrier] Barrier --> Copy[Copy output to readback buffer] Copy --> Wait[Wait for fence / completion] Wait --> Map[Map readback memory] Map --> CPU[CPU reads result array]

In OpenGL, a simple path is:

glDispatchCompute(groups_x, groups_y, 1);

glMemoryBarrier(GL_SHADER_STORAGE_BARRIER_BIT);

glBindBuffer(GL_SHADER_STORAGE_BUFFER, matrix_c_buffer);
void* data = glMapBufferRange(GL_SHADER_STORAGE_BUFFER, 0, output_size,
                              GL_MAP_READ_BIT);
// Copy data into CPU memory here.
glUnmapBuffer(GL_SHADER_STORAGE_BUFFER);

For real applications, persistent mapped buffers, sync objects, and double-buffering can avoid stalling the CPU every frame.

In Vulkan, a common path is:

  • Insert a pipeline barrier from compute shader writes to transfer reads.
  • Copy the device-local output buffer to a host-visible readback buffer with vkCmdCopyBuffer.
  • Submit the command buffer.
  • Wait on a fence.
  • Map the readback buffer memory with vkMapMemory.
  • Invalidate non-coherent memory if required.
  • Read the result array on the CPU.

Why Synchronization Matters

Without synchronization, the CPU may read old or incomplete data. The GPU may also reorder work unless barriers tell it which memory operations must be visible before later operations run.

flowchart TD A[Compute shader writes C] --> B{Barrier present?} B -->|Yes| C[Copy/read sees completed writes] B -->|No| D[Copy/read may see stale or partial data]

Synchronization is one of the biggest differences between CPU-only code and GPU programming. The shader code might be mathematically correct, but the result can still be wrong if memory visibility is not handled correctly.

Compute Without A Window

A compute workload does not need to present an image to the screen.

With Vulkan, you can create an instance, select a physical device, create a logical device with a compute queue, and run compute command buffers without creating a window or swapchain.

With OpenGL, compute shaders still require an OpenGL context. A context is often created through a windowing library, but it can be hidden or headless depending on the platform and driver.

Practical Notes

  • GPUs are fastest when the same operation runs over large amounts of data.
  • Small matrices may be slower on the GPU because upload, dispatch, synchronization, and readback overhead can dominate.
  • Reading results back to the CPU is expensive compared with keeping data on the GPU.
  • For best performance, keep intermediate data on the GPU and only read back final results.
  • Matrix multiplication can be optimized with shared memory, tiling, and vectorized loads, but the simple one-output-element-per-invocation version is easier to understand.

Summary

To use shaders for mathematical computation:

  1. Put input data into GPU buffers.
  2. Bind those buffers to a compute shader.
  3. Dispatch enough shader invocations to cover the problem.
  4. Write results into an output buffer.
  5. Synchronize GPU work.
  6. Copy or map the output buffer so the CPU can read the result.

The same GPU execution model used for rendering can also solve parallel mathematical problems. The main difference is that compute shaders write to buffers or images instead of producing fragments for the screen.