Sharing resource among GPU threads

When writing parallel code you often have to think about problems in a different way. There is a course on Udacity, Intro to Parallel Programming, that explains the basic ideas of how to solve such problems on the GPU. They use CUDA, so the terminology is a little different, but the approaches are the same.

These are hardware vector threads, where each thread does "exactly the same thing" at "exactly the same time". Dont confuse these with CPU threads, they are a totally different animal.

It is also important to understand that the hardware warp size is not the same as the workgroup size. So on the device, calculations done inside a warp occur simulaneously, but for a large work group, it will be broken up into warp size blocks (and each warp block is executed in series).

This is why you need to insert a barrier after initializing local memory (so all warp blocks are allowed to finish before the rest of the kernel is executed).

So parallel algorithms are a bit different.

Atomic operations are used with memory buffer operations where multiple hardware threads might write to or read from the same memory location.

You can also split your "merge" operation into a separate kernel (perhaps with a different workgroup size) or even hand the buffers back to the CPU,

if the merge is not a parallel operation.