It is the non-dynamic nature of the descriptor sets. In other words the fact that they can not be changed from within a command buffer.
The simplest solution would be to break the currently recorded cmd_buffer (and start new one) on texture/sampler/buffer change. But that would cause extreme performance loss.
The best solution i was able to come up with is to build descriptor sets in parallel with the command buffer.
More specifically to use large dsets and for every tex/sam/buf change to allocate new slots for ALL textures/samplers/buffers (and advance the current position in the dset).
This method utilizes the slot offset parameter of the dset binding command.
This should work more or less but results in a massive redundancy because even when only a single texture/buffer/sampler have changed, all slots must be replicated.
It wastes dset memory and CPU time for processing all the slots over and over again. Probably it also wastes some GPU cache/bandwidth because potentially all dset slots are changed for every draw command (and the GPU must fetch them).
I am almost sure this is not a hardware restriction, because the existance of the grCmdUpdateMemory command hints that any GPU memory can be updated from the command buffer and the dset is just another piece of GPU memory.
Only that if the updated slots are currently cached in the GPU, they must be flushed from there. But then there is the grCmdPrepareMemoryRegions command that does that - flushes various GPU caches.
If only there was special memory state for a memory region that is currently being used as a descriptor set, we could implement manual dset update from within the cmd buffer by using
grCmdPrepareMemoryRegions(change to transfer state) -> grCmdUpdateMemory -> grCmdPrepareMemoryRegions(change back to dset state)