Multiview application with many video channels (as many simultaneously streams you need) without requiring a lot of extra memory is possible if the VC-2 HQ codec is chosen.
The Multiview compositing pipeline is executed in the compressed domain. This means that the compressed slices are read from the stream buffers and are combined to create a compressed stream for the chosen output resolution.
A single VC-2 HQ decoder instance is therefore enough, keeping the resource usage at an extremely low level.