In this paper, we report the design experience and some new research results of a multiprocessor architecture to support image processing and matrix computations. We identify the architectural requirements for an integrated image understanding system. The hardware architectural features of a 16-processor orthogonal multiprocessor prototype system are presented. Designs of architectural supporting blocks are emphasized to support high-performance matrix structured computation, image processing, vision, and neural computing applications. This system is targeted to achieve a peak performance of 400 RISC integer MIPS or a maximum of 640 Mflops. We report simulated performance results of this prototype and emphasize on the scalability issues of this architecture to higher dimensions for solving problems requiring multi-dimensional matrix data structures.