A is a matrix of dimension mxn (both large numbers), containing many, many zeros?

Google for "sparse matrix" - this is a data structure which handles the above case. Typically it might use linked lists, e.g. one linked list per row, to allow zero elements to be skipped over.