	Introduction to Virtual Arrays in C

VIRTUAL AND HUGE ARRAYS Version 2.03
by Graham Robertson


	The Virtual Array manager in this package is designed 
specifically to allow more memory than is normally available to 
the C programmer by caching memory blocks to DISK or extended 
memory. This is done by the use of a virtual memory manager 
which transparently allows the programmer to access arrays of 
any size. This not only makes programming of C programs with 
large amounts of data easier but makes is portable to other 
architecture's.

	Several example programs are included in the package which 
quickly show the easy of use of the array manager. The array 
manager is simplest to use when used with C++.

	The example programs are as follows

C++ programs
--------------------------------------------------------------
	ARRAYS.EXE 		- Example of lots of Virtual 
Arrays
	TARRAYS.EXE		- Same examples but for compilers 
that have templates


C programs

--------------------------------------------------------------
	HUGEARR.EXE		- A huge array
	VIRARRAY.EXE		- Example of lots of Virtual 
Arrays


--------------------------------------------------------------

This software is copyright British Software Licensing 1991-
1992. The usual disclaimers apply.

The number of virtual arrays is restricted to sixteen in the 
unregistered version.

If you use this software please send 20 pounds to British 
Software Licensing. If you require a version which allows 
unrestricted virtual arrays please register your copy of this 
software. A copy of the source code can be obtained by sending 
50 pounds to British Software Licensing.

This price is for a single user license. A site license is 
available at a very reasonable price. 

If you use this software in any other Shareware or commercial 
application please contact the author to make arrangements.



--------------------------------------------------------------
Order Form - Single User License only
---------------------------------------------------------------
To	British Software Licensing
	280 (T/L) West Princes Street,
	Woodlands,
	Glasgow
	United Kingdom
	G4 9EU

(prices are the same anywhere in the World)
please supply me with:

an unrestricted copy of the large array manager  [   ]20 pounds

or

a copy of the source for the virtual array manger[   ]50 pounds

							-------------
Total payment
							--------------


Payment type	ACCESS/VISA/MASTERCARD/CHECK(U.K. pounds)/POSTAL 
ORDER

Name on Card                                      


Card Number                                       


Expiry date                                       


Signature of Cardholder                                     

Name                                        
Address    	                                                   

			                                                    

			                                                    


--------------------------------------------------------------

You may also purchase this software from:
Graham Robertson on
					041-339-8855x5021
or on an Answering Machine	041-339-7264
or via Email		Graham.J.S.Robertson@glasgow.ac.uk
or by Fax on 			041-334-1675
quoting the details required above.


--------------------------------------------------------------


 Files in the package.
VIRIMG.LIB		- Software to allow virtual and large arrays 
VIRIMG.H			- Prototypes of procedures for creating 
virtual arrays

VIRARRAY.C		- Source code of an example of using virtual 
arrays
VIRARRAY.EXE		- Example of using virtual arrays
				  Compiled VIRARRAY.C linked with VIRIMG.LIB

HUGEARR.C		- Source code of an example of using a huge 
array
HUGEARR.EXE		- Example of using a huge array

XARRAY.H			- C++ header for virtual arrays

ARRAYS.CPP		- Part One of the C++ example program
A2.CPP			- Part Two of the C++ example program

TXARRAY.H		- header with templates
TARRAYS.CPP		- part one with templates
TA2.CPP			- part two with templates


--------------------------------------------------------------
All the code here is compiled using the large data model
and Borland C or Turbo C compiler
--------------------------------------------------------------

This software is use by example, so the comments in this 
document mainly explain the procedures that have been developed 
to enable the use of huge and virtual arrays.




1.	Using virtual/extended arrays in C++

	This section describes using the Virtual Array manager in 
C++, the next section describes its use with C, however when 
using C much of the functionality and transparency of the C++ 
version is lost.

	The functions described can only be used if the file 
xarray.h is included in any files using extended arrays.

1.1.	Declaring arrays

	Arrays can be declared and used with any type or with a 
class or structure.

1.1.1.	Simple arrays

	In the header file xarray.h various variable types 
have extended definitions. These types are as follows.

	float, double, int, char

	To create an extended array an 'x' must be prefixed 
to the variable type and the size of the is in parenthesis 
after the variable name. 

	i.e.		x<type> <variable>(size of array);

	Therefore to declare an array of float called 
'costing' with 87654 elements the declaration would be as 
follows.

		xfloat costing(87654);

	This variable can then be used like any other array.

	e.g.		costing[5674]=5.334;

	To pass this array to a function you must retain the 
prefix in the function deflation.

	e.g.		sum (xfloat values)
			{
				.
				.
			}

			main (..
			.
			.
			sum (costing);
			.
			.
			}

1.1.2.	Two dimensional arrays

	Two dimensional arrays are declared in a similar way 
to one dimensional arrays except 'xx' is prefixed to the 
variable type.

	i.e.		xx<type> <variable>(number of rows, number 
of columns)

	Therefore to declare a 240x200 array of char called 
image the declaration would be as follows.

		xxchar image(240,200);

	Again you can make assignments as usual

	e.g.		image[100][100]=10;

	A two dimensional array can be passed to functions as 
for one dimensional arrays.

	Indexing a row of a two dimensional array return a 
one dimensional array of the same type.

	e.g.		xchar imagerow;
			imagerow = image[10];	/* i.e. image[10] 
returns a variable of type xchar */

1.1.3.	Arrays of undefined types

	Arrays of any type can be used but if the type is not 
from the list of defined types you need to insert the 
following after you include the file xarray.h.

		XARRAY (<type>)

			to declare one dimensional arrays and

		XXARRAY (<type>)

			to declare two dimensional array.

	Therefore to define extended arrays of type 'long' 
the following would be inserted into the code.

		XARRAY (long)
		XXARRAY (long)

	Note: If two dimensional arrays of a type are 
required then one dimensional arrays of that type need to 
be declared first.

	If you want to declare a type with a space in it such 
as 'unsigned char' then a typedef needs to be used first.

	e.g.		typedef byte unsigned char
			XARRAY (byte)
			XXARRAY (byte)

	These definitions allow the defined types to used as 
described previously (i.e. xlong, xxlong, xbyte, xxbyte)


1.1.4.	Arrays of structures or classes

	An extended array of a struct or a class can be 
defined in a similar way to other data types. An example 
is given in the program ARRAYS.CPP with a struct called 
'anything'.

First the struct is defined.

	struct anything
	{
		.
		.
	}

	Then the extended arrays are defined as before

	XARRAY (anything);
	XXARRAY (anything);

	This definition creates two types - xanything and 
xxanything.

1.2.	Using templates

The description of declaring virtual arrays in section 1.1. 
uses the file xarray.h that includes a couple of macros. Some 
C++ compilers have class templates. Enclosed is another header 
file (txarray.h) that uses templates instead of macros.

Templates are neated and easier to use. When using the template 
header, to declare a virtual array of any type or class use the 
following format (I have used square brackets here to indicate 
the parameter because templates use the <> symbols):

xarray<[type or class]> [variable]([size])

to declare a two dimensional array use the following:

xxarray<[type or class]> [variable](no. of rows, no. of cols)

Use the xarray declaration in your subroutines as well.

e.g.

	sum (xarray<float> values)

1.3.	Controlling your memory

	With the C++ interface the use of extended memory or disk 
caching is controlled in the background without intervention 
from the program.

	The virtual array manager splits the virtual memory into a 
number of blocks. A number of these blocks are held in 
conventional memory and the rest is held in extended memory or 
on disk. If a block is accessed which is not currently in 
conventional memory then a block is swapped out and the 
required block is swapped in. The size of a block, the number 
of blocks and the number of blocks that can be held in 
conventional memory all affect the speed and the size of 
conventional memory required by your program.

1.3.1.	Declaring size and number of blocks

	When the first Extended array is declared the 
extended memory manger is initialised or a swap file is 
created. All the virtual memory that is needed by your 
program must be defined before the first array is 
declared. The amount of memory allocated is affected by 
the following global variables.

	__block_size (default 32768)	-	Size of a block
	__blocks (default 16)	-	Total number of 
blocks declared.
	__mem_buf (default 4)	-	Number of blocks 
held in conventional 
memory.

	

The amount of conventional memory used is
	  block size*  mem buf.

The amount of virtual memory required is
	  block size*(  blocks-  mem buf)

	By increasing blocks and mem_buf and decreasing 
block_size higher performance may be achieved (And more 
memory used).



1.4.	Other virtual array manager parameters

There is a whole host of parameters that can be adjusted. Most 
of them have comments along with their declaration. You do not 
need to adjust them to get the Virtual array manager to work. 
But once you have got your program working you may want to play 
with them. The variables that hold these parameters are found 
in virimg.h. To change the default values of these variables 
they must be set before any virtual arrays are declared. I.e. 
the first thing in 'main()'.

___saftey
If the virtual array manager is using extended memory and 
your program crashes then the array manager with not be 
able to release its access to the extended memory. For 
this reason when the array manager uses extended memory it 
puts the extended memory handle in a file. When it 
finishes with its extended memory it deletes that file. 
However, if the program crashes then the next time you run 
a program with the array manager it will find the file and 
release the memory that had been taken. If you do not want 
to use this safety utility include the following line in 
your program : ___saftey=0;.
*___safe_filename
This is the name of the file where the extended memory 
file handle is kept. The default is "c:\!saftey!.xms". If 
you want to change the name of the file include the 
following line in your program : 
___safe_filename="<filename>";.

___msgs
By default various messages are output to the screen when 
the array manager is set up and closed down. If you want 
to stop these messages include the following line: 
___msgs=0;.

*___xfilename
When the virtual array manager uses the disk to swap 
memory the virtual arrays are put in the file 
"virarray.swp" which is deleted when the virtual array 
manager is closed down. To change the name of the swap 
files include the following line: 
___xfilename="<filename>";.
___xfile_buf
Your machine has a file buffer which is used when writing 
to and reading files. The size of this buffer for the 
array manager is 32768. To change the size of the buffer 
include the following line: ___xfile_buf=<buffer size>;.

___xarrays
By default the array manager first tries to cache to 
extended memory. If there is not enough extended memory 
available it automatically tries to cache to disk. If you 
want to force the array manager to cache to disk include 
the following line: ___xarrays=XDISK;. There is no option 
to cache part of the arrays to disk and the rest to 
extended memory because much higher performance would be 
achieved by using disk caching combined with a file disk 
cache such as comes with windows, msdos5 and drdos. But if 
you do chose this option allocate a lot of memory to the 
file disk cache. 
___auto_disk
If you do not want the array manager to try and cache to 
disk if it fails to allocated extended memory then include 
the following line: ___auto_disk=0;.



2.	Using the virtual array manager with C
If you do not have a C++ compiler the Array manager can still 
be used but with slight modification to the code. The 
underlying code of the virtual array manager is a C program 
that creates a large number of virtual arrays of type void and 
of a fixed size. The C++ code simulates the variable types and 
array sizes by extensive use of operator overloading.

Because C does not have operator overloading available similar 
routines need to be created by macros. A new macro is needs to 
be created for each extended array declared. Not only this but 
the program must directly interact with the array manager. The 
rest of this section explains the routines available to the 
programmer. Most of the variables described in section 1.3 
still apply.

The virtual array routines create a specified number of arrays 
in memory. If more arrays are used than can fit into the memory 
allocated then the array that has not been accessed for the 
longest is cached to disk or extended memory (The rest of this 
section assumes you are caching to disk). All the virtual 
arrays must be of the same size. The array memory must be 
initialised as follows.

initialise virtual arrays (number of arrays to be stored in 
conventional memory, size of each array, maximum number of 
virtual arrays to be used, whether to cache to disk or 
extended memory (XXMS or XDISK))

This routine initialises a specified number of virtual arrays. 
Each array have identified by a number. The amount of memory 
used depends on the size of the arrays and the number of array 
buffers in conventional memory and virtual memory. The actual 
arrays do not take up disk space or extended memory until they 
are accessed.

To access an array the following procedure will return a 
pointer to the array.

	void *img(<array number>)

(In the restricted version <array number> is between 0 and 15)

If the array is not in memory it will be created if it does not 
exist or it will be loaded off disk if it does exist . This 
pointer can be passed to a subroutine just like a normal array.

	e.g.   
		process (char picture[50][50])
		{
			.
			.
			Some code processing the array normally
			.
			.
		}

		main ()
		{
			.
			.
			process (img(5));
			.
			.
		}

When you pass a virtual array to a subroutine it becomes 
active. When the subroutine finishes the array becomes passive. 
More than one array can be passed to a subroutine but there is 
a limit to the number of arrays active arrays allowed at one 
time. This limit is the first parameter in the 
initialise_virtual_arrays routine.

Also note that if an array has been cached then when it is 
restored it may be restored to a different memory location. 
I.e. do not hold permanent links to and array after it has been 
made passive.

2.1.	Simulating larger arrays

	Another useful procedure is 

	void *large array (<array number>,<array index>)

This routine returns a pointer to the ith element of the <array 
number>.

A macro can then be defined to access an element of a certain 
array where an array number is assigned to an array name.


#define <array name>(x) *((<data type<)large array(<array 
number>,x))

---------------------------------------------------------------

Hugh Arrays

	One hugh array can be created from a series of virtual 
images to almost any size. An example is given in the file 
HUGHARR.C


