Dipartimento di Informatica e Scienze dell'Informazione
Always refer to Section How to install in the GAMMA page, for how to download the latest release.
15 April 2009:
New release of
GAMMA.
Major debug in the tg3-based driver. Minor fixes elsewhere. Some changes in the configure script.
26 January 2009:
New release of
GAMMA.
More debug in the intra-node communications.
Major debug in the FNN (Flat Neighbourhood Network) support (only available with the e1000-based GAMMA driver, so Intel NICs only).
13 January 2009:
New release of
GAMMA.
This release of GAMMA builds a module for Linux 2.6.24.
Major debug in the intra-node communications.
The GAMMA API for process spawning has slightly changed so allow for
an external process launcher, such as (a patched version of) the "mpiexec"
utility for Torque/PBS.
This implies a new release of MPI/GAMMA must be put in place. Visit the
MPI/GAMMA
web page and download the current release; there, you will also find
the source for tha "mpiexec" utility
(only needed if you want to submit MPI/GAMMA jobs through Torque/PBS).
Many thanks to Brian O'Neill at Philips Technologie GmbH Forschungslaboratorien, Aachen (Germany) for great support!
14 February 2008:
New release of
GAMMA.
This release of GAMMA is for Linux 2.6.24.
The installation procedure for GAMMA has been deeply changed. GAMMA now compiles as a kernel module. The module can be loaded/unloaded at will on the cluster nodes. The GAMMA module uses unexported Linux symbols, in an interesting way.
Although now GAMMA is a module, it still depends upon some Linux internals that tend to change over time. So, GAMMA is by no means ensured to compile with kernels other than 2.6.24.
User of previous releases of GAMMA should refer to the installation manual for the new installation guidelines.
There is also a new release of MPI/GAMMA, please check out, download, install.
25 July 2007:
Application-level performance comparisons (always reported under Section Performance in the GAMMA page): OpenFOAM 1.4 (more to follow)
13 July 2007:
New release of
GAMMA.
Improved intra-node communications, with some changes in the user-level library so also MPIGAMMA and applications must be recompiled. This is a bugfix over the 5 July GAMMA.
10 July 2007:
The 5 July 2007 release of
GAMMA has
shown some bugs during runs, so it has been withdrawn. Current release
is the
12
June 2007.
5 July 2007:
New release of
GAMMA.
Improved intra-node communications, with some changes in the user-level library so also MPIGAMMA and applications must be recompiled.
12 June 2007:
New release of
GAMMA.
Fixed another, minor bug in the flow-control algorithm.
11 June 2007:
New release of
GAMMA.
Fixed a bug in the flow-control algorithm for broadcast communications. No changes at user level.
4 June 2007:
New release of
GAMMA.
Fixed bugs that caused GAMMA to crash sometimes at job launch. For this fix to become effective with MPI jobs, MPIGAMMA must be recompiled against the new GAMMA library, and parallel programs must be relinked.
Added an utility, called gammaconf, to create /etc/gamma.conf configuration files in a semi-automatic way. This also required a slight syntax change in the /etc/gamma.conf files themselves; the old gamma.conf files will not work any more, and must be rebuilt using gammaconf. See the installation manual (in the GAMMA source package) for details. The author of gammaconf is Luigi Rocca, a CS student in our Department. 2002s125 at educ disi unige it
Luigi Rocca has also extended GAMMA with support for multiple NICs, allowing Flat Neighbourhood Network (FNN) cluster topologies. A FNN is equivalent to a star topology (each couple of nodes is separated by a single hop in the network) but is made of several smaller switches, rather that a single, big one. It also requires more NICs per node. Overall, it should cost less. See here for details. For now, only Universal FNNs are supported; so called ``sparse'' FNNs require additional machinery for message routing, that is not currently implemented within GAMMA. FNNs are only supported with the Intel PRO/1000 NICs.
5 May 2007:
New release of
GAMMA.
Fixed a bug in the algorithm for detection of packet losses. The bug caused GAMMA parallel jobs to enter a livelock (continuous network traffic but no job progress). All GAMMA users should upgrade. MPIGAMMA must be recompiled against the new GAMMA library, and parallel programs must be relinked.
The ``configure'' script has been slightly changed, in order to simplify installing GAMMA on nodes with multiple NICs with same chipset; there is no longer any need for the user to replace the original Linux driver by hand in this case (the install script will do this automatically). Please refer to the installation manual, in the GAMMA source package.
1 March 2007:
New release of
GAMMA.
GAMMA driver for Intel PRO/1000 NICs:
Added a workaround to detect and solve NIC hang-ups.
These NICs may block under heavy usage when Jumbo Frames are enabled.
The hang-up probability seems to increase with the MTU size,
and also increases if a node has to send messages to a large set of frequently
changing partner.
While the workaround will eventually solve a NIC hang-up episode, it will
not prevent the parallel job to experience some performance loss (every
hang-up takes a few seconds to be solved). Users who see too many episodes
of NIC hang-up should disable Jumbo Frames. The installation manual
(Section ``Troubleshooting'') explains how a user can detect hang-up episodes
while a parallel job is running.
22 February 2007:
New release of
GAMMA.
Minor bugfixes. A major bugfix in the abort protocol of GAMMA.
Apps and MPIGAMMA must be relinked against the new GAMMA library.
20 December 2006:
New release of
GAMMA.
The only change concerns the installation procedure. I've discovered another feature that must be disabled when configuring the Linux kernel, because it may conflict with GAMMA. This feature is the ``Network console logging support''. All GAMMA users should disable this from their Linux+GAMMA kernels. The installation manual has been updated to report this. Otherwise, the GAMMA code is unchanged.
13 December 2006:
New release of
GAMMA.
A couple of bugs have been fixed, that caused the WRF (Weather Forecast and Research) application to hang. These bugs could potentially show up on other applications as well, on large clusters. Many thanks to Mike Leuthold at atmo_arizona_edu for support on this.
Other changes, that improve the integration of GAMMA within the Linux kernel. In particular, the trap address for GAMMA system calls has been changed from 0x81 to 0x21. This implies, all GAMMA application must be recompiled.
MPIGAMMA must be recompiled/reinstalled as well ("make && make install"), along with the MPI applications.
The GAMMA utilities (gammareset, gammacredit, etc.) are to be recompiled and reinstalled as well. This is automatically done when compiling/installing GAMMA. The executables of these utilities must also be copied onto all cluster nodes. The old executables will not work with the new release of GAMMA.
Support for FNN is not yet ready. We hope to have it working soon...
I'm also working at the "GAMMA as a kernel module" feature. Not easy...
3 November 2006:
New release of
GAMMA.
GAMMA has been ported to Linux 2.6.18.1 . The Linux 2.6.18 kernel had a bug that badly affected NFS, so it was unusable on our cluster; we had to quickly move to 2.6.18.1 .
Next release will hopefully support the Flat Neighbourhood Networks (FNN) interconnection topology. Stay tuned...
17 October 2006:
New release of
GAMMA.
GAMMA has been ported to Linux 2.6.18. The GAMMA drivers are therefore more likely to support recent variants of the already supported chipset families.
Next release will hopefully support the Flat Neighbourhood Networks (FNN) interconnection topology. Stay tuned...
6 October 2006:
New release of
GAMMA.
Bugfix in the GAMMA-modified e1000 driver (eg1000_down()/eg1000_up()).
Next release of GAMMA will be for Linux 2.6.17.
There is also a new release of MPI/GAMMA, please check out, download, install.
14 September 2006:
New release of
GAMMA.
Bugfix in the mechanism for forced delivery of hanging Rx packets.
11 August 2006:
New release of
GAMMA.
Added a temptative workaround for occasional e1000 Tx hang-up (another HW bug of that NIC).
8 August 2006:
New release of
GAMMA.
Improved performance of the ``Tigon 3'' driver for GAMMA. After suppressing the pkt coalescing, the latency has dropped to less than 15 usec back-to-back. Still more than twice the e1000, but not bad.
3 August 2006:
New release of
GAMMA.
New GAMMA driver for the ``Tigon 3'' Gigabit Ethernet chipsets. The driver has not yet tested extensively though. Performance seems to be quite poor, compared to the Intel PRO/1000: back-to-back latency is in the range of 20 usec. More detailed measurements, as well as reliability tests, are in progress.
Fixed small bugs that prevented inter-operation between GAMMA on 32-bit and GAMMA on 64-bit processors.
In the GAMMA driver for the Intel PRO/1000: Rx ring enlarged quite a lot (4096 entries). This makes the driver more reliable under heavy load.
22 July 2006:
New release of
GAMMA.
Bugfixes in the intra-node communication. Other minor bugfixes.
17 July 2006:
New release of
GAMMA.
Fixed possible races in the GAMMA protocol. In the e1000 driver, hardware flow control now disabled by default. It seems though, that those NICs are unable to sustain bidirectional full speed. The GAMMA driver for the Broadcom ``tigon 3'' NICs is ready for testing but didn't find the time yet...
New version of MPI/GAMMA also released. See the MPI/GAMMA web page.
7 July 2006:
New release of
GAMMA.
Fixed bugs in the support for dual-x86_64, after extensive tests on a cluster of 32 dual-x86_64 nodes (courtesy of Tony Ladd, Univ. Florida).
15 June 2006:
New release of
GAMMA.
Fixed a bug that could occasionally give rise to a "registration failure" when launching a parallel job, especially when using subsets of a cluster.
6 June 2006:
New release of
GAMMA.
Improved reliability in the launch of parallel jobs. Also, better reliability when a job is aborted.
New version of MPI/GAMMA also released. See the MPI/GAMMA web page.
24 April 2006:
New release of
GAMMA.
Found what appears to be a hardware bug in our e1000 chipset (more precisely, the Intel 82564EB Gigabit Ethernet chipset). The bug affects GAMMA under heavy load (bidirectional full-duplex traffic tests with dual-CPU nodes). I was unable to find a real fix, but it seems that GAMMA is far more stable by disabling the NIC interrupts after GAMMA startup, and never enabling them again. This is what I did in this new release of GAMMA.
This in turn means IP is no longer supported by the GAMMA driver. I'm quite disappointed with this, but in the economy of the GAMMA project (which has no sponsors) this was the only viable way out. At this point, any cluster using GAMMA must have two separate LANs: one for GAMMA and the other for IP. The IP LAN can be as inexpensive as a 100base-T, if you like.
New version of MPI/GAMMA also released. See the MPI/GAMMA web page.
31 March 2006:
A new release of
GAMMA
is available.
Changes to libgamma: it can now be linked into shared MPI library, and added a configure option to strip away GAMMA-related args from argv.
A new version of MPI/GAMMA has been released as well. See the MPI/GAMMA web page.
20 March 2006:
A new release of
GAMMA
is available.
A small typo prevented GAMMA to compile without Jumbo Frames support. Fixed.
7 March 2006:
A new release of
GAMMA
is available.
The IP part of the GAMMA driver is now working properly.
4 March 2006:
The IP part of the GAMMA driver is not working properly. We are trying to
find the bug. More info and new release of GAMMA expected soon.
17 February 2006:
A new release of
GAMMA
is available.
Improved installation manual; code is untouched.
9 February 2006:
A new release of
GAMMA
is available.
Very minor bug fixes, and improved error messages.
1 February 2006:
A new release of
GAMMA
is available.
Small bug fixes and improved installation script. This version of GAMMA should be more easily adaptable to Linux kernel versions other than 2.6.12 .
24 January 2006:
A new release of
GAMMA
is available.
This is the first release of GAMMA supporting the Intel/AMD 64-bit CPU architecture, aka ``x86-64'' or ``amd64''. On a small cluster of EM64T single-CPU machines, the NAS benchmarks could run on MPI/GAMMA without troubles. More tests have been done on a dual-Opteron box (intra-node communication only). Performance of GAMMA communication (latency, throughput) on 64-bit machines is similar to usual 32-bit machines. However, we still lack precise performance comparisons (GAMMA vs. TCP/IP, 32-bit vs 64-bit, MPI/GAMMA vs. MPICH).
A new version of MPI/GAMMA has been released as well; previous releases will not work on 64-bit clusters. See the MPI/GAMMA web page.
Many thanks to OpenCFD Ltd, United Kingdom, for supporting this further step of GAMMA. They allowed me to remotely use their EMT-64 cluster, and with their assistance everything went fine :-)
Currently, GAMMA only supports the Intel PRO/1000 family of Gigabit Ethernet NICs. A GAMMA driver for the Broadcom ``Tigon 3'' chipset family is in preparation. We would be interested in trying with the 10-Gigabit Ethernet technology, and possibly mark another networking record! GAMMA currently holds the latency record on Gigabit Ethernet, with less than 7 usec back-to-back at user level.
22 August 2005:
A new release of
GAMMA
is available.
Fixed a bug in gamma_mlock(), that prevented non-root users to lock memory for non-blocking communication.
2 August 2005:
A new release of
GAMMA
is available.
Eliminated a race condition in the credit management (flow control).
GAMMA now runs with interrupts enabled, that is, kernel code of GAMMA can be interrupted by other OS tasks. However, IRQ from the NIC are still masked off (polling mode receive) for performance and simplicity reasons.
Internally, the send() routines have been simplified, by deleting obsolete optimizations (fast paths for zero-sized messages).
27 July 2005:
A new release of
GAMMA
is available.
More bug fixes. Eliminated possible race conditions between Tx and Rx, so Tx and Rx are now concurrent again. Tests shows stable execution with the IS benchmark, so I feel GAMMA is quite stable now.
26 July 2005:
A new release of
GAMMA
is available.
This release of GAMMA is for Linux 2.6.12. The only supported NIC is still the Intel PRO/1000, with an updated driver capable of operating more NICs of that family (Intel 8254x chipsets). Many thanks to Klaus Keppler, klaus.keppler AT informatik DOT stud DOT uni-erlangen DOT de.
Various bug fixes.
Still a bit unstable when running the IS benchmark (from NAS suite) on our test cluster (eight dual-Xeons boxes + Gigabit Ethernet). It becomes more stable when disabling concurrency between Tx and Rx (#define TX_RX_CONCURRENT in file gamma_def.h.in).
9 March 2005:
A new release of
GAMMA
is available.
Bug fix: eliminated a possible reentrancy in rx_gamma() through interrupt handler from the GAMMA driver. Many thanks to Carsten Kutzner, ckutzne at gwdg dot de for his kind support.
11 February 2005:
A new release of
GAMMA
is available.
Bug fixes: eliminated the "Power failure" bug, and fixed another dangerous mistake in the intra-node messaging (dual-CPU nodes).
In the ``configure'' script, added an option to enable/disable support to Jumbo Frames. Updated the installation manual accordingly.
17 September 2004:
A new release of
GAMMA
is available.
This release of GAMMA is for Linux 2.6.7. The only supported NIC is the Intel PRO/1000, presumably in all of its flavours (Intel 8254x chipset family). See Section Performance in the GAMMA page, to know about its impressively low end-to-end latency...
From this release on, GAMMA will always require two separate LANs for inter-processor communication: one for GAMMA and another for IP. Support for ``GAMMA and IP traffic on the same LAN'' has indeed been discontinued, because it degradates performance (especially the end-to-end latency) to a level comparable to modern Linux TCP, so it has no more point. Of course you can still use GAMMA ina cluster with a single LAN but all IP services will be suspended during any run of a GAMMA parallel job.
Another noticeable change is in the way receiver handlers are implemented. The trick ``run user code at kernel privilege level'' has been replaced by a smoother mechanism in which the handler runs at the user privilege level. The advantage is that now a receiver handler can run arbitrary code.
The error handlers have been eliminated from the API because they have never been used. So the API routines for port programming have slightly changed. Refer to the new GAMMA API web page.
As a byproduct of all the above changes the GAMMA source code is now simpler and almost all of the assembly code has disappeared. This should help porting GAMMA to different CPU architectures (but I will never do this, unless given a specific budget for such a task).
A new version of MPI/GAMMA has been released as well, see the MPI/GAMMA web page.
The last release of GAMMA for Linux 2.4, with old API and drivers, can be still referenced to by browsing the old GAMMA web page.
13 August 2004:
A new release of
GAMMA
is available.
Minor bug fixes.
This is the last release for Linux 2.4. Next one will be for Linux 2.6, and will bring a few important changes (and a new supported NIC). Stay tuned...
7 April 2004:
A new release of
GAMMA
is available.
Previous GAMMA release did not compile under gcc 2.95 anymore. This should.
In the ``configure'' script, added an option to choose ssh rather than rsh for the remote process instances spawining phase (when launching parallel jobs).
See section How to install in the GAMMA page.
2 March 2004:
A new release of
GAMMA
is available.
The max size of a GAMMA process group has been increased to potentially more than 256. Due to my chronic shortage of cluster nodes I could not test if this works correctly. The current theoretical limit is 64 K.
The max number of GAMMA port for each process instance has been increased past the old limit of 256. Now indeed is set up to 1025. Current limit is 2^15-1 (with huge memory occupancy though). You can change the current limit of 1025 by acting on constant MAX_PORT_NUMBER in the source code.
See section How to install in the GAMMA page.
29 February 2004:
A new release of
GAMMA
is available.
Bug fixes: incorrect use of registers in inline asm code in GAMMA library, and bad reentrance of code in various send() routines.
Now GAMMA seems to work well with multiple threads, under the constraints already mentioned in the previous announcement.
See section How to install in the GAMMA page.
28 February 2004, ``Mashisc'' birthday:
A new release of
GAMMA
is available.
This release of GAMMA is for Linux 2.4.21 and can run on clusters of dual-CPU processing nodes :-)
It also uses ssh in place of rsh to spawn process instances of parallel jobs at launch.
A new version of MPI/GAMMA has been released as well, see the MPI/GAMMA web page.
Performance of NAS NPB benchmark suggest there is room for improvement in the GAMMA intra-node communication. I think for now the important thing is that it works...stay tuned...
I have also played with pthreads and GAMMA. Some bugs still show up, probably related to message fragmentation, but evidence that GAMMA can be thread-compatible as long as different threads use different communication ports. Sharing a port among threads is OK on send, but thread-unsafe on receive: a multi-packet message could be received in different chunks by threads competing on the same input port. With single-packet messages, however, the receive contention ends up with only one winner. This is clearly what one could expect, but implies that GAMMA barrier sync and GAMMA bcast need additional thread synchronization at application level in order for the application to use them meaningfully.
This release of GAMMA compiles with gcc 3.3. No idea if it also compiles under gcc 2.95.3 like the previous one (could not check this out yet).
See section How to install in the GAMMA page.
13 June 2003:
A new version of
GAMMA
has been released.
A driver for the 3COM 3c996 Gigabit Ethernet adapter, but possibly working with the many other NICs engineered around the Broadcom BCM570{0,1,2,3} chipset, is now provided. The driver has been implemented by Marco Maniezzo (marco.maniezzo@polito.it) from Politecnico di Torino, Italy. The driver can probably be further optimized, yet it provides 12 usec latency and and 102 MByte/s maximum throughput which is not bad indeed!
A new version of MPI/GAMMA has been released as well, see the MPI/GAMMA web page.
See section How to install in the GAMMA page.
27 February 2003:
A new version of
GAMMA
has been released.
Minor code changes (into sys_gamma_atomic() and gamma_resume_irq()).
Detected a strange performance loss with the AceNIC adapters and its clones. Throughput of GAMMA with such adapters has significantly decreased with the last releases. I don't know the reason for this. I only know that the problem arises when shifting from Linux 2.4.17 to Linux 2.4.18, so I have released an extra release of GAMMA for the AceNIC adapter and clones, based on Linux 2.4.17.
See section How to install in the GAMMA page.
18 February 2003:
A new version of
GAMMA
has been released.
Marco Ehlert fixed a race condition in the GAMMA driver for the Netgear GA621/GA622T Gigabit Ethernet adapters. The bug was affecting the IP send path in the driver.
See section How to install in the GAMMA page.
18 December 2002:
A new version of
GAMMA
has been released.
This version of GAMMA is for Linux 2.4.20.
See section How to install in the GAMMA page.
8 October 2002:
A new version of
GAMMA for Linux 2.4.16
has been released.
After fixing a race condition between the GAMMA message receive path and the Linux scheduler, GAMMA is now OK when running more parallel jobs simultaneously (time sharing) on the same cluster, even in case of mixed GAMMA and IP traffic on the same LAN (IRQ-driven receive enabled).
Internally, almost all of the assembly code has been replaced by (efficient) C code.
See section How to install in the GAMMA page.
4 October 2002:
A new version of
GAMMA for Linux 2.4.16
has been released.
After fixing various bugs, GAMMA is now OK when running more parallel jobs simultaneously (time sharing) on the same cluster, but only when GAMMA packets run on a dedicate LAN. However, running more parallel jobs at the same time on a cluster in which GAMMA and IP share the same LAN leads to incorrect computations (as seen with the NAS benchmarks atop MPI/GAMMA), probably due to a race condition somewhere.
Some internal changes (migration of some data structures from user level to kernel level), which however do not affect functionality; a slight performance improvement can be observed, but it is a marginal one.
Fixed a minor bug in the GAMMA driver for Netgear GA621 (the boot-up messages from the driver are now printed only if the NIC is detected on the PCI bus).
See section How to install in the GAMMA page.
25 August 2002:
A new version of
GAMMA for Linux 2.4.16
has been released.
Fixed bugs in the flow control algorithm of the GAMMA protocol for broadcast communications.
See section How to install in the GAMMA page.
9 July 2002:
A new version of
GAMMA for Linux 2.4.16
has been released.
Fixed a bug in the GAMMA driver for the AceNIC. The bug was showing up with our new cluster (each PC now has 64 bit, 66 MHz PCI bus).
Added a tree-like algorithm for barrier synchronization, for switched networks. The optimized algorithm for shared networks is still there, it is enabled at configure time if the user does not select "full-duplex" for the cluster interconnect.
Added configure option for excluding flow control with barrier synchronization (better scalability).
Documentation has been updated.
See section How to install in the GAMMA page.
3 May 2002:
The WWW page describing the
Application Programming Interface of GAMMA has been updated.
12 April 2002:
A new version of
GAMMA for Linux 2.4.16
has been released.
A driver for the Netgear GA621/GA622 Gigabit Ethernet adapter is now provided. The driver has been excellently implemented by Marco Ehlert (mehlert@cs.uni-potsdam.de), with support of prof. Bettina Schnor (schnor@cs.uni-potsdam.de) and in cooperation with myself (supported by prof. Schnor during a nice stage at Potsdam). Latency is 8.5 usec, and maximum throughput is 122 MByte/s; these numbers are really impressive.
Minor changes to the GAMMA user API: the family of set_port() routines has been slightly rearranged. This has implications on MPI/GAMMA, a new release of which is also available for download; see the MPI/GAMMA home page. Older versions of MPI/GAMMA will no longer compile under the current version of GAMMA.
Documentation has been updated.
A mysterious lock-up problem has been reported by someone on the GAMMA mailing list, which might have been caused by the use of gcc 2.96. Still investigating.
See sections Performance on Gigabit Ethernet and How to install in the GAMMA page.
7 December 2001:
A new version of
GAMMA for Linux 2.4.16
has been released.
It is for Linux 2.4.16 . Only small adjustments to fit the new Linux kernel.
Improvements in the GAMMA driver for the AceNIC adapter. Basically, the non-blocking send routines now perform reasonably well.
The launch of a GAMMA (as well as MPI/GAMMA) job can hang up under not yet understood circumstances. It seems that such failure occurs only after a large number of GAMMA jobs have already been (successfully) launched. The failure seems to be harmless, and might have to do with a possible bug in the flow control algorithm in the case of broadcast communications (exploited in the launch phases).
See section How to install in the GAMMA page.
26 November 2001:
A new version of
GAMMA for Linux 2.4.9
has been released.
Changed the syntax of assembly macros, in order to compile successfully with gcc 2.95. Successful tests with gcc 2.95.3.
In the configure script of GAMMA, added an option for conditional compilation of the (still unstable) support for more process instances on same computation node.
The GAMMA manual has been updated accordingly.
Many thanks to Marek Duszynski
m.duszynski@ic.ac.uk for his precious (and still continuing) cooperation. It is thanks to him, that GAMMA now compiles with gcc 2.95.See section How to install in the GAMMA page.
14 November 2001:
A new version of
GAMMA for Linux 2.4.9
has been released.
Fixed a syntax error in a section of the GAMMA driver for Intel EEPro/100 adapter.
In the configure script of GAMMA, added an option which enforces the use of less performing but more stable internal routines of the protocol. Selecting this option at configure time will lead to higher stability during the launch of parallel jobs (which otherwise could occasionally cause a crash); the price for this is a performance degradation (+5 µs observed with MPI/GAMMA ``ping-pong'' microbenchmarks). The option has been called ``safe launch'' (in the configure script as well as in the manual).
The GAMMA manual has been updated, to include info about the ``safe launch'' configure option.
Support for more process instances on same computation node is not yet stable.
See section How to install in the GAMMA page.
2 November 2001:
A new version of
GAMMA for Linux 2.4.9
has been released.
Added a GAMMA driver for the Intel EtherExpress Pro/100 adapter. It seems to be stable, although I could only make tests with two machines (shortage of NICs). In particular, the packet loss rate seems now perfectly normal (that is, very low as it should be). Latency is less than 25 µs, maximum throughput is 12.1 MByte/s.
Minor restyling.
The GAMMA manual has been updated, to include info about the new GAMMA driver.
Support for more process instances on same computation node is not yet stable.
See section How to install in the GAMMA page.
30 October 2001:
A new version of
GAMMA for Linux 2.4.9
has been released.
Fixed a bug in the flow control algorithm for broadcast communications.
Found a bug in the receive part of the drivers for 3c905 and Tulip NICs (in case of simultaneus IP and GAMMA traffic). Did not understand what's wrong (it is a race condition though); applied a ``dirty patch'' which apparently works well...
The slightly more aggressive gcc optimization options in Makefiles have been turned off (the compiler I'm using is egcs-2.91.66, and seemingly produces wrong code when the option -mpentium is used).
Support for more process instances on same computation node is not yet stable.
See section How to install in the GAMMA page.
24 October 2001:
A new version of
GAMMA for Linux 2.4.9
has been released.
Fixed bugs in the retransmission algorithm for missing packets.
Fixed bug in the gamma_mlock() function (the compiler was used to silently ignore parts of the function as a result of a "smart optimization" which I had to defeat...).
Minor bug fixes in the receive part of GAMMA (in case of simultaneus IP and GAMMA traffic).
Slightly more aggressive gcc optimization options in Makefiles.
Reduced memory occupancy at runtime.
Support for more process instances on same computation node is not yet stable.
See section How to install in the GAMMA page.
5 October 2001:
A new version of
GAMMA for Linux 2.4.9
has been released.
It is for Linux 2.4.9 . Preliminary performance evaluation shows no significant performance changes since last release of GAMMA (which was for Linux 2.2.17). Supported adapters/chipsets: DEC 2114x ``tulip'' Fast Ethernet chipset, 3COM 3c905C adapter, Alteon Tigon-II Gigabit Ethernet chipset (discontinued).
The GAMMA driver for DEC 2114x ``tulip'' Fast Ethernet chipsets has been completely rewritten. It is now based on the tulip.c Linux network driver. This has been done in order to support more NICs (the tulip.c driver recognizes and operates a number of clones of the aforementioned chipset) with a more recent driver (the glorious de4x5.c Linux driver seems to be no longer maintained).
Minor bug fixes.
The GAMMA manual has been updated.
Support for more process instances on same computation node is not yet stable.
See section How to install in the GAMMA page.
17 July 2001:
Performance results from NAS benchmarks (class A) over a 4-nodes cluster
have been put on the WWW and linked to by the GAMMA home page, section
``Benchmarks''.
13 July 2001:
A new version of
GAMMA
has been released.
Added support for ``safe transmission'' of packets. This increases stability of GAMMA on full-duplex Fast Ethernet networks, at a minor performance penalty.
Minor bug fixes.
The GAMMA manual has been updated.
Support for more process instances on same computation node is not yet stable. The GAMMA driver for 3COM 3c905C may still show a transient failure the first time a GAMMA (or MPI/GAMMA) job is launched after a boot-up of the cluster.
See section How to install in the GAMMA page.
Fixed various bugs in the support for single-network clusters (where GAMMA and IP traffic coexist).
Changed the routines for time measurements in the kernel-level part of GAMMA. The previous routines used double-precision floating point arithmetics; thus, they could change the value of the FPU registers. Unfortunately, the Linux interrupt dispatcher does not save the FPU registers on interrupt, and so a computation phase at user level could be polluted by an interrupt of a GAMMA device. The current routines for time measurements use 64-bit integer arithmetics instead.
Added support for retransmission of missing GAMMA packets. In the current implementation, retransmission is available only with blocking send routines of GAMMA. This new feature makes GAMMA much more reliable than before, especially in case of LAN congestion, and comes at no performance penalty.
Support for the Intel EtherExpress Pro/100 adapter has been discontinued, due to shortage of NICs to play with, and lack of manpower.
The GAMMA manual has been updated.
Support for more process instances on same computation node is not yet stable. The GAMMA driver for 3COM 3c905C may still show a transient failure the first time a GAMMA (or MPI/GAMMA) job is launched after a boot-up of the cluster.
See section How to install in the GAMMA page.
Fixed bug in gamma_tsend() (test for completion of non-blocking GAMMA send).
Fixed bugs in flow control algorithm (broadcast communication).
Added utilities for changing the size of credits in the GAMMA credit-based flow control. The utilities can be used at run-time (that is, no more kernel re-compilation nor cluster reboot when changing credit size).
Found a conceptual bug in the installation procedure of GAMMA: the Linux kernel must have been configured before compiling GAMMA. However, kernel compilation must be done after having compiled GAMMA (as usual). Read the GAMMA manual to know more.
The GAMMA manual has been updated.
GAMMA still appears to loose packets with the Intel EtherExpress Pro 100 adapters (a bug in the NIC firmware?). Support for more process instances on same computation node is not yet stable. The GAMMA driver for 3COM 3c905C may show a transient failure when a GAMMA (or MPI/GAMMA) job is launched after a fresh boot-up of the cluster.
See section How to install in the GAMMA page.
It is for Linux 2.2.17 .
Fixed bugs in the support for mixed GAMMA/IP communication.
Added support to spin-yield waiting on receive.
Added support to so-called ``machine files''; that is, when launching a
GAMMA (or MPI/GAMMA) parallel job, you are now allowed to specify which nodes
of the cluster are to host which process instances, by creating a file
containing a list of machine names.
Read the manual (inside the source distribution) for more info.
The installation procedure has been changed, and now it is much easier
than ever.
Read the manual (inside the source distribution) for more info.
EXPERIMENTAL: More process instances of the same parallel job are now allowed to run on the same CPU (``virtual parallelism''). This is a step towards support for SMP nodes, but still lacks thread safety (and is not very stable...).
Updated the GAMMA installation manual.
Support for the glorious Packet Engines GNIC-II has been discontinued (because the NIC has been discontinued).
GAMMA still appears to loose packets with the Intel EtherExpress Pro 100 adapters (a bug in the NIC firmware?). Support for more process instances on same computation node is not yet stable. The GAMMA driver for 3COM 3c905C may show a transient failure when a GAMMA (or MPI/GAMMA) job is launched after a fresh boot-up of the cluster.
See section How to install in the GAMMA page.
Fixed bugs in the GAMMA driver for 3COM 3c905{rev.B,B,C} adapters.
Updated the GAMMA installation manual.
GAMMA still appears to loose packets with the Intel EtherExpress Pro 100 adapters (a bug in the NIC firmware?).
See section How to install in the GAMMA page.
Fixed bugs in non-blocking send() routines.
From many tests carried out on a number of different clusters with different types of CPUs and NICs, it appears to be no longer necessary to choose ``Pentium'' as the preferred CPU when configuring the Linux+GAMMA kernel.
In the GAMMA source code tree: removed all the README files, replaced by a directory "docs" containing a comprehensive manual for installation and troubleshooting of GAMMA; it is a PostScript file, named installGAMMA.ps.
See section How to install in the GAMMA page.
Fixed a race condition between transmission of data and transmission of credit renewal (flow ctl algorithm).
Minor change: smaller Tx rings (for lower memory occupancy, and thus less troubles with machines with small RAM).
Still to fix: non-blocking send() routines with the supported Gigabit Ethernet NICs; possible bug in flow ctl algorithm, that apparently arises when a GAMMA job terminates with empty flow ctl credits.
See section How to install in the GAMMA page.
Fixed a bug in the flow control algorithm for broadcast communication.
Still to fix: non-blocking send() routines with the supported Gigabit Ethernet NICs.
See section How to install in the GAMMA page.
Major bug fixes on the IP part of the GAMMA driver for 3COM 3c905 family of adapters.
Still to fix: non-blocking send() routines with the supported Gigabit Ethernet NICs.
See section How to install in the GAMMA page.
Major bug fixes on the GAMMA driver for 3COM 3c905 family of adapters. Successfully tested on a pair of PCs equipped with 3COM 3c905C cards.
Still to fix: non-blocking send() routines with the supported Gigabit Ethernet NICs.
See section How to install in the GAMMA page.
The GAMMA driver for Intel EtherExpress Pro 100 NICs is more stable than before, but not yet perfect. Users are required to invoke the gammareset utility on all nodes in between two consecutive launches of GAMMA or MPI/GAMMA jobs; this however prevents simultaneous launches of distinct parallel jobs on the same cluster.
Understood why the GAMMA non-blocking send routines do not work on all the NICs based on the TIGON-II Gigabit Ethernet chipset (Alteon AceNIC, NetGear GA620, 3COM 3c985): it depends on the IRQ coalescing feature being enabled on the Tx side of the driver. Disabling the feature will solve the problem, at the price of a sensible performance decrease for the GAMMA blocking send routines. I need to find a different solution...
Minor bug fixes, code restyling.
See section How to install in the GAMMA page.
I will fix the bug as soon as possible.
Added a GAMMA driver for 3COM 3c905B adapters, by Alessio Menale and Salvatore Filippone (Universita' di Roma ``Tor Vergata'').
Added a GAMMA driver for Gigabit Ethernet NICs based on the Alteon TIGON-II chipset, by Luca Landi and Giuseppe Ciaccio (DISI, Universita' di Genova).
Major changes to the GAMMA protocol: by default, all the blocking send() routines now perform a temporary copy of the message into kernel memory space, in order to decouple the CPU from the physical transmission and therefore decrease the CPU overhead on transmission.
Added a few non-blocking send routines, namely: gamma_isend() (to initiate a GAMMA best-effort transmission), gamma_isend_flowctl() (to initiate a GAMMA flow-controlled transmission), gamma_isend_2p() (same as gamma_isend() but message comes from two distinct memory regions), gamma_isend_2p_flowctl() (same as gamma_isend_flowctl() but message comes from two distinct memory regions), gamma_wsend (to wait for completion of a nonblocking transmission), gamma_tsend (to test for completion of a nonblocking transmission).
Found and fixed a bug which prevented the GAMMA driver for EEPro NICs from running on PCs containing more than one EEPro installed.
See section How to install in the GAMMA page.
Minor performance improvement: saved one PCI bus access (0.7 usec) at the end of GAMMA message receive (by quitting a NIC polling when a message is complete, rather than when the NIC Rx queue is empty; indeed the former test can be done entirely in memory, whereas the latter was done in the I/O space).
Found and fixed a minor bug in function gamma_test() (a necessary device polling cycle was erroneously disabled). However the major bug pointed out yesterday remains; the temporary fix to this major bug, introduced in the yesterday version of GAMMA, seems to yield a slight performance improvement at MPI/GAMMA level, so it might become a permanent optional feature.
See section How to install in the GAMMA page.
Ported to Linux 2.2.13.
Improved global termination by Ctl-C (using SIGSEGV instead of SIGQUIT: will not dump core on local file systems, therefore it is much faster).
Added a TEMPORARY fix to a (not yet well understood) bug in the support for MPI/GAMMA: IRQ-driven message delivery is suspended by those GAMMA init functions specifically used by MPI/GAMMA. This fix makes it impossible (?) to run more MPI/GAMMA applications simultaneously on the same machines in the cluster.
See section How to install in the GAMMA page.
All GAMMA processes spawned by a gamma_init() now change to the working directory of the spawner.
See section How to install in the GAMMA page.
GAMMA for Linux 2.0.36: The efficiency of the GAMMA driver for GNIC-II Gigabit Ethernet NICs has been improved in the range of short messages (up to 12 KByte long)
GAMMA for Linux 2.2.10: Minor bug fixes.
See section How to install in the GAMMA page.
See section Communication Performance in the GAMMA page, and section Communication Performance in the MPI/GAMMA page.
The efficiency of the GAMMA driver for GNIC-II Gigabit Ethernet NICs has been improved in the range of short messages (up to 12 KByte long).
See section How to install in the GAMMA page.
With respect to past releases of GAMMA, both now have a correct (and apparently robust) support to global termination of GAMMA parallel apps in case of a Ctrl-C issued from the user shell. In the release for Linux 2.2.10, GAMMA now reads the clock freq. of the underlying CPU from the Linux kernel.
See section How to install in the GAMMA page.
The former has only minor improvements over the past version.
The latter now includes support for Packet Engines GNIC-II (unavailable in the past version of GAMMA for Linux 2.2). This new GAMMA driver has been quite extensively tested, but only on a two-node cluster, so it should be considered as "not yet fully tested". The GAMMA driver for Intel EEPro100 NICs is still missing in GAMMA for Linux 2.2 .
See section How to install in the GAMMA page.
The former contains an improved support for Packet Engines GNIC-II Gigabit Ethernet NICs, thanks to which now MPI/GAMMA runs at 12 usec one-way latency and 90 MByte/s max throughput.
The latter contains only minor adjustments over the past version.
See sections How to install and Communication Performance in the GAMMA page.
Supported NICs:
Support for Intel EtherExpress Pro 10/100 is enclosed but not yet tested.
See section How to install in the GAMMA page.
Some performance numbers measured with the Intel EtherExpress Pro 10/100 NIC have been added, see section Communication Performance in the GAMMA page.
Supported NICs:
See section How to install in the GAMMA page.
Supported NICs:
See section How to install in the GAMMA page.