Last update: Apr 1, 2000
Speaker: Ian BirdOver the past few years, LSF has become a standard for job management on batch farms. However, there are many instances where it cannot be deployed for a variety of reasons. In large farms the cost may be prohibitive for the set of features actually used; small university groups who wish to clone the farms and software of larger laboratories often have constraints which preclude the use of LSF.
This paper discusses a generic interface developed at Jefferson Lab to provide a set of common services to the user, while using any one of a variety of underlying batch management software products. Initially the system provides an interface to LSF and an alternative - Portable Batch System (PBS) developed by NASA and freely available in source form. It is straightforward to extend this to other systems. Such a generic interface allows users to move from one location to another and run their jobs with no modification, and by extension provides a framework for a "global" batch system where jobs submitted at one site may be transparently executed at another. The interface also provides additional features not found in the underlying batch software. Being written in Java, the client can be easily installed anywhere and allows for authenticated remote job submission and manipulation, including a web interface.
This paper will also discuss the problem of keeping a large batch farm occupied with work without waiting for slow tape access. The use of file caching, pre-staging of files from tape and the interconnection with the batch system will be discussed. As well as automated techniques, the provision of appropriate information to the user to allow optimization should not be overlooked.
|| | Home | Bulletins | Committees | Scientific Program | Docs by topics | Social Event | Conference Location | Secretariat ||