1998 International Workshop on Description Logics
(DL'98)

Trento, Italy
Saturday June 6 - Monday June 8, 1998

(coordinated event with KR'98 -- Sixth International Conference on
Principles of Knowledge Representation and Reasoning)

[DL]

Call for DL Systems Comparison


DL'98 will host a DL systems comparison session. Systems will be compared on the basis of benchmarks which will be prepared in order to emphasize the best out of each system. This first round of the comparison at DL'98 will serve mostly as an evaluation of the benchmark testing set, and it is not intended to be the last word on a possible ranking.

Benchmarks are available <--- (click here)

Aims

We have four aims:

Criteria

A fair comparison is very difficult, since different computers are used, systems are implemented in different programming languages, and expressivity of the different logics varies considerably. We hope to overcome these problem to a certain extent by using different families of parametrized knowledge bases. The time to decide the provability of these KBs is expected to be exponential in the parameter for most reasoners. The participants will determine the largest parameter value that their system can handle within a given time limit. Consequently, the results can only be slightly improved by using a faster computer or a better programming language.

Moreover, we believe that expressivity and efficiency are not the only qualities of a reasoner. Even if reasoner B is `really' considerably less expressive or slower than reasoner A, one may prefer B to A, e.g. if:

Such advantages can be stated in entries. For example, the number of lines of the source code of of a system could be mentioned (if the small size is an advantage for that system), or so could the storage required to process certain KBs (if the system requires little space).

Who can take part?

Everybody who has written a reasoner for DL which accepts a reasonable subset of the language specified by the KRSS document can submit an entry. All the knowledge bases interchanges will be done using the neutral syntax specified by the KRSS document. We prefer reasoners that are easily available to everybody or can be used by some means (e.g., over the WWW).

Everybody who takes part automatically accepts the procedure of the comparison.

Procedure

The benchmark KBs will be available via the comparison home page at http://dl.kr.org/dl/dl98/comparison/. People willing to participate must confirm their participation to Peter Patel-Schneider at pfps@research.bell-labs.com or to Ian Horrocks at horrocks@cs.man.ac.uk by Friday March 6, 1998. We encourage participants to indicate their interest as soon as possible and to participate in the selection of benchmarks.

Participants may additionally submit a set of benchmarks which possibly show where their systems excel. These benchmarks should be written in the neutral KRSS syntax, and should be representable in at least one other available DL system. If these benchmarks are submitted early enough, they may be used as the basis of a parameterized set.

Important: The participants time their systems at their site before the workshop.

The result should contain the following information:

The result should be submitted as a self-contained postscript document to Peter Patel-Schneider at pfps@research.bell-labs.com or to Ian Horrocks at horrocks@cs.man.ac.uk by Friday April 17, 1998.

Publication/presentation of results

The organisers will compile short summaries of these results. If the organisers want to check the results of some participants, then the participants are responsible for providing sufficient support. They must give the organisers the possibility to check the results at the participant's site.

The summaries will appear in the proceedings of the DL'98 workshop.

At the workshop, the participants will have the possibility to demonstrate their systems either on site or over the internet to another site. Posters describing the systems may be displayed.

Everybody is welcome to submit for evaluation by the PC of DL'98 either full papers describing any new approaches or system descriptions describing new implementations (or both).

Timetable

Preliminary disclosure of the basic benchmarks: February 16, 1998
Identification of the DL systems willing to partecipate: March 6, 1998
Optional submission of partecipants' benchmarks: March 6, 1998
Disclosure of the official comparison benchmarks: March 23, 1998
Deadline for the submission of the results: April 17, 1998
Summaries of the results available: May 18, 1998
Meeting at DL'98:
  • Presentation of the results
  • Demonstration of the reasoners
  • Comments and evaluation of the benchmarks
  • Agreement on a corpus of benchmarks
June 6 - 8, 1998

Committee:

Peter F. Patel-Schneider
Bell Labs Research
600 Mountain Ave, 2A-409
Murray Hill, NJ 07974; USA
pfps@research.bell-labs.com
Ian Horrocks
(currently visiting IRST)
Department of Computer Science
University of Manchester
Oxford Road, Manchester, M13 9PL, UK
horrocks@cs.man.ac.uk


RESOURCES

Enquiries about the DL'98 workshop can be made by mailing to dl98@dl.kr.org.

The WWW page of the DL'98 workshop is at http://dl.kr.org/dl/dl98/; check it for updates.

The official Description Logics WWW home page is at http://dl.kr.org/dl/.

Advertising (click for more info):
CLICK HERE for info on KR'98



This page maintained by Enrico Franconi <franconi@irst.itc.it>.
Comments and suggestions are welcome.
Last modified: Wed Apr 15 12:06:37 METDST 1998