\documentclass{llncs}

\usepackage{latexsym}
\usepackage{colortbl}
\usepackage{stdpjmmacros}
\usepackage{casetool}
\usepackage{hdm}



\def\hdmtabletab{\hspace{1em}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\def\email#1{\mbox{\tt #1}}
\def\fname#1{\ensuremath{\mathsf{#1}}}
\def\smalltextsf#1{\begin{small}\textsf{#1}\end{small}} 
\def\overlap{\simeq}
\def\disjoint{\not\approx}

\def\yattacompound(#1,#2)(#3,#4)#5#6#7{\labline(#1,#2)(#3,#4){}{#5}
\putcolourcircle(#3,#4){midgrey}{1.5}{#6}
\stdx=#3
\advance\stdx 1
%\putbox[l,i](\stdx,#4)(0,0){#7}}
\put(\stdx,#4){#7}}


\def\yattaatomic(#1,#2)(#3,#4)#5#6{\labline(#1,#2)(#3,#4){}{#5}
\putcolourcircle(#3,#4){lightgrey}{1.5}{}
\stdy=#4
\advance\stdy -2
%\putbox[b,i](#3,\stdy)(0,0){#6}}
\put(#3,\stdy){\makebox[0em]{#6}}}

\stdpicmag{0}


% Use over bar for sns inverse trasnformations
%\def\isnsmode{0}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\pagestyle{myheadings}

\title{AutoMed: A Data Integration System for Heterogeneous Data Sources}

\author{Michael Boyd, Charalambos Lazanitis, Sasivimol Kittivoravitkul,\\
Peter \mcbrien\ and Nikolaos Rizopoulos}

%\email{\set{mboyd,cl201,sk297,pjm,nr600}@doc.ic.ac.uk}

\institute{Dept. of Computing, Imperial College, London SW7 2AZ}

%\date{}


\begin{document}
\maketitle
\bibliographystyle{plain}

\begin{abstract}

This paper describes the AutoMed repository and some associated tools,
which provide the first implementation of the BAV approach to data
integration.  Apart from being a semantically rich data integration
language, BAV has the additional advantages that provides a method to
support a wide range of data modelling languages, and to describe
transformations between those data modelling languages. This paper
documents how BAV can be implemented, including how the approach can
scale up to handle large and evolving systems, by (1) breaking down
the system into well defined units called subnets that may be edited
independently of each other, (2) providing a template definition
system to automate the production of common mappings between data
sources, and (3) providing a schema-matching tool that assists in the
identification of which transformations should apply.  We illustrate
the implementation with examples in the relational, ER data models,
and semi-structured text files.

\end{abstract}

\section{Introduction}
%%%%%%%%%%%%%%%%%%%%%%
\label{introduction-sec}
\marginpar{Peter}

The AutoMed project\footnote{The AutoMed project was an British EPSRC
funded research project, jointly run by Birkbeck and Imperial
Colleges, in the university of London. The Imperial College group
implemented the data integration toolkit described here, with the the
exception of the query processing component based on the IQL language,
which was developed at Birkbeck College. Software and documentation
are available from the AutoMed website
\textsf{http://www.doc.ic.ac.uk/automed/}.} has developed the first
implementation of a data integration technique called
\keywordabbr{both-as-view}{BAV} \cite{MP03a}, which subsumes the
expressive power of other published data integration techniques
\keywordabbr{global-as-view}{GAV}, \keywordabbr{local-as-view}{LAV},
and \keywordabbr{global-local-as-view}{GLAV} \cite{Len02}. BAV also
distinguishes itself in being the approach which has a clear
methodology for handling a wide range of data models in the
integration process, as apposed to the other approaches that assume
integration is always performed in a single common data model.

In this paper we describe core \keyword{repository} of the AutoMed
toolkit, and several packages that make use of this repository. Apart
from giving a overview of this freely available software product, we
describe the solutions to practical problems of using the BAV approach
to integrate large schemas from heterogeneous and evolving data
sources.

The paper is structured as follows.
Section~\ref{bav-data-integration-sec} reviews the BAV approach and
demonstrates how it models a relational data source, and introduces a
new method for handling semi structured text file data
sources. Section~\ref{repository-sec} then describes how the AutoMed
system handles the BAV description of such data modelling languages
and their integration. We show how to divide a large integration of
data sources into a set of well defined subnetworks, where each
subnetwork may by be created, modified and deleted independently of
the others.  Details of how we approach the transformation between
modelling languages are given in Section~\ref{inter-model-sec}, and
the description of how to program higher level transformations as
sequences of primitive transformations in a \keyword{template}
language are given in Section~\ref{template-sec}.  Finally
Section~\ref{schema-matching-sec} address the problem of automating
the schema matching process in the AutoMed framework.

\section{BAV Data Integration}\marginpar{Peter Sasi}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\label{bav-data-integration-sec}

Data integration is the process of combining several data sources such
they may be queried and updated via some common interface.  This
requires that the each \keyword{local schema} of a data source be
mapped in some way to the \keyword{global schema} of the common
interface.  In the GAV approach \cite{Len02}, this mapping is
specified by writing a definition of each global schema construct as a
view over local schema constructs. In LAV \cite{Len02}, this mapping
is specified as defining each local schema construct as a view over
global schema constructs. GLAV \cite{MH03} is a variant of LAV that
allows the head of the view definition of contain any query on the
local schema.

To describe a modelling language in the BAV approach requires that we
relate each modelling language construct to the nodes, edges and
constraints of the underlying HDM \cite{PM98}. Using a graph as the
underlying model makes it straightforward to represent any data modelling
language. We have shown in
previous work \cite{MP99a,MP03a}, that to model a relational model, we

\begin{itemize}

\item
represent each relation $r$ by the \textsf{Rel} construct
\keyword{scheme} $\hdms{r}$, which takes as instances the primary key
values of the relation. For example, the relation \textsf{result} in
Figure~\ref{schema-three-fig}(a) has a scheme \hdme{result} and
instances
\iqlcode{\iqllist{\iqltuple{\iqlstr{DB},\iqlstr{Mary}},%
\iqltuple{\iqlstr{Fin},\iqlstr{Jane}},\ldots}}.

\item represent each attribute $a$ of $r$ by the \textsf{Att} construct scheme
$\hdms{r,a}$, which takes as instances the primary key values of the
relation and the associated value of the attribute. For example, the
relation \textsf{result} in Figure~\ref{schema-three-fig}(a) has a
scheme \hdme{result,year} and instances
\iqlcode{\iqllist{\iqltuple{\iqlstr{DB},\iqlstr{Mary},1},%
\iqltuple{\iqlstr{Fin},\iqlstr{Jane},3},\ldots}}.

\item represent the primary key integrity constraint named $pk$ on
relation $r$ by the \textsf{PK} construct scheme
$\hdms{pk,r,a_1,\ldots,a_n}$, where $a_1,\ldots,a_n$ are the primary
key attributes. For example, the primary key of the \textsf{result}
table would be
\hdme{result\_pk,result,\hdme{result,code},\hdme{result,name}}
(assuming that \textsf{result\_pk} was the name given in the
database). Since this is a constraint, is takes no instances.

\end{itemize}

One the constructs of the data modelling language have been 
defined in this manner, the mapping
between schemas could be described as a \keyword{pathway} of
\keyword{primitive transformation} steps applied in sequence, each
adding, deleting, or renaming the basic constructs of the schema
modelling language being used. For each construct type in the
modelling language $C$, there exists five primitive transformations:

\begin{enumerate}

\item \textsf{add$C$(\hdmn{s},q)} applied to a schema $S$ produces a
new schema $S'$ that differs from $S$ in having a new construct of
type $C$ detailed by the \keyword{scheme} \hdmn{s}, and the extent of
\hdmn{s} is given by the query \textsf{q} on schema $S$.

\item \textsf{extend$C$(\hdmn{s},ql,qu)} applied to a schema $S$ produces a
new schema $S'$ that differs from $S$ in having a new construct of
type $C$ detailed by the \keyword{scheme} \hdmn{s}. The minimum extent of
\hdmn{s} is given by query \textsf{ql}, which may take the special value
\textsf{Void} if no values of this extent may be derived from $S$. The maximum 
extent of \hdmn{s} is given by \textsf{qu}, which may take the special value 
\textsf{Any} if no limit on this extent may be derived from $S$. 

\item \textsf{delete$C$(\hdmn{s},q)} applied to a schema $S$ produces a
new schema $S'$ that differs from $S$ in not having a construct of
type $C$ detailed by \hdmn{s}, and the extent of
\hdmn{s} may be restored by \textsf{q} on schema $S'$.

Note that \textsf{delete$C$(\hdmn{s},q)} applied to a schema $S$ 
producing schema $S'$ is equivalent to  \textsf{add$C$(\hdmn{s},q)}
applied to $S'$ producing $S$.

\item \textsf{contract$C$(\hdmn{s},ql,qu)} applied to a schema $S$ produces a
new schema $S'$ that differs from $S$ in not having a construct of
type $C$ detailed by \hdmn{s}. The lower bound query \textsf{ql} and upper
bound query \textsf{qu} on schema $S'$ have the same semantics as for
\textsf{extend}.

Note that \textsf{contract$C$(\hdmn{s},ql,qu)} applied to a schema $S$ 
producing schema $S'$ is equivalent to  \textsf{extend$C$(\hdmn{s},ql,qu)}
applied to $S'$ producing $S$.

\item \textsf{rename$C$(\hdmn{s},\hdmn{s'})} applied to a schema
$S$ produces a new schema $S'$ that differs from $S$ in not having
construct of type $C$ detailed \hdmn{s} but having the same type \hdmn{s'}, 
and the two constructs differ only in their textual naming. 

Note that \textsf{contract$C$(\hdmn{s},\hdmn{s'})} applied to a schema $S$ 
producing schema $S'$ is equivalent to  \textsf{rename$C$(\hdmn{s'},\hdmn{s})}
applied to $S'$ producing $S$.

\end{enumerate}

The fact the transformations decribe both how add and delete
constructs means that it is possible to extract GAV, LAV, and GLAV
rules from a BAV.

To demonstrate the approach in use, support we need to transform the
relational \schemaname{S_3} in Figure~\ref{schema-three-fig}(a) into
the relational version of the global schema \schemaname{S_{rg}} in
Figure~\ref{global-schema-fig}(a). The \textsf{level} attribute in
\schemaname{S_3} may be used to divide \textsf{student}s into those
that belong to the \textsf{ug} table, created by transformations
\snsref{s3TosgaddRelug}--\snsref{s3TosgaddPKugpk}, and those the
belong to the \textsf{pg} table by transformations
\snsref{s3TosgaddRelpg}--\snsref{s3TosgaddAttpgname}. Then
transformation \snsref{s3TosgdelAttstudentlevel} removes the
\textsf{level} attribute from \textsf{student}, since it may be
recovered from the \textsf{ug} and \textsf{pg} tables. Finally
\snsref{s3TosgaddAttugppt}--\snsref{s3TosgdelAttstudentppt} moves the
\textsf{ppt} attribute from \textsf{student} to \textsf{ug}, since it
only takes non-null values for undergraduate students.

\begin{small}
\begin{snstranslist}
\multicolumn{2}{l}{\pathway{S_3}{S_{rg}}}\\
\snslabel{s3TosgaddRelug}  
& \fname{addRel}(\hdme{ug},
\iqllist{\iqltuple{x} \mid 
         \iqltuple{x,y} \iqlin \hdme{student,level}; y=\iqlstr{ug}})\\
\snslabel{s3TosgaddAttugname} 
& \fname{addAtt}(\hdme{ug,name}, \iqllist{ \iqltuple{x,y} \mid 
\iqltuple{x,y} \iqlin \hdme{student,name};\\
& \iqltuple{x,z} \iqlin \hdme{student,level}; z=\iqlstr{ug}})\\
\snslabel{s3TosgaddPKugpk}
& \fname{addPK}( \hdme{ug\_pk,ug,\hdme{ug,name}} )\\
\snslabel{s3TosgaddRelpg} 
&\fname{addRel}( \hdme{pg},\iqllist{\iqltuple{x} \mid 
\iqltuple{x,y} \iqlin \hdme{student,level}; y=\iqlstr{pg}})\\
\snslabel{s3TosgaddAttpgname} 
& \fname{addAtt}(  \hdme{pg,name}, 
\iqllist{ \iqltuple{x,y} \mid \iqltuple{x,y} \iqlin \hdme{student,name};\\
& \iqltuple{x,z} \iqlin \hdme{student,level}; z=\iqlstr{pg}})\\
\snslabel{s3TosgaddPKpgpk} 
& \fname{addPK}( \hdme{pg\_pk,pg,\hdme{pg,name}} )\\
\snslabel{s3TosgdelAttstudentlevel} &
\fname{delAtt}(\hdme{student,level},\iqllist{\iqltuple{x,y} \mid
\iqltuple{x} \iqlin \hdme{ug}; y=\iqlstr{ug}} \iqlappend\\
& \iqllist{ \iqltuple{x,y}\mid\iqltuple{x} \iqlin \hdme{pg}; y=\iqlstr{pg}})\\
\snslabel{s3TosgaddAttugppt} 
& \fname{addAtt}( \hdme{ug,ppt},  \iqllist{ \iqltuple{x,y} \mid 
\iqltuple{x}\iqlin\hdme{ug}; \iqltuple{x}\iqlin\hdme{student}; 
\iqltuple{x,y}\iqlin\hdme{student,ppt}} \\
\snslabel{s3TosgdelAttstudentppt} 
&  \fname{delAtt}( \hdme{student,ppt},  \iqllist{ \iqltuple{x,y} \mid 
\iqltuple{x}\iqlin\hdme{student}; \iqltuple{x,y}\iqlin\hdme{ug,ppt}})\\
\end{snstranslist}
\end{small}



\begin{stdfigure}{\schemaname{S_1}: Semi-structured text file of undergraduates}{schema-one-fig}

\stdpicmag{-1}
\small

\subfigure[YATTA schema]{
\begin{picture}(18,14)
\yattaatomic(18,6)(12,2){}{code}
\yattaatomic(18,6)(16,2){}{dept}
\yattaatomic(18,6)(20,2){}{year}
\yattaatomic(18,6)(24,2){?}{grade}
\yattacompound(9,10)(18,6){*}{\iqltuple{}}{course}
\yattaatomic(9,10)(0,6){}{name}
\yattaatomic(9,10)(6,6){}{tutor}
\yattaatomic(9,10)(12,6){}{ppt}
\yattacompound(9,14)(9,10){+}{\iqltuple{}}{student}
\yattacompound(9,14)(9,14){}{}{root}
\end{picture}}
\hspace{1em}
\subfigure[text file]{
\begin{picture}(28,14)
\put(0,7){\colorbox{lightgrey}{\begin{minipage}{8em}
name = Mary\\
tutor = PJM\\
ppt = NR\\
course = DB\\
dept = CS\\
year = 1\\
grade = A
\end{minipage}}}
\put(14,7){\colorbox{lightgrey}{\begin{minipage}{8em}
name = Jane\\
tutor = AP\\
ppt = SK\\
course = Fin\\
dept = CS\\
year = 3\\
grade = C
\end{minipage}}}
\end{picture}}

\subfigure[YATTA data]{
\stdpicmag{-3}
\begin{picture}(59,14)
\yattaatomic(52,6)(46,2){}{`Fin'}
\yattaatomic(52,6)(50,2){}{`CS'}
\yattaatomic(52,6)(54,2){}{3}
\yattaatomic(52,6)(58,2){}{`C'}
\yattacompound(40,10)(52,6){}{}{course}
\yattaatomic(40,10)(32,6){}{`Jane'}
\yattaatomic(40,10)(37,6){}{`AP'}
\yattaatomic(40,10)(42,6){}{`SK'}
\yattacompound(29,14)(40,10){}{}{student}
\yattaatomic(22,6)(16,2){}{`DB'}
\yattaatomic(22,6)(20,2){}{`CS'}
\yattaatomic(22,6)(24,2){}{1}
\yattaatomic(22,6)(28,2){}{`A'}
\yattacompound(9,10)(22,6){}{}{course}
\yattaatomic(9,10)(0,6){}{`Mary'}
\yattaatomic(9,10)(6,6){}{`PJM'}
\yattaatomic(9,10)(12,6){}{`NR'}
\yattacompound(29,14)(9,10){}{}{student}
\yattacompound(29,14)(29,14){}{}{root}
\end{picture}}

\end{stdfigure}

\begin{stdfigure}{\schemaname{S_2}: Semi-structured text file of postgraduates}{schema-two-fig}

\stdpicmag{-1}
\small

\subfigure[YATTA schema]{
\begin{picture}(15,14)
\yattaatomic(15,6)(9,2){}{name}
\yattaatomic(15,6)(13,2){}{tutor}
\yattaatomic(15,6)(17,2){}{year}
\yattaatomic(15,6)(21,2){?}{grade}
\yattacompound(9,10)(15,6){*}{\iqltuple{}}{student}
\yattaatomic(9,10)(0,6){}{code}
\yattaatomic(9,10)(6,6){}{dept}
\yattacompound(9,14)(9,10){+}{\iqltuple{}}{course}
\yattacompound(9,14)(9,14){}{}{root}
\end{picture}}
\hspace{1em}
\subfigure[text file]{
\begin{picture}(15,14)
\put(0,7){\colorbox{lightgrey}{\begin{minipage}{8em}
$[$Fin,CS$]$\\
Fred,PJM,4,A\\
$[$Geo,Maths$]$\\
Fred,PJM,4,A\\
John,AP,4,A
\end{minipage}}}
\end{picture}}

\end{stdfigure}

\begin{stdfigure}{\schemaname{S_3}: relational database 
covering both undergraduates and MSc students}{schema-three-fig}

\stdpicmag{-1}
\small

\subfigure[Relational database schema and data]{\begin{tabular}{llll}
\multicolumn{4}{>{\columncolor{header}}c}{student} \\
\rowcolor{subheader}
\underline{name} & tutor & ppt & level \\
Mary & PJM & NR & ug \\
John & AP  & null & pg\\
Jane & AP  & SK & ug \\
Fred & PJM & null & pg  
\end{tabular}
\hspace{1em}
\begin{tabular}{ll}
\multicolumn{2}{>{\columncolor{header}}c}{course} \\
\rowcolor{subheader}
\underline{code} & dept\\
DB   & CS \\
Fin  & CS \\
Geo  & Maths  
\end{tabular}
\hspace{1em}
\begin{tabular}{llrl}
\multicolumn{4}{>{\columncolor{header}}c}{result} \\
\rowcolor{subheader}
\underline{code} & \underline{name} & year & grade \\
DB   & Mary &  1 & A  \\
Fin  & Jane &  3 & C  \\
Fin  & Fred &  4 & null \\
Geo  & Fred &  4 & A \\
Geo  & John &  4 & B 
\end{tabular}}

\subfigure[ER model used to design relational database]{%
\begin{picture}(44,8)
\entity(8,1){student}
\attribute(8,5)(4,6){level}
\keyattribute(8,4)(4,4){name}
\attribute(8,2)(4,2){tutor}
\nullattribute(8,1)(4,0){ppt}
\entity(28,1){course}
\keyattribute(36,4)(40,5){code}
\attribute(36,2)(40,1){dept}
\attribute(22,3)(21,6){year}
\attribute(22,3)(23,6){grade}
\relationship(16,3)(28,3){result}{0:N}{0:N}
\end{picture}}

\end{stdfigure}

\begin{stdfigure}{Global Schema}{global-schema-fig}

\subfigure[\schemaname{S_{rg}}: Global schema in the relational model]{%
\begin{tabular}{ll}
\multicolumn{2}{>{\columncolor{header}}c}{student} \\
\rowcolor{subheader}
\underline{name} & tutor \\
Mary & PJM \\
John & AP \\
Jane & AP  \\
Fred & PJM
\end{tabular}
\hspace{1em}
\begin{tabular}{lll}
\multicolumn{2}{>{\columncolor{header}}c}{ug} \\
\rowcolor{subheader}
\underline{name} & ppt \\
Mary & NR \\
Jane & SK
\end{tabular}
\hspace{1em}
\begin{tabular}{lll}
\multicolumn{1}{>{\columncolor{header}}c}{pg} \\
\rowcolor{subheader}
\underline{name} \\
Fred \\
John 
\end{tabular}
\hspace{1em}
\begin{tabular}{ll}
\multicolumn{2}{>{\columncolor{header}}c}{course} \\
\rowcolor{subheader}
\underline{code} & dept\\
DB   & CS \\
Fin  & CS \\
Geo  & Maths  
\end{tabular}
\hspace{1em}
\begin{tabular}{llrl}
\multicolumn{4}{>{\columncolor{header}}c}{result} \\
\rowcolor{subheader}
\underline{code} & \underline{name} & year & grade \\
DB   & Mary &  1 & A  \\
Fin  & Jane &  3 & C  \\
Fin  & Fred &  4 & null \\
Geo  & Fred &  4 & A \\
Geo  & John &  4 & B 
\end{tabular}}

\subfigure[\schemaname{S_{yg}}: Global schema in the YATTA model]{%
\stdpicmag{-3}
\begin{picture}(60,14)
\yattaatomic(57,10)(54,6){-}{code}
\yattaatomic(57,10)(60,6){}{dept}
\yattacompound(30,14)(57,10){+}{\iqltuple{}}{course}
\yattaatomic(39,10)(30,6){-}{code}
\yattaatomic(39,10)(36,6){}{name}
\yattaatomic(39,10)(42,6){}{year}
\yattaatomic(39,10)(48,6){?}{grade}
\yattacompound(30,14)(39,10){+}{\iqltuple{}}{result}
\yattaatomic(21,10)(18,6){-}{name}
\yattaatomic(21,10)(24,6){}{tutor}
\yattacompound(30,14)(21,10){+}{\iqltuple{}}{student}
\yattaatomic(12,10)(12,6){-}{name}
\yattacompound(30,14)(12,10){+}{\iqltuple{}}{pg}
\yattaatomic(3,10)(0,6){-}{name}
\yattaatomic(3,10)(6,6){}{ppt}
\yattacompound(30,14)(3,10){+}{\iqltuple{}}{ug}
\yattacompound(30,14)(30,14){}{}{root}
\end{picture}}

\end{stdfigure}

\subsection{Handling Semi-Structured Data}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{YATTA-sec}

\keyword{YATTA} (YAT for Transformation-based Approach) is a variation
of the YAT model \cite{CDSS98} to support the handling of
semistructured data in AutoMed. YATTA extends YAT to distinguish
between ordered and unordered data. YATTA simplifies YAT to allow only
two levels of abstraction: the \keyword{schema level} where the
structure of data is defined, and the \keyword{data level} where
actual data is presented.

In the YATTA model, schemas and data are both represented by rooted
labelled trees. In a \keyword{YATTA schema}, each node is labelled by
a tuple \tuple{{\it name, type}}, where {\it name} is a string
describing what a node represents and {\it type} is the data type of a
node. Type can be either atomic \eg \textsf{string}, \textsf{integer},
{\it etc.}, or compound \ie \textsf{list} (marked `[]'), \textsf{set}
(marked `\iqltuple{}'), or \textsf{bag} (marked `\tuple{}'). Each node
in a \keyword{YATTA data tree} is labelled by a triple \tuple{{\it
name, type, value}}, where \textit{value} is the value associated with
the node. If the node is of atomic type, the value is a data value of
that type. If the node is of compound type, the value is an integer
identifier.  Outgoing edges of \textsf{list} nodes are ordered from
left to right, and are otherwise unordered. The edges of a schema are
labelled with cardinality constraints. The cardinality constraints
determine the number of times corresponding edges may occur in a data
tree.  `$\ast$' indicates zero or more occurrences, `+' indicates one
or more occurrences, `?' indicates zero or one occurrence, and no
label indicates exactly one occurrence. `-' is used to identify a
subset of child nodes, designated as key nodes, which uniquely
identify subtrees in corresponding data tree.

Data sources and their YATTA schemas, \schemaname{S_1} and \schemaname{S_2}, are shown in 
Figure~\ref{schema-eg-fig}. In order to populate a data tree with
the data in the source, a set of \keyword{patterns} that specify the format of
sources is identified. Each atomic node in a schema is associated
with a pattern expressed as a \textit{regular expression}.
For example, ``year$\backslash$s=$\backslash$s($\backslash$d)'' 
, in which `\textsf{$\backslash$s}' matches a white space,
'\textsf{$\backslash$d}' matches integer, would be associated 
with node \textsf{year} in the schema \schemaname{S_1}. The schema and the associated patterns are used to 
retrieve the data in the source and built a corresponding data tree. Figure~\ref{schema-one-fig}(c)
shows the data tree representing the source \schemaname{S_1}.


The YATTA model can be represented in the HDM as shown in
Table~\ref{tab:yatta}. A root node $r$ is represented as a nodal construct. 
Non-root nodes are represented by nodal-linking constructs since 
they can only exist when their parent exists.
A non-root node $n$ is identified by a scheme
$\hdms{n_p,n,t,c}$ and a \textsf{root} node $r$ is identified 
by $\hdms{r,t}$ where $n_p$ is a parent node, $t$ is the
type of a node and $c$ is a cardinality constraint.
Cardinality constraints map to HDM constraints.
To capture the \textsf{list} semantics on the node within HDM, 
an additional edge is inserted between the HDM
representation of the edge and an HDM \textsf{order} node, where
the value of \textsf{order} indicates where its child node appears. 
Similarly \textsf{bag} semantics are implemented by use of 
a node \textsf{count} to indicate how many
occurrences of child nodes are present.

\begin{stdtable}{Definition of YATTA model constructs}{tab:yatta}

\tabcolsep=2pt
\begin{tabular}{|l|l|} \hline
\rowcolor{header} YATTA Construct & HDM Representation \\ \hline
construct: \textsf{rootNode}
& node: $\hdms{r}$ \\

class: \textsf{nodal}
& \\

scheme: $\hdms{r,t}$
& \\ \hline

construct: \textsf{yattaNode}
& node: $\hdms{n}$ \\

class: \textsf{nodal-linking}
& edge: $\hdms{\_,n_p,n}$ \\

and \textsf{constraint}
&       where $n_p$ is a parent of $n$ \\

scheme: $\hdms{n_p,n,t,c}$
& if $\iqltuple{n_p,\textsf{list}}$ then edge: \\

& \hdmtabletab $\hdms{\_,\hdms{\_,n_p, n},\textsf{order}}$ \\

& if $\iqltuple{n_p,\textsf{bag}}$ then edge: \\

& \hdmtabletab $\hdms{\_,\hdms{\_,n_p, n},\textsf{count}}$ \\

&  if $c$ = ``\space'' or ``-'' then constraint: \\

& \hdmtabletab $\tuple{\hdms{\_,n_p,n},\{1\},d}$ \\

& if $c$ = ``\textsf{$\ast$}'' then constraint: \\

& \hdmtabletab $\tuple{\hdms{\_,n_p,n},\{0..N\},d}$ \\

& if $c$ = ``\textsf{+}'' then constraint: \\

& \hdmtabletab $\tuple{\hdms{\_,n_p,n},\{1..N\},d}$ \\

& if $c$ = ``\textsf{?}'' then constraint: \\

& \hdmtabletab $\tuple{\hdms{\_,n_p,n},\{0,1\},d}$ \\

& where if atomic($n$) \\

& then $d=\iqltuple{1..N}$ else $d=\iqltuple{1}$ \\ \hline
\end{tabular}
\end{stdtable}

As with the relational model earlier, primitive transformations
are automatically generated from this specification for adding,
deleting and renaming YATTA nodes.

In order to integrate schemas \schemaname{S_1}, \schemaname{S_2} and \schemaname{S_3},
each schema are first transformed into a union-compatible schema.
A union-compatible schema \schemaname{S_{yg}} for \schemaname{S_1} and \schemaname{S_2} is shown in Figure~\ref{global-schema-fig}(b).
\schemaname{S_{yg}} is designed in such a way that gives a simple representation of both schema \schemaname{S_1} and \schemaname{S_2},
and also allows a natural mapping between the YATTA and relational models.
Once \schemaname{S_1} and \schemaname{S_2} are integrated into \schemaname{S_{yg}} then
\schemaname{S_{yg}} and \schemaname{S_3} are transformed into the global schema as shown in Figure~\ref{global-schema-eg-fig}.


To determine a sequence of primitive transformations from one YATTA 
model $Y$ to another one $Y'$, the following methodology can be used.

\begin{enumerate}
\item \label{first} Conduct a breadth first iteration over the nodes $n'$ of
$Y'$, and for each node $n'$ not found in $Y$:

\begin{enumerate}

\item If $n'$ is of complex type, determine if there is a query $q$ on
$Y$ such that there is a one to one mapping between values returned by
$q$ and values associated to $n'$. If there is, then a new node $n'$
is added into $Y$ by applying a rule \fname{addYattaNode}, with the
special function \textsf{generateId} used on the values returned by
$q$ to generate the identifiers of the complex node. This function
acts as a kind of hashing function, which always returns the
\textit{same} identifier for the same input values, and always returns
\textit{distinct} identifiers for distinct input values.

\item If $n'$ is of simple type, determine if there is a query $q$ on
$Y$ such that the values returned by $q$ are equal to the values
associated with $n'$. If there is, then a new node $n'$ is added into
$Y$ by applying a rule \fname{addYattaNode}, with $q$ placed as the
query part of the transformation. If the query only returns some of
the values of $n'$ then instead use \fname{extendYattaNode}.

\item If no query can be determined, then use \fname{extendYattaNode} with
the value \textsf{void}, which indicates that there is no method to determine
anything about the instances of $n'$ in $Y'$ from the information in $Y$.
\end{enumerate}

\item \label{second} Once all the constructs of $Y'$ have been added
into the model $Y$, all redundant constructs of $Y$ that do not appear
in $Y'$ are removed. This entails a breadth first search over the
nodes $n$ of $Y$, and for each node $n$ which do not appear in $Y'$:

\begin{enumerate}
\item If $n$ is of complex type, determine if there is a query $q$ on
the constructs of $Y'$ in $Y$ such that
there is a one to one mapping between values returned by $q$ and values
associated to $n$. If there is, then the node $n$ is deleted by applying a rule
\fname{deleteYattaNode}, with the function \textsf{generateId} used on the values
returned by $q$ to restore the values of $n$.

\item If $n$ is of simple type, determine if there is a query $q$ on the constructs
of $Y'$ in $Y$ such that the values returned by $q$ are equal to the values
associated with $n$. If there is, then the node $n$ is deleted from $Y$ by applying
a rule \fname{deleteYattaNode}, with $q$ to restore the values of $n$.
If the query only restores some of the values of $n$ then \fname{contractYattaNode}
is used instead.

\item If no query can be determined, then use \fname{contractYattaNode} with
the value \textsf{void}, which indicates that there is no method to restore the instances of $n$ in $Y$
after the deletion.
\end{enumerate}

\end{enumerate}

Pathways constructed with this methodology consist
of a growing phase where the new constructs are added to the
model in step \ref{first}, followed by shrinking
phase where redundant constructs are removed in step
\ref{second}.

To illustrate the methodology, a pathway transforming 
\schemaname{S_1} to \schemaname{S_{yg}} is shown below.

By comparing schemas \schemaname{S_1} and \schemaname{S_{yg}}, the values of node
\textsf{student} in \schemaname{S_1} are associated with the node \textsf{ug}
in \schemaname{S_{yg}}. In transformation \snsref{sone}, therefore, the
\textsf{student} node is renamed to \textsf{ug}. Since there is no
information about postgraduates in \schemaname{S_1}, the \textsf{extend}
transformation with the value \textsf{void} is applied to add the
nodes \textsf{pg} and \textsf{name} in \snsref{stwo} and
\snsref{sthree}. In transformation \snsref{sfour} to
\snsref{ssix}, the node \textsf{student} and its child nodes
\textsf{name} and \textsf{tutor} are added with their accompanying
queries $q$ which partially construct the extent of the new
constructs. The function `\textsf{generateId} in the queries
generates the distinct identifiers of the \textsf{student} node for
the values of \textsf{root} and \textsf{name} nodes. A similar
concept is applied in transformation steps \snsref{sseven} to
\snsref{seleven} where \textsf{results} and its child nodes are
added, and steps \snsref{stwelve} to \snsref{sfourteen} where
\textsf{course} and its child nodes are added.

Transformation steps \snsref{sfifteen} to \snsref{stwenty}
eliminate redundant constructs, and leave only the constructs of
\schemaname{S_{yg}}. The queries accompanying the transformations show that
the extents of the removed constructs are partially restored from
\schemaname{S_{yg}}.

\vspace{0.2cm}

{\bf Pathway \schemaname{S_1} $\rightarrow$ \schemaname{S_{yg}}}

\vspace{0.2cm}

\begin{math}
%\iqltuplelength{\extrarowheight}{1.5pt}
\begin{array}{ll}
%Growing Phrase
\snslabel{sone}   & \fname{renameYattaNode}(\hdme{root, student, set,+},\hdme{root, ug, set,+} ) \\
\snslabel{stwo}   & \fname{extendYattaNode}(\hdme{root, pg, set,+}, \mathrm{void}) \\
\snslabel{sthree} & \fname{extendYattaNode}(\hdme{\hdme{root,pg},name, string,-}, \mathrm{void}) \\
\snslabel{sfour}  & \fname{extendYattaNode}(\hdme{root, student, set,+}, \\
                  & \iqllist{\iqltuple{r,s} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name};\ s \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{sfive}  & \fname{extendYattaNode}(\hdme{\hdme{root,student}, name, string,-}, \\
                  & \iqllist{\iqltuple{s,n} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name};\ s \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{ssix}   & \fname{extendYattaNode}(\hdme{\hdme{root,student}, tutor, string,\ }, \\
                  & \iqllist{\iqltuple{s,t} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name}; \\
                  & \iqltuple{u,t} \iqlin \hdme{\hdme{root,ug},tutor};\ s \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{sseven} & \fname{extendYattaNode}(\hdme{root, result, set,+}, \\
                  & \iqllist{\iqltuple{r,re} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name}; \\
                  & \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code}; \\
                  & \ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{seight} & \fname{extendYattaNode}(\hdme{\hdme{root,result}, code, string,-}, \\
                  & \iqllist{\iqltuple{re,co} \mid \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name}; \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \\
                  & \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code};\ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{snine}  & \fname{extendYattaNode}(\hdme{\hdme{root,result}, name, string,-}, \\
                  & \iqllist{\iqltuple{re,n} \mid \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name}; \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \\
                  & \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code};\ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{sten}   & \fname{extendYattaNode}(\hdme{\hdme{root,result}, year, integer,\ }, \\
                  & \iqllist{\iqltuple{re,y} \mid \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name}; \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \\
                  & \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code}; \iqltuple{c,y} \iqlin \hdme{\hdme{course,\hdme{root,ug}},year}; \\
                  & \ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\end{array}
\end{math}

\begin{math}
%\iqltuplelength{\extrarowheight}{1.5pt}
\begin{array}{ll}
\snslabel{seleven}  & \fname{extendYattaNode}(\hdme{\hdme{root,result}, grade, string,?}, \\
                    & \iqllist{\iqltuple{re,g} \mid \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name}; \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \\
                    & \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code}; \iqltuple{c,g} \iqlin \hdme{\hdme{course,\hdme{root,ug}},grade}; \\
                    & \ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{stwelve}  & \fname{extendYattaNode}(\hdme{root, course, set,+}, \\
                    & \iqllist{\iqltuple{r,x} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \\
                    & \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code};\ x \iqlin \iqllist{\mathrm{generateId}\ r\ co}})  \\
\snslabel{sthirteen}  & \fname{extendYattaNode}(\hdme{\hdme{root,course}, code, string,-}, \\
                      & \iqllist{\iqltuple{x,co} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \\
                      & \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code};\ x \iqlin \iqllist{\mathrm{generateId}\ r\ co}})  \\
\snslabel{sfourteen}  & \fname{extendYattaNode}(\hdme{\hdme{root,course}, dept, string,\ }, \\
                      & \iqllist{\iqltuple{x,d} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,c} \iqlin \hdme{\hdme{root,ug},course}; \\
                      & \iqltuple{c,co} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code}; \iqltuple{c,d} \iqlin \hdme{\hdme{course,\hdme{root,ug}},code}; \\
                      & \ x \iqlin \iqllist{\mathrm{generateId}\ r\ co}})  \\
%Shrinking Phrase
\snslabel{sfifteen}   & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,ug},course}, grade, string,?}, \\
                      & \iqllist{\iqltuple{c,g} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \iqltuple{re,g} \iqlin \hdme{\hdme{root,result},grade};\ c \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{ssixteen}   & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,ug},course}, year, integer,\ }, \\
                      & \iqllist{\iqltuple{c,y} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \iqltuple{re,y} \iqlin \hdme{\hdme{root,result},year};\ c \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{sseventeen} & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,ug},course}, dept, string,\ }, \\
                      & \iqllist{\iqltuple{c,d} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \iqltuple{x,co} \iqlin \hdme{\hdme{root,course},code}; \iqltuple{x,d} \iqlin \hdme{\hdme{root,course},dept}; \\
                      & \ c \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{seightteen} & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,ug},course}, code, string,-}, \\
                      & \iqllist{\iqltuple{c,co} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \ c \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{snineteen}  & \fname{contractYattaNode}(\hdme{\hdme{root,ug}, course, set,+}, \\
                      & \iqllist{\iqltuple{u,c} \mid \iqltuple{r,re} \iqlin \hdme{root,result}; \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \\
                      & \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \iqltuple{ug,n} \iqlin \hdme{\hdme{root,ug},name}; \\
              & \ c \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{stwenty}    & \fname{contractYattaNode}(\hdme{\hdme{root,ug}, tutor, string,\ }, \\
                      & \iqllist{\iqltuple{u,t} \mid \iqltuple{r,u} \iqlin \hdme{root,ug}; \iqltuple{u,n} \iqlin \hdme{\hdme{root,ug},name}; \\
                      & \iqltuple{s,n} \iqlin \hdme{\hdme{root,student},name}; \iqltuple{s,t} \iqlin \hdme{\hdme{root,student},name}}) \\
\vspace{0.1cm}
\end{array}
\end{math}

\vspace{0.2cm}

Similar to the pathway \schemaname{S_1} $\rightarrow$ \schemaname{S_{yg}}, the
transformation steps \snsref{ssone} to \snsref{ssthirteen} in
\schemaname{S_2} $\rightarrow$ \schemaname{S_{yg}} incrementally define the
constructs of \schemaname{S_{yg}} from the constructs of \schemaname{S_2}. Steps
\snsref{ssfourteen} to \snsref{sseighteen} then incrementally
remove the constructs of \schemaname{S_2} and leave only the constructs of
\schemaname{S_{yg}} as desired.

{\bf Pathway \schemaname{S_2} $\rightarrow$ \schemaname{S_{yg}}}

\vspace{0.2cm}

\begin{math}
%\iqltuplelength{\extrarowheight}{1.5pt}
\begin{array}{ll}
%Growing Phrase
\snslabel{ssone}   & \fname{extendYattaNode}(\hdme{root, ug, set,+}, \mathrm{void}) \\
\snslabel{sstwo}      & \fname{extendYattaNode}(\hdme{\hdme{root,ug},name, string,-}, \mathrm{void}) \\
\snslabel{ssthree} & \fname{extendYattaNode}(\hdme{\hdme{root,ug},ppt, string,\ }, \mathrm{void}) \\
\snslabel{ssfour}  & \fname{extendYattaNode}(\hdme{root, pg, set,+}, \\
                  & \iqllist{\iqltuple{r,pg} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
          & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name};\ pg \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{ssfive}  & \fname{extendYattaNode}(\hdme{\hdme{root,pg}, name, string,-}, \\
                  & \iqllist{\iqltuple{pg,n} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
          & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name};\ pg \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{sssix}   & \fname{extendYattaNode}(\hdme{root, student, set,+}, \\
                  & \iqllist{\iqltuple{r,x} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
          & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name};\ x \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{ssseven} & \fname{extendYattaNode}(\hdme{root, name, string,-}, \\
                  & \iqllist{\iqltuple{x,n} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
          & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name};\ x \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{sseight} & \fname{extendYattaNode}(\hdme{root, tutor, string,\ }, \\
                  & \iqllist{\iqltuple{x,t} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
          & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name}; \iqltuple{s,t} \iqlin \hdme{\hdme{\hdme{root,course},student},tutor}; \\
          & \ x \iqlin \iqllist{\mathrm{generateId}\ r\ n}})  \\
\snslabel{ssnine}  & \fname{extendYattaNode}(\hdme{root, result, set,+}, \\
                  & \iqllist{\iqltuple{r,re} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
          & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name}; \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code}; \\
                  & \ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{ssten}   & \fname{extendYattaNode}(\hdme{\hdme{root,result}, code, string,-}, \\
                  & \iqllist{\iqltuple{re,co} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
          & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name}; \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code}; \\
                  & \ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{sseleven}  & \fname{extendYattaNode}(\hdme{\hdme{root,result}, name, string,-}, \\
                    & \iqllist{\iqltuple{re,n} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
            & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name}; \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code}; \\
                    & \ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{sstwelve}  & \fname{extendYattaNode}(\hdme{\hdme{root,result}, year, integer,\ }, \\
                    & \iqllist{\iqltuple{re,y} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
            & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name}; \iqltuple{s,y} \iqlin \hdme{\hdme{\hdme{root,course},student},year};  \\
                    & \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code};\ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\
\snslabel{ssthirteen}  & \fname{extendYattaNode}(\hdme{\hdme{root,result}, grade, string,?}, \\
                      & \iqllist{\iqltuple{re,g} \mid \iqltuple{r,c} \iqlin \hdme{root,course}; \iqltuple{c,s} \iqlin \hdme{\hdme{root,course},student}; \\
              & \iqltuple{s,n} \iqlin \hdme{\hdme{\hdme{root,course},student},name}; \iqltuple{s,g} \iqlin \hdme{\hdme{\hdme{root,course},student},grade};  \\
                      & \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code};\ re \iqlin \iqllist{\mathrm{generateId}\ co\ n}})  \\

\end{array}
\end{math}

\begin{math}
%\iqltuplelength{\extrarowheight}{1.5pt}
\begin{array}{ll}
%Shrinking Phrase
\snslabel{ssfourteen}  & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,course},student}, grade, string,?}, \\
                      & \iqllist{\iqltuple{s,g} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \iqltuple{re,g} \iqlin \hdme{\hdme{root,result},grade};\ s \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{ssfifteen}   & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,course},student}, year, integer,\ }, \\
                      & \iqllist{\iqltuple{s,y} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \iqltuple{re,y} \iqlin \hdme{\hdme{root,result},year};\ s \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{sssixteen}   & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,course},student}, tutor, string,\ }, \\
                      & \iqllist{\iqltuple{s,t} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \iqltuple{x,n} \iqlin \hdme{\hdme{root,student},name}; \iqltuple{x,t} \iqlin \hdme{\hdme{root,student},tutor}; \\
                      & \ s \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{ssseventeen} & \fname{contractYattaNode}(\hdme{\hdme{\hdme{root,course},student}, name, string,-}, \\
                      & \iqllist{\iqltuple{s,n} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \\
                      & \ s \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{sseighteen}  & \fname{contractYattaNode}(\hdme{\hdme{root,course}, student, set,\ensuremath{\ast}}, \\
                      & \iqllist{\iqltuple{c,s} \mid \iqltuple{r,re} \iqlin \hdme{root,result}; \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code}; \\
              & \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \ c \iqlin \iqllist{\mathrm{generateId}\ r \ co}; \\
                      & \ s \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\

\vspace{0.1cm}
\end{array}
\end{math}

The pathways \schemaname{S_1} $\rightarrow$ \schemaname{S_{yg}} and \schemaname{S_2} $\rightarrow$
\schemaname{S_{yg}} show how \schemaname{S_1} and \schemaname{S_2} are transformed into \schemaname{S_{yg}}
step by step. The sequences of transformations in both pathways in
fact follow the same processes, flattening, splitting and pruning.
Intuitively, the flattening process first adds to each complex node 
the key nodes of its parent and lifts each such node with its atomic 
child nodes to the \textsf{root}. The splitting process
decomposes the complex nodes and their child nodes according to
constraints that may be known. 
%As mentioned in Section~\ref{YATTA-sec}, a
%YATTA model follows semistructured data sources naturally, hence
%constraints are not explicitly presented in the model. However,
%hidden constraints are important, especially when constraints
%needs to be represented in other models. 
Redundant nodes are then removed in the pruning phase.

The flattening and splitting processes are applied to
transformation steps \snsref{sfour}-\snsref{sfourteen} in the
pathway \schemaname{S_1} $\rightarrow$ \schemaname{S_{yg}} and transformation steps
\snsref{sssix}-\snsref{ssthirteen} in the pathway \schemaname{S_2}
$\rightarrow$ \schemaname{S_{yg}}. The shrinking phase in both pathways
applies the pruning process.

In the pathway \schemaname{S_1} $\rightarrow$ \schemaname{S_{yg}}, the complex node
\textsf{course} with a new name \textsf{result} and 
its atomic child nodes are lifted to the \textsf{root}, 
transformation steps \snsref{sseven}-\snsref{seleven}. 
The key node \textsf{name} is added to the \textsf{result}. 
Since there is a known functional dependency 
where the values of the \textsf{dept} node depend on
the values of the \textsf{code} node, the splitting process
decomposes the \textsf{result} node by adding the complex node
\textsf{course} together with the \textsf{code} and \textsf{dept}
nodes to \textsf{root}. The similar processes are applied to the
pathway \schemaname{S_2} $\rightarrow$ \schemaname{S_{yg}} where the complex node
\textsf{student} is moved and the \textsf{code} node is added.
Then the functional dependency in which the values \textsf{year}
and \textsf{grade} depend on the values of \textsf{code} and
\textsf{name} results in adding the node \textsf{result} with
\textsf{code,name,year} and \textsf{grade} as its child nodes.

YATTA constructs can be mapped to simple relational constructs according to the rules as follows.

\begin{enumerate}
\item A YATTA node $n$ of complex type maps to a relation in the relational model,
the primary key of which s determined by the key nodes of $n$.

\item YATTA nodes of atomic type are defined as attributes in the relational model.
The cardinality constraints on attributes are determined by the cardinality constraints of the YATTA node and its parent.
If the cardinality constraint of the YATTA node is either `$\ast$' or `?',
the cardinality constraint of the attribute is $\iqltuple{1}$,$\iqltuple{1..N}$.
Otherwise, the cardinality constraint of the attribute is $\iqltuple{0,1}$,$\iqltuple{1..N}$
unless the cardinality constraint of the parent of the YATTA node is `$\ast$' or `?'.
\end{enumerate}

As mentioned earlier, the represenation of the union-compatible schema \schemaname{S_{yg}} allows 
a direct mapping between the YATTA and relation model.   
A sequence of transformations on the YATTA schema \schemaname{S_{yg}}
to the relational schema \schemaname{S_{rg}} therefore can be simply defined using the rules described above. 
Below is the pathway of transformations on \schemaname{S_{yg}} to \schemaname{S_{rg}}.  
In the pathway, each complex node together with its key node in \schemaname{S_{yg}} is transformed into a
relation with a primary key in \schemaname{S_{rg}} as shown in transformation
steps \snsref{sone}, \snsref{sfour}, \snsref{sseven} and
\snsref{sfourteen}. As can be seen in the transformation, 
the attributes of the relation are the child nodes of atomic type.  
Once all the constructs of \schemaname{S_{rg}} are defined in \schemaname{S_{yg}}, 
the constructs of \schemaname{S_{yg}} are removed.   

\vspace{0.2cm}

{\bf Pathway \pathway{S_{yg}}{S_{rg}}}

\vspace{0.2cm}

\begin{math}
%\iqltuplelength{\extrarowheight}{1.5pt}
\begin{array}{ll}
%Growing Phrase
\snslabel{sone}   & \fname{addRel}(\hdme{student}, \iqllist{\iqltuple{n} \mid \iqltuple{s,n} \iqlin \hdme{\hdme{root,student},name}}) \\
\snslabel{stwo}   & \fname{addAtt}(\hdme{student,name}, \iqllist{\iqltuple{n,n} \mid \iqltuple{s,n} \iqlin \hdme{\hdme{root,student},name}}) \\
\snslabel{sthree} & \fname{addAtt}(\hdme{student,tutor}, \iqllist{\iqltuple{n,t} \mid \iqltuple{s,n} \iqlin \hdme{\hdme{root,student},name}; \\
          & \iqltuple{s,t} \iqlin \hdme{\hdme{root,student},tutor} }) \\
\snslabel{sfour}  & \fname{addRel}(\hdme{ug}, \iqllist{\iqltuple{n} \mid \iqltuple{ug,n} \iqlin \hdme{\hdme{root,ug},name}}) \\
\snslabel{sfive}  & \fname{addAtt}(\hdme{ug,name}, \iqllist{\iqltuple{n,n} \mid \iqltuple{ug,n} \iqlin \hdme{\hdme{root,ug},name}}) \\
\snslabel{ssix}   & \fname{addAtt}(\hdme{ug,ppt}, \iqllist{\iqltuple{n,p} \mid \iqltuple{ug,n} \iqlin \hdme{\hdme{root,ug},name}; \iqltuple{ug,p} \iqlin \hdme{\hdme{root,ug},ppt}}) \\
\snslabel{sseven} & \fname{addRel}(\hdme{pg}, \iqllist{\iqltuple{n} \mid \iqltuple{pg,n} \iqlin \hdme{\hdme{root,pg},name}}) \\
\snslabel{seight} & \fname{addAtt}(\hdme{pg,name}, \iqllist{\iqltuple{n,n} \mid \iqltuple{pg,n} \iqlin \hdme{\hdme{root,pg},name}}) \\
\snslabel{snine}  & \fname{addRel}(\hdme{result}, \iqllist{\iqltuple{co,n} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code};\\
          & \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}}) \\
\snslabel{sten}   & \fname{addAtt}(\hdme{result,code}, \iqllist{\iqltuple{co,n,co} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code};\\
          & \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}}) \\
\snslabel{seleven}  & \fname{addAtt}(\hdme{result,name}, \iqllist{\iqltuple{co,n,n} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code};\\
            & \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}}) \\
\snslabel{stwelve}  & \fname{addAtt}(\hdme{result,year}, \iqllist{\iqltuple{co,n,y} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code};\\
            & \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \iqltuple{re,y} \iqlin \hdme{\hdme{root,result},year}}) \\
\snslabel{sthirteen}  & \fname{addAtt}(\hdme{result,grade}, \iqllist{\iqltuple{co,n,y} \mid \iqltuple{re,co} \iqlin \hdme{\hdme{root,result},code};\\
              & \iqltuple{re,n} \iqlin \hdme{\hdme{root,result},name}; \iqltuple{re,g} \iqlin \hdme{\hdme{root,result},grade}}) \\
\end{array}
\end{math}

\begin{math}
%\iqltuplelength{\extrarowheight}{1.5pt}
\begin{array}{ll}
\snslabel{sfourteen}  & \fname{addRel}(\hdme{course}, \iqllist{\iqltuple{co} \mid \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code}}) \\
\snslabel{sfifteen}   & \fname{addAtt}(\hdme{course,code}, \iqllist{\iqltuple{co,co} \mid \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code}}) \\
\snslabel{ssixteen}   & \fname{addAtt}(\hdme{course,dept}, \iqllist{\iqltuple{co,d} \mid \iqltuple{c,co} \iqlin \hdme{\hdme{root,course},code}; \\
              & \iqltuple{c,d} \iqlin \hdme{\hdme{root,course},dept}}) \\
%Shrinking Phrase
\snslabel{sseventeen} & \fname{deleteYattaNode}(\hdme{\hdme{root,result}, grade, string,?}, \\
                      & \iqllist{\iqltuple{re,g} \mid \iqltuple{co,n,g} \iqlin \hdme{result,grade};\ re \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{seighteen}  & \fname{deleteYattaNode}(\hdme{\hdme{root,result}, year, integer,\ }, \\
                      & \iqllist{\iqltuple{re,y} \mid \iqltuple{co,n,y} \iqlin \hdme{result,year};\ re \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{snineteen}  & \fname{deleteYattaNode}(\hdme{\hdme{root,result}, name, string,-}, \\
                      & \iqllist{\iqltuple{re,n} \mid \iqltuple{co,n} \iqlin \hdme{result};\ re \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{stwenty}    & \fname{deleteYattaNode}(\hdme{\hdme{root,result}, code, string,-}, \\
                      & \iqllist{\iqltuple{re,co} \mid \iqltuple{co,n} \iqlin \hdme{result};\ re \iqlin \iqllist{\mathrm{generateId}\ co \ n}}) \\
\snslabel{stwentyone} & \fname{deleteYattaNode}(\hdme{root, result, set,+}, \\
                      & \iqllist{\iqltuple{r,re} \mid \iqltuple{co,n} \iqlin \hdme{result};\ re \iqlin \iqllist{\mathrm{generateId}\ co \ n}; \iqltuple{r} \iqlin \hdme{root}}) \\
\snslabel{stwentytwo} & \fname{deleteYattaNode}(\hdme{\hdme{root,course}, dept, string,\ }, \\
                      & \iqllist{\iqltuple{c,d} \mid \iqltuple{co,d} \iqlin \hdme{course,dept}; \iqltuple{r} \iqlin \hdme{root};\ c \iqlin \iqllist{\mathrm{generateId}\ r \ co}}) \\
\snslabel{stwentythree} & \fname{deleteYattaNode}(\hdme{\hdme{root,course}, code, string,-}, \\
                        & \iqllist{\iqltuple{c,co} \mid \iqltuple{co} \iqlin \hdme{course}; \iqltuple{r} \iqlin \hdme{root}; \ c \iqlin \iqllist{\mathrm{generateId}\ r \ co}}) \\
\snslabel{stwentyfour}  & \fname{deleteYattaNode}(\hdme{root, course, set,+}, \\
                        & \iqllist{\iqltuple{r,c} \mid \iqltuple{co} \iqlin \hdme{course}; \iqltuple{r} \iqlin \hdme{root}; \ c \iqlin \iqllist{\mathrm{generateId}\ r \ co} }) \\
\snslabel{stwentyfive}  & \fname{deleteYattaNode}(\hdme{\hdme{root,student}, tutor, string,\ }, \\
                        & \iqllist{\iqltuple{s,t} \mid \iqltuple{n,t} \iqlin \hdme{student,tutor}; \iqltuple{r} \iqlin \hdme{root};\ s \iqlin \iqllist{\mathrm{generateId}\ r \ n}}) \\
\snslabel{stwentysix}   & \fname{deleteYattaNode}(\hdme{\hdme{root,student}, name, string,-}, \\
                        & \iqllist{\iqltuple{s,n} \mid \iqltuple{n} \iqlin \hdme{student}; \iqltuple{r} \iqlin \hdme{root}; \ s \iqlin \iqllist{\mathrm{generateId}\ r \ n}}) \\
\snslabel{stwentyseven} & \fname{deleteYattaNode}(\hdme{root, student, set,+}, \\
                        & \iqllist{\iqltuple{r,s} \mid \iqltuple{n} \iqlin \hdme{student}; \iqltuple{r} \iqlin \hdme{root}; \ s \iqlin \iqllist{\mathrm{generateId}\ r \ n} }) \\
\snslabel{stwentyeight} & \fname{deleteYattaNode}(\hdme{\hdme{root,pg}, name, string,-}, \\
                        & \iqllist{\iqltuple{pg,n} \mid \iqltuple{n} \iqlin \hdme{pg}; \iqltuple{r} \iqlin \hdme{root}; \ pg \iqlin \iqllist{\mathrm{generateId}\ r \ n}}) \\
\snslabel{stwentynine}  & \fname{deleteYattaNode}(\hdme{root, pg, set,+}, \\
                        & \iqllist{\iqltuple{r,pg} \mid \iqltuple{n} \iqlin \hdme{pg}; \iqltuple{r} \iqlin \hdme{root}; \ pg \iqlin \iqllist{\mathrm{generateId}\ r \ n} }) \\
\snslabel{sthirty}      & \fname{deleteYattaNode}(\hdme{\hdme{root,ug}, ppt, string,\ }, \\
                        & \iqllist{\iqltuple{ug,p} \mid \iqltuple{n,p} \iqlin \hdme{ug,ppt}; \iqltuple{r} \iqlin \hdme{root};\ ug \iqlin \iqllist{\mathrm{generateId}\ r \ n}}) \\
\snslabel{sthirtyone}   & \fname{deleteYattaNode}(\hdme{\hdme{root,ug}, name, string,-}, \\
                        & \iqllist{\iqltuple{ug,n} \mid \iqltuple{n} \iqlin \hdme{ug}; \iqltuple{r} \iqlin \hdme{root}; \ ug \iqlin \iqllist{\mathrm{generateId}\ r \ n}}) \\
\snslabel{sthirtytwo}   & \fname{deleteYattaNode}(\hdme{root, ug, set,+}, \\
                        & \iqllist{\iqltuple{r,ug} \mid \iqltuple{n} \iqlin \hdme{ug}; \iqltuple{r} \iqlin \hdme{root}; \ ug \iqlin \iqllist{\mathrm{generateId}\ r \ n} }) \\
\snslabel{sthirtythree} & \fname{deleteYattaNode}(\hdme{root, set}, \\
                        & \iqllist{\iqltuple{r} \mid (=)\ r\ 0 }) \\

\vspace{0.1cm}
\end{array}
\end{math}

\section{The AutoMed Repository for BAV Data Integration}\marginpar{Peter}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\label{repository-sec}

This is a Java API, that uses a RDBMS to provide a persistent storage
for data modelling language descriptions in the HDM, database schemas,
and transformations between those schemas \cite{BT01}. The repository
also provides some of the shared functionality that tools accessing
the repository may require.

The AutoMed repository has two logical components, assessed via one
API. The \keywordabbr{model definitions repository}{MDR} allows for
the description of how a data modelling language is represented as
combinations of nodes, edges and constraints in the HDM. It is used by
AutoMed `experts' to configure AutoMed so that it can handle a
particular data modelling language.  The \keywordabbr{schema
transformation repository}{STR} allows for schemas to be defined in
terms of the data modelling concepts in the MDR. It also allows for
transformations to be specified between those schemas. Most AutoMed
tools and users will be concerned with editing this repository, as new
databases are added to the AutoMed repository, or those databases
evolve \cite{MP02a}.


\begin{stdfigure}{Repository Schema}{rep-schema-fig}

\stdpicmag{-2}
\footnotesize

\begin{picture}(50,31)(-1,10)
\putcolourbox[bl,i](-1,10)(50,8){PCAone}{\textbf{\textsf{\large MDR}}}
\putcolourbox[tr,i](-1,22)(46,19){PCBone}{\textbf{\textsf{\large STR}}}
\entity(40,12){Model}
\relationship(40,14)(28,14){}{0:N}{1:1}
\entity(20,12){Construct}
\relationship(8,14)(20,14){}{1:1}{1:N}
\entity(0,12){Scheme}
\relationship(24,16)(24,24){}{0:N}{1:1}
\entity(20,24){Schema\\Object}
\relationship(28,26)(40,26){}{0:N}{}
\path(40,26)(40,30)
\putbox[l,i](40,28)(0,2){~1:2}
\putbox[l,i](40,34)(0,2){~2:2}
\entity(36,30){Transforma\-\\tion}
\relationship(28,38)(40,38){}{0:N}{}
\path(40,38)(40,34)
\relationship(8,26)(20,26){}{1:1}{0:N}
\entity(0,24){Object\\Scheme}
\relationship(24,28)(24,36){}{1:N}{1:N}
\entity(20,36){Schema}
\relationship(8,38)(20,38){}{1:1}{0:N}
\entity(0,36){Access\\Method}
\end{picture}

\end{stdfigure}

\subsection{Describing a Data Modelling Language in the MDR}\marginpar{Mike}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\label{mdr-sec} 

In \cite{MP99a} we proposed a general technique for the modelling of
any structured data modelling language in the HDM. This has been used
as a basis for the design of the MDR.
The AutoMed approach uses a repository to store descriptions of
schemas and transformations between them.  A schema is described by a
set of schema objects which represent objects from a particular
modelling language (or possibly more than one in some cases).  Each
schema object has a type (construct) which is described in the MDR.
This construct describes how an object of a given type can be
constructed in terms of strings and references to other schema objects
as well as the relationship of each constituent with HDM.  Schemas
(which are described using any modelling language constructs described
in the MDR) are therefore readily translatable into HDM.

It is our belief that any data modelling language can be modelled
using sets of values and constraints between these sets.  In HDM we
represent a set of values with a graph node and in a particular
instance of the schema we call the the set of members the extent of
the node.  As constraints describing relationships between extents are
so common we use edges to represent sets of values which are
constrained such that each member is a tuple where each value is drawn
from the extent of one of the nodes or edges which the edge connects.

We use schemes to textually describe schema objects\cite{PM98}.
The description of a schema object's construct (its type) is actually
a description of the construct's valid schemes.  Each construct must
be classified as being one of the following:


\subsubsection{Nodal}

A nodal construct represents a simple set of values.  Exactly one
element of the construct scheme must be identified as the name of the
resulting HDM node and be of type string (i.e. an scheme would have to
be supplied with the name of the node).  Typically a nodal construct
has just this one scheme element.  For example in a typical ER model
the construct scheme would be:

er:entity:$\hdms{(string)hdm\_node\_name}$

and the entities student and course in Figure~\ref{schema-eg-fig}(d) would be:

$\hdms{student}$

$\hdms{course}$


\subsubsection{Linkage}

A linking construct is one that can only be instantiated (i.e. a
schema object of its type be constructed) by referring to other schema
objects.  One scheme element may be identified as the resulting HDM
edge's name and at least two of the instance scheme's elements must
refer to other schema objects that have resulting HDM nodes or edges.
For example we may express er binary relationships with mandatory
names using the following construct scheme:

er:relationship:$\hdms{(string)name, (reference)er:entity 2:2}$

The lower and upper cardinality constraint means that when an
er:relationship is created, exactly two references to existing
entities must be supplied along with the name of the relationship
(1:1 is assumed when a construct scheme element isn't given a
cardinality).

The scheme instance for the relationship in Figure~\ref{schema-eg-fig}(d)
would then be:

$\hdms{result, student, course}$


\subsubsection{Link-Nodal}

A link-nodal construct is a combination of a link and a node.  It
models a node type which cannot exist in isolation but requires
another construct with which to be associated.  The construct scheme
must contain one string element for the name of the new HDM node, an
optional name for the HDM edge name and a mandatory reference to an
existing construct.  A typical example is an ER attribute:

er:attribute:$\hdms{(reference)er:entity, (string)new\_node\_name, (string)card}$

The attributes of student in Figure~\ref{schema-eg-fig}(d) would then have the following instance schemes:

$\hdms{student, name, 1:1}$ 
$\hdms{student, tutor, 1:N}$ 
$\hdms{student, ppt, 1:N}$ 

The cardinality constraints given in the instances are macros that
would expand into proper constraint sentences limiting the extent of
the attributes nodes accordingly.  We could have instead created a
cardinality constraint construct and referenced it in the construct
scheme and a ``1:1'' and a ``1:N'' instance of the constraint to be
referenced from the attribute instance schemes.


\subsubsection{Alternation}

An alternation doesn't have any corresponding modelling language
construct or HDM but is used to create a construct to be referenced by
other constructs that need alternatives.  Each construct scheme
element is an alternative for the construct scheme element that
references the alternation.

In Figure~\ref{schema-eg-fig}(d) we have two non-key attributes on the relationship
``results''.  If we allow attributes on relationships and don't want
to have two different constructs for attributes (one for entity
attributes and one for relationship attributes) then we could instead
define the target of an attribute as an alternation:
\\
\\
(alternation)\\
er:attribute\_target$\hdms{(reference)er:entity, (reference)er:relationship}$

Then we could define attribute as follows:
\\
\\
(link-nodal)\\
er:attribute$\hdms{(reference)er:attribute\_target, (string)new\_node\_name, (String)card)}$

Now we can define the attributes on student as before and define the
attributes on results as follows:

$\hdms{results, year, 1:N}$
$\hdms{results, grade, 1:N}$

The key attributes of result are contained in the extent of the edge
representing the relationship.


\subsubsection{Sequence}

Like an alternation, a sequence doesn't have any corresponding
modelling language construct or HDM but is used to create a construct
to be referenced by other constructs that need a repeatable sequence.
The sequence is just the construct scheme elements in order.

Suppose we want to be able to describe n-ary relationships rather than
just binary relationships.  If we didn't need cardinality constraints
then the relationship construct scheme given above could be
generalised to n-ary as follows:
\\
\\
(linkage)\\
er:relationship:$\hdms{(string)name, (reference)er:entity 2:N}$

However, if we were given a similar same binary relationship
definition which also included cardinality constraints:
\\
\\
(linkage)\\
er:relationship:$\hdms{(string)name, (reference)er:entity 2:2, (string)card 2:2}$

then we cannot generalise it thus:
\\
\\
(linkage)\\
er:relationship:$\hdms{(string)name, (reference)er:entity 2:N, (string)card 2:N}$

because this doesn't say what we want it to: that there be exactly one
cardinality constraint for each entity in the relationship.  We could
however construct an entity-constraint sequence and give that the
required cardinality.
\\
\\
(sequence)\\
er:entity\_role:$\hdms{(reference)er:entity, (string)cardinality, (string)role}$

Note that we have also added a string to describe the role of the
entity in the relationship here, this is diagrammatical and has no
effect on the resulting HDM.
\\
\\
(linkage) \\
er:relationship$\hdms{(string)name, (reference)er:entity\_role 2:N}$


Now we can use this definition to create our results relationship:

$\hdms{result, $\hdms{student, 1:N, obtained}$, $\hdms{course, 1:N, awarded}$ }$
We can now create relationships like this between as many entities as we like.

\label{relational-sec}





\subsection{Describing Schemas and Transformations in the STR}\marginpar{Peter}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{str-sec}

In AutoMed, each \textsf{Schema} contains a set of \textsf{SchemaObject}
instances, each of which must be based on a \textsf{Construct} instance
that exists in the MDR. It should be noted that there is not restriction
on which \textsf{Model} these constructs belong to, allowing a schema to
contain objects from more that one data modelling language. This allows
us to describe within AutoMed the mapping between different data modelling
languages. In particular, the
actions of a wrapper on a data source are described using AutoMed pathways.
For example,  \textsf{sc\_s3}
will convert the relational model in its AutoMed version using a sequence
of transformations:

\begin{snstranslist}

\end{snstranslist}

\begin{stdfigure}{Overview of Schemas held in AutoMed}{integration-schemas-fig}

\def\extschema(#1,#2)#3{\putcolourbox(#1,#2)(3,2){PCAtwo}{#3}}
\def\intschema(#1,#2)#3{\putcolourbox(#1,#2)(3,2){PCAone}{#3}}

\stdpicmag{2}

\begin{picture}(38,14)
\putcolourbox[t,i](-1,10)(40,4){lightgrey}{subnet3}
\putcolourbox[r,i](-1,5)(30,4){lightgrey}{subnet2}
\putcolourbox[b,i](-1,0)(40,4){lightgrey}{subnet1}
\extschema(0,1){sc\_s1}
\intschema(5,1){sc\_s1a}
\intschema(10,1){sc\_s1b}
\intschema(20,1){sc\_s1u}
\intschema(25,1){sc\_s1t}
\intschema(25,1){sc\_s1v}
\intschema(35,1){sc\_s1zzg}
\extschema(0,6){sc\_s2}
\intschema(5,6){sc\_s2a}
\intschema(10,6){sc\_s2b}
\intschema(20,6){sc\_s2s}
\extschema(0,11){sc\_s3}
\intschema(5,11){sc\_s3a}
\intschema(10,11){sc\_s3b}
\intschema(35,11){sc\_s3zxxxxx}
\end{picture}

\end{stdfigure}


In a large data integration, there will be many schemas produced as
intermediate steps in the process of mapping one data source to
another.  At first this would appear to make the BAV approach
unworkable, since there are so many versions of schemas being
kept. The AutoMed approach addresses this issue by distinguishing
between \keyword{extensional} and \keyword{intensional}
representations of schemas. Each data source will be represented in
the AutoMed repository by describing its schema as a set of schema
objects, which is the \keyword{extensional} representation of
schemas. Then transformations produce new intensional schemas, for
which the schema objects are not stored, but which can be derived by
applying transformation rules in sequence to an extensional schema.

Futhermore, a special transformation called \keyword{ident} is
introduced, which states that two schemas have the same logical set of
schema objects, but that they are derived from distinct extensional
schemas.  For example, in Figure~\ref{integration-schemas-fig}, the
schema \textsf{sc\_s1u} is derived from \textsf{sc\_s1}, and schema
\textsf{sc\_s2s} (identical to \textsf{sc\_s1u}) is derived from
\textsf{sc\_s2}. The set of schemas connected by transformations other
than ident is called a \keyword{subnet}. By the nature of this
arrangement, Each subnet can exist identpendently of other
subnets. This means that a subnet can be created, editted, connected
to other subnets via \textsf{ident} transformations, and deleted all
without changing anything in another subnet.


\section{Inter Model Transformations}\marginpar{Mike}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{inter-model-sec}

\section{Template Transformations}\marginpar{Nikos}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{template-sec}

Schema integration in the AutoMed framework frequently relies on the
reuse of specific sequences of primitive transformations.  These
sequences are called \keyword{composite} transformations and resemble
well-known equivalences between schemas \cite{Lar89,MP98}, e.g.  the
equivalence between a relation with a mandatory attribute and a
relation with specializations for each instance of the mandatory
attribute.  In the example of Section~\ref{bav-sec}, the first step in
the transition from schema \schemaname{S_3} to the global schema
(Table~\ref{s3Tosg-table}) is the execution of a composite
transformation that resembles the aforementioned equivalence and
transforms relation \textsf{student} with mandatory attribute
\textsf{level} and values \textsf{ug},\textsf{pg} into relation
\textsf{student} with specializations \textsf{ug} and \textsf{pg}.

%\caption{Transformation pathway from \schemaname{S_3} to the Global Schema}
%\label{s3Tosg-table}

The disadvantage of composite transformations is that they are schema
and data dependent; an expert user needs to define their primitive
transformation steps in every detail for every schema and every
situation. For example, in an undergraduate student database a
composite transformation needs to be defined that transforms a relation
\textsf{student} with mandatory attribute <dept> to relation \textsf{student} with a
specialization relation for the students of each department. This
composite transformation defines a different transformation sequence
from the previous example, but resembles the same notion of schema
equivalence.

In order to solve this problem of rewriting similar composite
transformations and in order to reduce user effort in data
integration, we have created a framework that enables the definition
of \keyword{template} transformations that are schema and data
independent and can therefore be reused in different situations.
Based on this framework, a template transformation is a parameterised
definition of a composite transformation.  These parameters are
instantiated in each template transformation execution based on the
schema that the template is performed upon, the underlying data and
the desired outcome. Thus, a template transformation can be applied on
any schema by appropriately setting its parameters.  In our example,
the parameters of the template transformation that decomposes a
relation to its specializations are: (a) the schema that the template
transformation is going to be performed upon, (b) the existing
relation, (c) its mandatory attribute, (d) the specialization
relations that are going to be added to the schema and (e) the
instances of the mandatory attribute that correspond to the
specializations.

In addition to the parameters, the template transformation definition
includes a sequence of statements that define the primitive
transformations that are going to be applied on the initial schema
when the template is performed.  The most significant statements in
our framework are <primitive transformation>, which defines a
primitive transformation like the ones in Section~\ref{mdr-sec}, and
<foreach>, which is an iterative statement.

Figure~\ref{template-def-fig} illustrates the template of the previous
examples. Initially, the parameters of the template transformation are
defined \footnote{The first parameter of every template transformation
  defines the schema that transformation is going to be performed upon
  and it is always implicitly defined}; every parameter has a
description and a \textit{type}. In the main body, the template's
sequence of statements is defined.  In this case, the sequence of
statements consists of a <foreach> statement and a <delete> statement.
The <foreach> iterates the list of names for the specialization
relations and the corresponding values of the mandatory attribute and
in each iteration it adds a specialization and its primary key. The
<delete> statement deletes the mandatory attribute from the existing
relation.

Figure~\ref{template-exec-fig} illustrates the invocation of the
template transformation on schema \schemaname{S_3}. The execution produces the
transformations
\snsref{s3TosgaddRelug}-\snsref{s3TosgdelAttstudentlevel} in
Table~\ref{s3Tosg-table}. The rest of the transformation pathway
\snsref{s3TosgaddAttugppt}-\snsref{s3TosgdelAttstudentppt} can be
produced by another template transformation that removes an attribute
from a relation and moves it down to its specialization relations.



\begin{stdfigure}{Template Transformation Definition}{template-def-fig}
\begin{verbatim}
INPUTS();
  OBJECT parentTable = askForObject("Existing parent relation", table);
  OBJECT mandatoryField = askForObject("Mandatory attribute", field);
  OBJECT parentPrimaryKey = askForObject("Primary key of parent relation", field);
  NAMELIST specializationTableNames = askForNameList("Names of specializations");
  NAMELIST descriptiveInstances = askForNameList("Values of the attribute ...", 
      SIZEOF(specializationTableNames));

START();
  FOREACH(); NAME specializationTableName = IN (specializationTableNames);
             NAME descriptiveInstance = IN (descriptiveInstances);
             OBJECT parentfield = VARIES_WITH(mandatoryField);
             OBJECT parent = VARIES_WITH(parentTable);
             OBJECT primaryKey = VARIES_WITH(parentPrimaryKey);

    DO();
      ALIAS(specializationTableName, "specializationTableName");
      ALIAS(descriptiveInstance, "descriptiveInstance");
      ALIAS(parentfield, "parentfield");
      ALIAS(parent, "parent");
      ALIAS(primaryKey,"primarykey");

      FUNCTION tableExtent = DEFINE_FUNCTION("[ {x} | 
              {x,y} <- @parentfield?scheme; y='@descriptiveInstance']");
      OBJECT newSpecialization= ADD( CONSTRUCT.IS(table), 
          SCHEME.IS(new Object[]{my(specializationTableName)}),
          FUNCTION.IS(tableExtent));
  
  
      FUNCTION fieldExtent = DEFINE_FUNCTION("[ {x,y} | 
              {x,y} <- @primarykey?scheme; {x,z} <- @parentfield?scheme; 
          z='@descriptiveInstance']");
      NAME fieldName = DEFINE_NAME("@primarykey?name");
      OBJECT newField = ADD( CONSTRUCT.IS(field),
          SCHEME.IS(new Object[]{newSpecialization, my(fieldName)}),
          FUNCTION.IS(fieldExtent));
  
      OBJECTLIST primaryKeyFields = (OBJECTLIST)CREATE_LIST(new ITEM[]{newField});
      NAME primarykeyName = DEFINE_NAME("@(specializationTableName)_pk");
      ADD( CONSTRUCT.IS(primarykey),
          SCHEME.IS(new Object[]{my(primarykeyName),newSpecialization, 
                                        primaryKeyFields}));
  
      ALIAS(newSpecialization,"specialization");
      STRING partialQuery= DEFINE_STRING("[ {x,y}|{x} <- @specialization?scheme; 
              y='@descriptiveInstance']");
      STRINGLIST partialQueriesForMandatoryField = (STRINGLIST)COLLECTS(partialQuery);                          
  
  ENDFOREACH();

  STRING query = DEFINE_STRING(partialQueriesForMandatoryField, "++");
  ALIAS(query,"query");
  FUNCTION mandatoryFieldExtent = DEFINE_FUNCTION("@query");
  mandatoryField.DELETE(mandatoryFieldExtent);
END();
\end{verbatim}
\end{stdfigure}

\begin{stdfigure}{Template Transformation Invocation}{template-exec-fig}
\begin{verbatim}
        SchemaObject student=s3.getSchemaObject("<\textsf{student}>");
        SchemaObject studentlevel = s3.getSchemaObject("<<student,level>>");
        SchemaObject studentname = s3.getSchemaObject("<<student,name>>");

        CompositeTransformation template1 = 
            CompositeTransformation.getCompositeTransformation(
                "Field to Hierarchy Equivalence");
        TTemplateInputArgumentList argumentList1 = 
            new TTemplateInputArgumentList(template1);
        argumentList1.addArgument(s3);
        argumentList1.addArgument(student);
        argumentList1.addArgument(studentlevel);
        argumentList1.addArgument(studentname);
        argumentList1.addArgument(new String[]{"ug","pg"});
        argumentList1.addArgument(new String[]{"ug","pg"});
        CompositeTransformationRecord intermediateRecord = 
            argumentList1.executeTransformation();
        Schema intermediateSchema = intermediateRecord.getFinalSchema();
\end{verbatim}
\end{stdfigure}


Our template transformation framework except from enabling the
definition of template transformations that are schema and data
independent, it is additionally model-independent. Template
transformations can be written for any modelling language by
specifying the correct types for the templates' parameters.  For
example, a template transformation can be written for YATTA schemas,
e.g. \textit{flattening} YATTA subtrees by moving complex nodes one
level up in the YATTA hierarchy, as in the pathways $S_1 \rightarrow
S_{1,2}$ and $S_2 \rightarrow S_{1,2}$ (Section~\ref{}).  Intermodel
templates can also be defined, e.g. transforming a YATTA schema into a
relational one.
 
More details about template transformations can be found in
\cite{Riz01,Lan03}.


\section{Schema Matching}\marginpar{Nikos}
%%%%%%%%%%%%%%%%%%%%%%%%%
\label{schema-matching-sec}


In all the examples seen so far, an expert user specifies the
primitive transformations to be applied on the available schemas and
integrate the underlying data sources. The key issue, as in every
manual data integration framework, is the identification of the
existing semantic relationships between the elements of the schemas
\cite{Kas96}, and based on these relationships the appropriate
primitive or template transfrormations can be performed.

The process of discovering semantic relationships between schema
elements is called \keyword{schema matching}.  \marginpar{remove this
  paragraph?}  Manual schema matching requires intense user-effort,
thus ways of automating the procedure have been considered and
proposed in the literature \cite{RB01}. Most of the existing
methodologies are focused on discovering equivalence relationships
between schema elements \cite{DMDH02,LC00,MBR01}, or \keyword{direct
  matches}, but in many cases more expressive relationships exist
between elements, which yield \keyword{indirect matches} \cite{XE03}.

In our framework, we define five types of semantic relationships
between schema elements based on the comparison of the elements'
\textit{intentional domains}, i.e. the sets of real-world entities
represented by the elements. Our relationships are similar to the ones
described in \cite{Lar89} but our definitions differ:

\begin{enumerate}
\item \textbf{equivalence}:
Two schema elements \smalltextsf{A} and \smalltextsf{B} are equivalent, $\mathsf{A = B}$, iff 

\begin{tabular}{l}
$Dom_{int}(\textsf{A}) = Dom_{int}(\textsf{B})$
\end{tabular}

\item \textbf{subsumption}:
Schema element \smalltextsf{A} subsumes schema element \smalltextsf{B}, $\mathsf{B \subset A}$, iff 

\begin{tabular}{l}
$Dom_{int}(\textsf{B}) \subset Dom_{int}(\textsf{A})$
\end{tabular}

\item \textbf{overlapness}:
Two schema elements \smalltextsf{A} and \smalltextsf{B} are overlapping, $\mathsf{A \overlap B}$, iff 

\begin{tabular}{l}
$Dom_{int}(\textsf{A}) \cap Dom_{int}(\textsf{B}) \neq \emptyset,$\\
$\exists \textsf{C}: Dom_{int}(\textsf{A}) \cap Dom_{int}(\textsf{B}) = Dom_{int}(\textsf{C})$ 
\end{tabular}


\item \textbf{disjointness}:
Two schema elements \smalltextsf{A} and \smalltextsf{B} are disjoint, $\mathsf{A \disjoint B}$, iff

\begin{tabular}{l}
$Dom_{int}(\textsf{A}) \cap Dom_{int}(\textsf{B}) = \emptyset,$ \\
$\exists \textsf{C}: Dom_{int}(\textsf{A}) \cup Dom_{int}(\textsf{B}) \subseteq Dom_{int}(\textsf{C})$ 
\end{tabular}

\item \textbf{incompatibility}:
Two schema elements \smalltextsf{A} and \smalltextsf{B} are incompatible, $\mathsf{A \neq B}$, iff

\begin{tabular}{l}
$Dom_{int}(\textsf{A}) \cap Dom_{int}(\textsf{B}) = \emptyset,$ \\
$\neg \exists \textsf{C}: Dom_{int}(\textsf{A}) \cup Dom_{int}(\textsf{B}) \subseteq Dom_{int}(\textsf{C})$ 
\end{tabular}
 
\end{enumerate}  

It is important to notice that element \smalltextsf{C} in the
definition of overlapness and disjointness may or may not exist in the
existing schemas. The notation $\exists\smalltextsf{C}:condition$
means that there is a real-world concept that can be represented by an
existing or non-existing schema element \smalltextsf{C} that satisfies
the $condition$. The notation $\neg\exists \smalltextsf{C}:condition$
in the definition of incompatibility means that there is no real-world
concept that would be represented by a schema element \smalltextsf{C}
to satisfy the specified $condition$.

In our example in Figure~\ref{schema-eg-fig}, a direct match exists
between the \keyword{equivalent} \textsf{tutor} nodes in schemas \schemaname{S_1}
and \schemaname{S_2} because they both represent the same set of tutors
\marginpar{constraint must be defined when schemas are defined!}
\footnote{The reader is reminded that according to the constraints a
member of staff has to tutor at least one undergraduate and one
postgraduate student}. An indirect match can be identified between the
\textsf{student} nodes in $S_1,S_2$ and \textsf{student} in \schemaname{S_3},
since both undergraduates ($S_1.student$) and postgraduates
($S_2.student$) are \keyword{subsumed} by all students
($S_3.student$).  Other indirect matches are between the
\textsf{course} nodes in $S_1,S_2$ which \keyword{overlap} (both
undergraduates and postgraduates can attend a common set of courses)
and between the \textsf{student} nodes in $S_1,S_2$ which are
<disjoint>. Based on these relationships, the transformation pathways
$S_1 \rightarrow S_{1,2}$ and $S_2 \rightarrow S_{1,2}$ merge the
\textsf{tutor}s and union the undergraduates and postgraduates into
\textsf{student}.

Currently, we have proposed a methodology to automatically discover
the aforementioned semantic relationships. Our objective is the
discovery of these relationships based on the elements' data and
metadata, without relying on external knowledge \cite{HG02,BCVMV98} or
user knowledge and schema structure \cite{PTU00,XE03}.

In our automatic schema matching approach a bidirection comparison of
schema elements is performed, which has been motivated by the fact
that a bidirectional comparison of the elements' intentional domains
can be used to identify equivalence, subsumption and overlapness
relationships. This is depicted by the following formula:

$d(X,Y) = \frac{|Dom_{int}(X) \cap Dom_{int}(Y)|}{|Dom_{int}(X)|}$, 

where $X,Y$ are schema elements and $|Z|$ defines the number of
entities in set $Z$.  This formula gives $d(X,Y)=d(Y,X)=1$ when $X,Y$
are equivalent, $d(X,Y)=1$ and $0<d(Y,X)<1$ when $Y$ subsumes $X$, and
$0<d(X,Y)<1, 0<d(Y,X)<1$ when $X,Y$ are overlapping. The problems with
this approach is that the disjointness and incompatibility
relationships cannot be distinguished and that the
\textit{bidirectional similarity degrees} $d(X,Y), d(Y,X)$ cannot be
automatically computed since a comparison of the elements' real-world
entities is required.

We attempt to resemble the above formula by examining the elements
instances and their metadata.  The equivalence, subsumption and
overlapness relationships can still be discovered as explained
previously, now, however, the similarity degrees are fuzzier, e.g.
$d(X,Y)$ and $d(Y,X)$ are unlikely to have values equal to 1 when $X$
and $Y$ are equivalent, but they will be above an equivalence
threshold.  Disjointness can also be discovered since disjoint
elements will exhibit similarity in their instances and metadata,
arising from their relationship with the same \textit{super} element.
Thus, disjoint pairs of elements will have higher similarity degrees
that incompatible pairs.

\begin{stdfigure}{Architecture}{architecture-fig}
\stdpicmag{-2}
\begin{picture}(34,34)

\pathvector(6,5)(8,5)
\put(4,4.5){\schemaname{S_1}}

\pathvector(6,3)(8,3)
\put(4,2.5){\schemaname{S_2}}

\putframebox(8,2)(18,4){}

\putcolourbox(8.5,2.5)(8,3){lightgrey}{}
\putframebox(9,3)(2,2){m}
%\put(10,5){\vector(0,1){6}}
\path(10,5)(10,7)
\put(10,9){\vector(0,1){2}}

\put(11.5,4){\ldots}

\putframebox(14,3)(2,2){m}
%\put(15,5){\vector(0,1){6}}
\path(15,5)(15,7)
\put(15,9){\vector(0,1){2}}

\put(11.5,10){\ldots}


\putcolourbox(17.5,2.5)(8,3){midgrey}{}
\putframebox(18,3)(2,2){m}
%\put(19,5){\vector(0,1){6}}
\path(19,5)(19,7)
\put(19,9){\vector(0,1){2}}

\put(20.5,4){\ldots}

\putframebox(23,3)(2,2){m}
%\put(24,5){\vector(0,1){6}}
\path(24,5)(24,7)
\put(24,9){\vector(0,1){2}}
\put(20.5,10){\ldots}


%\putcolourbox(9,7)(16,2){white}{\scriptsize{partial bidirectional similarity degrees}}
\putbox(9,7)(16,2){\scriptsize{partial bidirectional similarity degrees}}

\putframebox(9,11)(16,2){\normalsize Filter}
%\pathvector(3,12)(5,12)
%\putbox(0,11)(4,2){\scriptsize{filter}\\ \scriptsize{strategy}}
%\put(10,13){\vector(0,1){5}}
%\put(15,13){\vector(0,1){5}}
%\put(11.5,16.5){\ldots}
%\put(19,13){\vector(0,1){5}}
%\put(24,13){\vector(0,1){5}}
%\put(20.5,16.5){\ldots}

\path(10,13)(10,14)
\put(10,16){\vector(0,1){2}}
\path(15,13)(15,14)
\put(15,16){\vector(0,1){2}}
\put(11.5,16.5){\ldots}
\path(19,13)(19,14)
\put(19,16){\vector(0,1){2}}
\path(24,13)(24,14)
\put(24,16){\vector(0,1){2}}
\put(20.5,16.5){\ldots}

%\putcolourbox(9,14)(16,2){white}{\scriptsize{bidirectional similarity degrees of compatible elements}}
\putbox(9,14)(16,2){\scriptsize{bidirectional similarity degrees of compatible elements}}

\putframebox(9,18)(16,2){\normalsize Aggregator}
%\pathvector(3,19)(5,19)
%\putbox(0,18)(4,2){\scriptsize{aggregation}\\ \scriptsize{strategy}}

%\put(17,20){\vector(0,1){6}}
\path(17,20)(17,21)
\put(17,24){\vector(0,1){2}}
\putbox(10,21)(13,3){\scriptsize{aggregated bidirectional} \\\scriptsize{similarity degrees}}

\putframebox(9,26)(16,4){\normalsize Degree Combinator}
\pathvector(7,28)(9,28)
\putbox(3,28)(3,2){\scriptsize{thresholds}}

\pathvector(17,30)(17,32)
\putbox(11,32)(13,2){\scriptsize{semantic relationships}}

\path(24,16)(30,16)(30,21)
\path(30,24)(30,29)
\put(24,16){\circle*{0.5}}
\path(19,17)(29,17)(29,21)
\path(29,24)(29,27)
\put(19,17){\circle*{0.5}}
\pathvector(30,29)(25,29)
\pathvector(29,27)(25,27)

\putbox(28,21)(5,3){\scriptsize{clarification} \\\scriptsize{similarity degrees}}

%\putcolourbox(11,3)(2,2){lightgrey}{}
%\putframebox(11,3)(2,2){\scriptsize  m}
\put(-5,0){\scriptsize{relationship identification modules}}
\dashpathvector(10,1)(12,3)

%\putcolourbox(11,0)(2,2){midgrey}{}
%\putframebox(11,0)(2,2){\scriptsize  m}
\put(18,0){\scriptsize{relationship clarification modules}}
\dashpathvector(23,1)(22,3)
\end{picture}
\end{stdfigure}

Our schema matching architecture (Figure~\ref{architecture-fig})
consists of several comparison modules that exploit different types of
information to compute bidirectional similarity degrees of schema
elements. Our currently implemented modules compare element names,
instances, statistical data over the instances, data types, value
ranges and lengths.  There are two types of modules:
\textit{relationship identification} and \textit{relationship
  clarification} modules. Relationship identification modules attempt
to discover compatible pairs of elements and relationship
clarification modules attempt to specify the type of the semantic
relationship in each compatible pair.

Initially in the schema matching process, the bidirectional similarity
degrees produced by the modules are combined by the Filter, using the
average aggregation strategy, to separate the compatible from the
incompatible pairs of elements. Then, the Aggregator component
combines the similarity degrees of the compatible elements using the
product aggregation strategy and indicates their semantic
relationships.  The output of the Aggregator becomes the input of the
Degree Combinator, which based on (a) the relationship clarification
modules, (b) the fuzzy thresholds and (c) the previous discussion on
the values of the similarity degrees, it outputs the discovered
semantic relationships.  The user is then able to validate or reject
these relationships and proceed to the data integration process.

More details about the implemented tool and an evaluation of it can be
found in \cite{Riz03b}.


\section{Conclusions}\marginpar{Peter}
%%%%%%%%%%%%%%%%%%%%%


\bibliography{database}

\end{document}







