Monday 14 November 2011

Stop Word Listing

Stop words in computation domain are the terms which need to be removed before natural language text(data) processing because they do not make any significant sense.

A precise definition:

“Words that do not appear in the index in a particular database because they are either insignificant (i.e., articles, prepositions) or so common that the results would be higher than the system can handle (as in the case of IUCAT where terms such as United States or Department are stop words in keyword searching.) Stop words vary from system to system. Also, some systems will merely ignore stop words where use of stop words in other systems will result in retrieving zero hits. ”

You have to build your own stop word (manually) as per the use of it, suppose you want to build a topic Canonicalization then stop word list could be quite big. For topics you should remove negations like "not","nothing"; and you should remove question tokens like "why", "what","how". But if you are building Canonicalization for questions, then you should keep negations and question tokens.

Stop words for Topic Canonicalization
------------------------------------
a
about
above
across
again
against
all
almost
alone
along
already
also
although
always
am
among
an
and
another
any
anybody
anyone
anything
anywhere
are
area
areas
around
as
ask
asked
asking
asks
at
away
b
backed
backing
backs
be
became
because
become
becomes
been
began
behind
being
beings
best
better
between
big
both
but
by
c
came
can
cannot
case
cases
certain
certainly
clear
clearly
come
communityid
could
d
did
differ
do
does
done
down
downed
downing
downs
e
each
early
either
end
ended
ending
ends
enough
even
evenly
ever
every
everybody
everyone
everything
everywhere
f
face
faces
fact
facts
far
felt
few
find
finds
first
for
four
from
full
fully
further
furthered
furthering
furthers
g
gave
general
generally
get
gets
gif
give
given
gives
go
going
good
goods
got
great
greater
greatest
group
grouped
grouping
groups
h
had
has
have
having
he
her
here
herself
high
higher
highest
him
himself
his
how
however
i
icon
if
im
important
in
interest
interested
interesting
interests
into
is
it
its
itself
j
just
k
keep
keeps
kind
knew
know
known
knows
l
large
largely
last
later
latest
least
less
let
lets
like
likely
long
longer
longest
m
made
make
making
man
many
may
me
member
members
men
might
more
most
mostly
mr
mrs
much
must
my
myself
n
necessary
need
needed
needing
needs
never
new
newer
newest
next
no
nobody
non
noone
not
nothing
now
nowhere
number
numbers
o
of
off
often
older
oldest
on
once
one
only
open
opened
opening
opens
or
ordered
ordering
orders
other
others
our
out
over
p
part
parted
parting
parts
per
perhaps
pl
place
places
pls
plz
pointed
pointing
possible
presented
presenting
presents
put
puts
q
quite
r
rather
really
regname
right
room
rooms
s
said
same
saw
say
says
second
seconds
see
seem
seemed
seeming
seems
sees
several
shall
she
should
show
showed
showing
shows
side
sides
since
small
smaller
smallest
so
some
somebody
someone
something
somewhere
state
states
still
such
sure
t
take
taken
than
that
the
their
them
then
there
therefore
these
they
thing
things
think
thinks
this
those
though
thought
thoughts
three
through
thus
to
today
together
too
took
toward
turn
turned
turning
turns
two
u
under
until
up
upon
us
use
used
uses
v
very
w
want
wanted
wanting
wants
was
way
ways
we
well
wells
went
were
what
when
where
whether
which
while
who
whole
whose
why
will
with
within
without
work
worked
working
works
would
x
y
yet
you
young
younger
youngest
your
yours
yr
z

Stop word for Question Canonicalization
----------------------------------------------
a
about
above
across
again
against
all
almost
alone
already
also
although
always
am
among
an
and
any
anybody
anyone
anything
anywhere
are
area
areas
around
as
ask
at
away
b
backed
backing
backs
be
became
because
become
becomes
been
being
beings
best
better
but
by
c
came
cannot
communityid
d
differ
do
does
done
downed
downing
downs
e
each
either
end
ended
ending
ends
enough
even
evenly
ever
every
everybody
everyone
everything
everywhere
f
fact
facts
far
felt
few
find
finds
first
for
four
from
full
fully
further
furthered
furthering
furthers
g
general
generally
get
gets
gif
give
given
gives
greater
greatest
h
had
has
have
having
he
her
here
herself
him
himself
his
however
i
icon
if
im
important
in
into
is
it
its
itself
j
just
k
keep
keeps
l
large
largely
last
later
latest
least
less
let
lets
m
many
may
me
might
more
most
mostly
mr
mrs
much
must
my
myself
n
needed
needing
needs
new
newer
newest
nobody
noone
now
nowhere
o
of
off
on
one
or
others
our
out
p
parted
parting
per
perhaps
pl
please
pls
plz
put
puts
q
quite
r
rather
really
regname
s
said
same
saw
say
says
see
seem
seemed
seeming
seems
sees
several
she
so
some
somebody
someone
somewhere
such
sure
t
that
the
their
them
then
there
therefore
these
they
this
those
though
three
through
thus
to
today
too
toward
u
up
upon
us
v
very
w
wanted
wanting
was
we
well
wells
went
were
will
with
would
x
y
yet
you
your
yours
yr
z

No comments: