Use opennlp to customize named entities

  nlp

Order

This article mainly studies how to use opennlp to customize named entities, label training and model application.

maven

        <dependency>
            <groupId>org.apache.opennlp</groupId>
            <artifactId>opennlp-tools</artifactId>
            <version>1.8.4</version>
        </dependency>

Practice

Training model

        // train the name finder
        String typedEntities = "<START:organization> NATO <END>\n" +
                "<START:location> United States <END>\n" +
                "<START:organization> NATO Parliamentary Assembly <END>\n" +
                "<START:location> Edinburgh <END>\n" +
                "<START:location> Britain <END>\n" +
                "<START:person> Anders Fogh Rasmussen <END>\n" +
                "<START:location> U . S . <END>\n" +
                "<START:person> Barack Obama <END>\n" +
                "<START:location> Afghanistan <END>\n" +
                "<START:person> Rasmussen <END>\n" +
                "<START:location> Afghanistan <END>\n" +
                "<START:date> 2010 <END>";
        ObjectStream<NameSample> sampleStream = new NameSampleDataStream(
                new PlainTextByLineStream(new MockInputStreamFactory(typedEntities), "UTF-8"));

        TrainingParameters params = new TrainingParameters();
        params.put(TrainingParameters.ALGORITHM_PARAM, "MAXENT");
        params.put(TrainingParameters.ITERATIONS_PARAM, 70);
        params.put(TrainingParameters.CUTOFF_PARAM, 1);

        TokenNameFinderModel nameFinderModel = NameFinderME.train("eng", null, sampleStream,
                params, TokenNameFinderFactory.create(null, null, Collections.emptyMap(), new BioCodec()));

Opennlp uses < START > and < END > to customize labeling entities, and names entities are marked with colons after START, such as < START:person >

Parameter description

  • ALGORITHM_PARAM

On the engineering level, using maxent is an excellent way of creating programs which perform very difficult classification tasks very well.

  • ITERATIONS_PARAM

number of training iterations, ignored if -params is used.

  • CUTOFF_PARAM

minimal number of times a feature must be seen

Use model

After training the model above, the model can be used for analysis.

      NameFinderME nameFinder = new NameFinderME(nameFinderModel);

        // now test if it can detect the sample sentences

        String[] sentence = "NATO United States Barack Obama".split("\\s+");

        Span[] names = nameFinder.find(sentence);

        Stream.of(names)
                .forEach(span -> {
                    String named = IntStream.range(span.getStart(),span.getEnd())
                            .mapToObj(i -> sentence[i])
                            .collect(Collectors.joining(" "));
                    System.out.println("find type: "+ span.getType()+",name: " + named);
                });

The output is as follows:

find type: organization,name: NATO
find type: location,name: United States
find type: person,name: Barack Obama

Summary

The annotation of custom named entities in opennlp gives a certain customization space, which is convenient for developers to customize special named entities in their respective fields, so as to improve the accuracy of word segmentation for specific named entities.

doc