Dbpedia-spotlight API usage limit

Ok, thanks. I’m now working with the Portuguese docker image and it is working fine.

Hello all,

I am making many requests (~10k) to the english api (http://api.dbpedia-spotlight.org/en/annotate) and receiving a lot of 403s… I’ve realized that the percentage of errors I get is inversely proportional to the interval between requests. I am also getting some 502 errors from time to time.

I hope you find this comment useful and that this issue can be solved soon.

Best,
Lucas.

@lucas0,
@JulioNoe has reworked the spotlight docker, which works for @mwon. We will deploy it on the server as a replacement and maybe you can test it for us. We will ping you then.

I’d be happy to do so.

@kurzum
@mwon
Seems like I’ve managed to run the docker image:

$sudo docker ps
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS                  NAMES
47b3469ea4fa        dbpedia/spotlight-english   "/bin/sh"           6 minutes ago       Up 6 minutes        0.0.0.0:2222->80/tcp   focused_wu

but when I do

curl http://0.0.0.0:2222/rest/annotate?text=COOPER+Has+the+FBI+said+anything+about+a+reward+or+anything+%5C%3F+Because+there+was+that+there+was+a+reward+for+finding+her&confidence=0.5

I get:

curl: (56) Recv failure: Connection reset by peer

what is the problem? Is there a wiki or instructions on how to send requests to the docker?

Best,
Lucas.

Update: getting 400s now when requesting to the api (not the docker)

Hi @lucas0,

An example of curl to query the dbpedia docker version is as follows:

curl http://localhost:2222/rest/annotate --data-urlencode "text=President Obama called Wednesday on Congress to extend a tax break for students included in last year's economic stimulus package, arguing that the policy provides more generous assistance." --data "confidence=0.35" -H "Accept: text/turtle"

where “Accept:text/turtle” returns a nif output but if you want a json output you must replace it for “Accept: application/json”

It could be nice if help us to try the dbpedia/spotlight-databus version. This version allows you to locally download more than one language, e.g., English (en), German (de), etc.

The docker version is another way to run DBpedia Spotlight and it follows the same instructions as the DBpedia Spotligh Github project. Thanks for the observation, I will add a “how to run” to the docker instructions.

My best regards

Julio

Hi @mwon, @lucas0

I want to ask you for your help and support to test a new docker version of the DBpedia Spotlight. I know that you are interested into using a specific version of DBpedia Spotlight (Portuguese and English) then, it could be very useful if you can use the dbpedia/spotlight-databus docker version, in particular the v.0.2 version. Additional to your interested languages, you could try any of the available languages in this link.

docker pull dbpedia/spotlight-databus:v.0.2

If you found any issue/bug or have any question, please post it here. Thanks in advance.

My best regards

Julio

Ok, I tried to follow the instructions but not really sure if I’m doing it correctly. I run:

docker volume create spotlight-databus

followed by:

docker run --name spotlight-databus --mount source=spotlight-databus,target=/opt/spotlight -i -p 2222:80 dbpedia/spotlight-databus:v.0.2 spotlight.sh ln -l pt

But got this message:

Possible options:
-a Available models
-l Define the model language
-r Remove a model language (if exists)

and no docker is running

@mwon, sorry it was an error the “ln” don’t have to be there. I already update the instructions in the docker page, thanks.

docker run --name spotlight-databus --mount source=spotlight-databus,target=/opt/spotlight -i -p 2222:80 dbpedia/spotlight-databus:v.0.2 spotlight.sh -l pt

if the following error is shown:

docker: Error response from daemon: Conflict. The container name "/spotlight-databus" is already in use by container "24884ad77682ee1af27cad41286eb8b886602b9ef902164e8a84cfe3c6dc69de". You have to remove (or rename) that container to be able to reuse that name.

just run

 docker rm spotlight-databus

Thanks.

My best regards

Julio

@JulioNoe
I’d be happy to help. I’ve finally found what was wrong on my docker and the reason why i was getting error 56…

I was able to put the docker on with sudo docker run -i-p 2222:80 dbpedia/spotlight-english

but since there was no entry-point created by the docker itself, it is imperative that i run the spotlight.sh bash file so the entry-point is created. Still, I was getting the same error…

Another thing was that, for some reason the docker wouldnt stay on for more than a wee bit on my local mac machine (it wasnt the same on an ubuntu 18.04 i was working through shh, where it stayed listed with docker ps for a longer while), so adding the --restart unless-stopped flag made it work. Still not completely sure why.

I’ll try to set up the databus with the instructions you’ve placed here.

Best,
Lucas.

Anything you would recommend on how to run the docker and how to stop it from within a python script?

Hi @lucas0

If I understand, you download the latest docker English version (docker pull dbpedia/spotlight-english). The new version is tagged as “databus”, if this version works fine it will be the next latest version:

 docker pull dbpedia/spotlight-english:databus

After downloading this new version, run the following command:

  docker run -i -p 2222:80 dbpedia/spotlight-english:databus spotlight.sh

but I am not sure how it works on mac machines.

If it is possible, maybe you can try with the docker image dbpedia/spotlight-databus:v.0.2, following the steps posted. Thanks in advanced.

and about the other question:

Anything you would recommend on how to run the docker and how to stop it from within a python script?

An option could be to run a bash script from python. This bash script will contain the instructions to stop docker:

dockerStop.sh
    #!/bash/bin
    docker kill [name_of_docker_image]
    docker rm [name_of_docker_image] 

and other file to start the docker image. I hope this helps you.

My best regards

Julio

Just tested with the databus tag and it works the same as the v.0.2! What is the advantage of the databus? instead of not specifying the tag?

Hi @lucas0

Thanks for your help. The main advantage of dbpdia/spotlight-databus is to download from the Databus repository the lastest version of DBpedia Spotlight model. The dbpedia/spotlight-english:databus was built with the latest available DBpedia Spotlight model version. The spotlight-databus image does not need to be updated to work with the latest DBpedia Spotlight model version, meanwhile, for the spotlight-english it could be needed to update the docker image after three or four months (only if the model change).

Thanks for everything, guys! Let me know if I could help with anything else!

Best,
Lucas.

Thanks to you @lucas0, if you find any other issue or bug in the dbpedia/spotlight-[databus, english], please let us know by posting it here. Have a great day

My best regards

Julio

@JulioNoe what about the things you have created before?
In this version you pull the docker image
docker pull dbpedia/spotlight-databus:v.0.2
then, create a volume to store the model languages
docker volume create spotlight-databus
then, run the image, specyfing the volume and the target directory and one of three possible options:
-a Shows the downloaded models
-l Defines the model language to be download and run or just run if the model was downloaded
-r Remove a model language (if exists)
For example, to download and run the english version:
docker run --mount source=spotlight-databus,target=/opt/spotlight -i -p 2222:80 dbpedia/spotlight-databus:v.0.2 spotlight.sh -l en
to see which models are available
docker run --mount source=spotlight-databus,target=/opt/spotlight -i -p 2222:80 dbpedia/spotlight-databus:v.0.2 spotlight.sh -a
to remove a model
docker run --mount source=spotlight-databus,target=/opt/spotlight -i -p 2222:80 dbpedia/spotlight-databus:v.0.2 spotlight.sh -r en
This version gives control to add and delete models
Additionally I update the model-quickstarter providing the following error message about missing artifacts:
“Artifact(s) (XXXX) missing, for more details please refer to Tasks for Volunteers, task: ‘Languages with missing redirects/disambiguations/instance-type’”

Seems like you had it generic at some time, but now it is with -english again. This does not scale. It could be that we have spotlight in 40-50 versions now. So it floods the dockerhub and makes it unusable. I was hoping to test the generic version and then delete the -language ones.

Can you add these two feedback items to the docu, i.e. the answer?

No (but I understand why do you think this), In fact, @lucas0 and @mwon tested the docker dbpedia/spotligh-english:databus and the dbpedia/spotlight-portuguese:databus versions and after that they help me testing the new dbpedia/spotlight-databus:v.0.2 version. The result was that the -language and -databus versions run without any problem, that means, the spotlight-databus version provides the same functionality as the spotlight-english and spotlight-portuguese versions.

My comment about the docker spotlight-english version generated this misunderstanding:

I should have mentioned the dbpedia/spotlight-databus version instead of -english.

I will add the feedbacks items to the docu. Have a great day.

My. best regards

Julio

ok, cool. Seems like I misunderstood it, sorry.