Hello,
I’m trying to use your API to extract and link entities from a corpus made of Portuguese news. I’m using pyspotlight with the URL:
http://api.dbpedia-spotlight.org/pt/annotate
I would like to know if there is a usage limit because just after a few correct annotations it gives me a “403 Client Error: Forbidden” response.
Thanks
Dear @mwon,
the service should be free and unlimited per se, except for server resources, i.e. if there are many concurrent requests.
We recently created a new docker for self hosting. @JulioNoe could you post a link to instructions? Also could you check the service, whether there is a limit, either by looking at the code or running 1000 abstracts in pt through it, either the docker or http://api.dbpedia-spotlight.org/pt/annotate
I will also ping Alexioman, whether this is a pyspotlight problem.
– Sebastian
Hi @mwon,
It would surprise me if someone introduced a rate limiter without telling us. The 403 definitely doesn’t come from pyspotlight, which simply passes on responses that it gets from the server.
Please try again, you should be okay now. The status page was also receiving 403 a couple of days ago: https://status.dbpedia-spotlight.org/778676712
I don’t know what caused this unfortunately. It may have something to do with how the containers are hosted.
I’ve never had this issue with self-hosted Spotlight, so if this keeps happening, please let us know, but also give @kurzum’s suggestion a try.
About the docker versions. Here it is a link to the dbpedia/spotlight-databus docker image (which let you run the available languages model in Databus). But if you only want to run the Portuguese version, here it is the dbpedia/spotlight-portuguese docker image. The instructions to run the databus or Portuguese version is explained in each link.
About the “403 Client Error”: I ran the dbpedia/spotlight-databus docker image version with the Portuguese language model and I had no problems running 1000 petitions. However, I did the same with http://model.dbpedia-spotlight.org/pt/annotate and after 21 petitions sended to the server the following message is shown:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr>
<address>Apache/2.4.25 (Debian) Server at model.dbpedia-spotlight.org Port 80</address>
</body></html>
I will check the code to determine where it is defined this restriction or maybe it could be a server configuration.
@mwon if you want to use the server URL (http://model.dbpedia-spotlight.org/pt/annotate) you can set a wait/sleep function for 1-2 minutes between each 20 petitions (approximately) or you can download the Databus or Portuguese docker versions which let you run any number of petitions.
Ok, thanks. I’m now working with the Portuguese docker image and it is working fine.
Hello all,
I am making many requests (~10k) to the english api (http://api.dbpedia-spotlight.org/en/annotate) and receiving a lot of 403s… I’ve realized that the percentage of errors I get is inversely proportional to the interval between requests. I am also getting some 502 errors from time to time.
I hope you find this comment useful and that this issue can be solved soon.
Best,
Lucas.
@lucas0,
@JulioNoe has reworked the spotlight docker, which works for @mwon. We will deploy it on the server as a replacement and maybe you can test it for us. We will ping you then.
I’d be happy to do so.
@kurzum
@mwon
Seems like I’ve managed to run the docker image:
$sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
47b3469ea4fa dbpedia/spotlight-english "/bin/sh" 6 minutes ago Up 6 minutes 0.0.0.0:2222->80/tcp focused_wu
but when I do
curl http://0.0.0.0:2222/rest/annotate?text=COOPER+Has+the+FBI+said+anything+about+a+reward+or+anything+%5C%3F+Because+there+was+that+there+was+a+reward+for+finding+her&confidence=0.5
I get:
curl: (56) Recv failure: Connection reset by peer
what is the problem? Is there a wiki or instructions on how to send requests to the docker?
Best,
Lucas.
Update: getting 400s now when requesting to the api (not the docker)
Hi @lucas0,
An example of curl to query the dbpedia docker version is as follows:
curl http://localhost:2222/rest/annotate --data-urlencode "text=President Obama called Wednesday on Congress to extend a tax break for students included in last year's economic stimulus package, arguing that the policy provides more generous assistance." --data "confidence=0.35" -H "Accept: text/turtle"
where “Accept:text/turtle” returns a nif output but if you want a json output you must replace it for “Accept: application/json”
It could be nice if help us to try the dbpedia/spotlight-databus version. This version allows you to locally download more than one language, e.g., English (en), German (de), etc.
The docker version is another way to run DBpedia Spotlight and it follows the same instructions as the DBpedia Spotligh Github project. Thanks for the observation, I will add a “how to run” to the docker instructions.
My best regards
Julio
I want to ask you for your help and support to test a new docker version of the DBpedia Spotlight. I know that you are interested into using a specific version of DBpedia Spotlight (Portuguese and English) then, it could be very useful if you can use the dbpedia/spotlight-databus docker version, in particular the v.0.2 version. Additional to your interested languages, you could try any of the available languages in this link.
docker pull dbpedia/spotlight-databus:v.0.2
If you found any issue/bug or have any question, please post it here. Thanks in advance.
My best regards
Julio
Ok, I tried to follow the instructions but not really sure if I’m doing it correctly. I run:
docker volume create spotlight-databus
followed by:
docker run --name spotlight-databus --mount source=spotlight-databus,target=/opt/spotlight -i -p 2222:80 dbpedia/spotlight-databus:v.0.2 spotlight.sh ln -l pt
But got this message:
Possible options:
-a Available models
-l Define the model language
-r Remove a model language (if exists)
and no docker is running
@mwon, sorry it was an error the “ln” don’t have to be there. I already update the instructions in the docker page, thanks.
docker run --name spotlight-databus --mount source=spotlight-databus,target=/opt/spotlight -i -p 2222:80 dbpedia/spotlight-databus:v.0.2 spotlight.sh -l pt
if the following error is shown:
docker: Error response from daemon: Conflict. The container name "/spotlight-databus" is already in use by container "24884ad77682ee1af27cad41286eb8b886602b9ef902164e8a84cfe3c6dc69de". You have to remove (or rename) that container to be able to reuse that name.
just run
docker rm spotlight-databus
Thanks.
My best regards
Julio
@JulioNoe
I’d be happy to help. I’ve finally found what was wrong on my docker and the reason why i was getting error 56…
I was able to put the docker on with sudo docker run -i-p 2222:80 dbpedia/spotlight-english
but since there was no entry-point created by the docker itself, it is imperative that i run the spotlight.sh bash file so the entry-point is created. Still, I was getting the same error…
Another thing was that, for some reason the docker wouldnt stay on for more than a wee bit on my local mac machine (it wasnt the same on an ubuntu 18.04 i was working through shh, where it stayed listed with docker ps for a longer while), so adding the --restart unless-stopped flag made it work. Still not completely sure why.
I’ll try to set up the databus with the instructions you’ve placed here.
Best,
Lucas.
Anything you would recommend on how to run the docker and how to stop it from within a python script?
Hi @lucas0
If I understand, you download the latest docker English version (docker pull dbpedia/spotlight-english). The new version is tagged as “databus”, if this version works fine it will be the next latest version:
docker pull dbpedia/spotlight-english:databus
After downloading this new version, run the following command:
docker run -i -p 2222:80 dbpedia/spotlight-english:databus spotlight.sh
but I am not sure how it works on mac machines.
If it is possible, maybe you can try with the docker image dbpedia/spotlight-databus:v.0.2, following the steps posted. Thanks in advanced.
and about the other question:
Anything you would recommend on how to run the docker and how to stop it from within a python script?
An option could be to run a bash script from python. This bash script will contain the instructions to stop docker:
dockerStop.sh
#!/bash/bin
docker kill [name_of_docker_image]
docker rm [name_of_docker_image]
and other file to start the docker image. I hope this helps you.
My best regards
Julio
Just tested with the databus tag and it works the same as the v.0.2! What is the advantage of the databus? instead of not specifying the tag?
Hi @lucas0
Thanks for your help. The main advantage of dbpdia/spotlight-databus is to download from the Databus repository the lastest version of DBpedia Spotlight model. The dbpedia/spotlight-english:databus was built with the latest available DBpedia Spotlight model version. The spotlight-databus image does not need to be updated to work with the latest DBpedia Spotlight model version, meanwhile, for the spotlight-english it could be needed to update the docker image after three or four months (only if the model change).
Thanks for everything, guys! Let me know if I could help with anything else!
Best,
Lucas.