I think I've found another problem, this time in the BOINC client for Linux, that may be exacerbating this.
I tried switching to another application (the Gamma-ray pulsar binary search). Couldn't fetch any work immediately, because I'm quota-limited, but found this in my Event Log once the witching hour had passed:
Fri 18 Feb 2022 00:34:33 GMT | Einstein@Home | Started download of hsgamma_FGRPB1G_1.28_x86_64-pc-linux-gnu__FGRPopencl2Pup-nvidia
Fri 18 Feb 2022 00:34:35 GMT | Einstein@Home | File hsgamma_FGRPB1G_1.28_x86_64-pc-linux-gnu__FGRPopencl2Pup-nvidia exists already, skipping download
Fri 18 Feb 2022 00:34:35 GMT | Einstein@Home | [error] Signature verification failed for hsgamma_FGRPB1G_1.28_x86_64-pc-linux-gnu__FGRPopencl2Pup-nvidia
Fri 18 Feb 2022 00:34:35 GMT | Einstein@Home | [error] Checksum or signature error for hsgamma_FGRPB1G_1.28_x86_64-pc-linux-gnu__FGRPopencl2Pup-nvidia
Investigating, I found:
The file existed on my hard drive, but was zero length
Fetching the file manually from Hannover generated the 404 error again
But I could get it from Syracuse
That "exists already, skipping download" seems to be a major flaw. Just because you got zero bytes last time, doesn't mean you don't still need the real thing!
I've copied my manual download into the proper place (over-writing the zero byte version), and I'll try it during waking hours tomorrow morning.
This (potential) workround won't help for files that relate to a single task, but it may help for applications or reference files which are re-used many times over.
I made the experimentation using a VPN in America (congratulations to MIKEY and RICHARD HASELGROVE for the idea) and it worked ! I was able to add the project without download errors once the BOINC client detects the American server :
This also means that any user that will use the European server (without access to the American server) cannot join einstein@home for now and will encounter download errors for some work units, so this bug concerns a lot of users :(.
To complete the experiment, I will keep work units running and see what happens when they are sent to the European server.
Notice how all the download subdirectories are subtly different - that makes it extremely hard to guess the right location. There's a fiendishly complicated algorithm for working it out (involving selecting three characters from the MD5 of the name), but it's not worth it.
I'll give up now, and write up what we've found so far.
Notice how all the download subdirectories are subtly different - that makes it extremely hard to guess the right location. There's a fiendishly complicated algorithm for working it out (involving selecting three characters from the MD5 of the name), but it's not worth it.
I'll give up now, and write up what we've found so far.
I think the affected files cannot be determined by their name or URL.
But einstein@home staff could be able to find problematic files using their creation dates.
I think the affected files were created from a certain date but this created date is not visible in the URL.
We are investigating, there seems to be something wrong with the webserver configuration of einstein2.
For now I removed einstein2 from the list of download servers, but this will only affect tasks that are delivered from now on.
We keep investigating and will fix this problem ASAP. But maybe over the weekend we eill have to do without the hannover mirrror - there's a serious hurricane warning for this evening, and we all are busy to protect our homes.
We are investigating, there seems to be something wrong with the webserver configuration of einstein2.
For now I removed einstein2 from the list of download servers, but this will only affect tasks that are delivered from now on.
We keep investigating and will fix this problem ASAP. But maybe over the weekend we eill have to do without the hannover mirrror - there's a serious hurricane warning for this evening, and we all are busy to protect our homes.
Thank a lot for investigating on this issue.
I have a question, the server http://einstein-dl.syr.edu is unreachable from France, is this an expected behavior ?
I think I've found another
)
I think I've found another problem, this time in the BOINC client for Linux, that may be exacerbating this.
I tried switching to another application (the Gamma-ray pulsar binary search). Couldn't fetch any work immediately, because I'm quota-limited, but found this in my Event Log once the witching hour had passed:
Investigating, I found:
That "exists already, skipping download" seems to be a major flaw. Just because you got zero bytes last time, doesn't mean you don't still need the real thing!
I've copied my manual download into the proper place (over-writing the zero byte version), and I'll try it during waking hours tomorrow morning.
This (potential) workround won't help for files that relate to a single task, but it may help for applications or reference files which are re-used many times over.
Hello again,I made the
)
Hello again,
I made the experimentation using a VPN in America (congratulations to MIKEY and RICHARD HASELGROVE for the idea) and it worked ! I was able to add the project without download errors once the BOINC client detects the American server :
http://einstein-dl.syr.edu
So the experimentation confirmed that European servers of einstein@home is no longer synced with the American server !
Exemple with a work unit file which failed to download before :
GET http://einstein-dl.syr.edu/download/206/h1_0636.80_O3aC01Cl1In0 -> HTTP 200 [OK] (only with a vpn, the server is unreachable in Europe)
GET http://einstein2.aei.uni-hannover.de/download/206/h1_0636.80_O3aC01Cl1In0 -> HTTP 404 [NOT OK]
Another exemple with a binary file which failed to download before :
GET http://einstein-dl.syr.edu/download/einstein_O3AS_1.01_windows_x86_64__GW-opencl-nvidia.exe -> HTTP 200 [OK] (only with a vpn, the server is unreachable in Europe)
GET http://einstein2.aei.uni-hannover.de/download/einstein_O3AS_1.01_windows_x86_64__GW-opencl-nvidia.exe -> HTTP 404 [NOT OK]
The issue is now identified.
This also means that any user that will use the European server (without access to the American server) cannot join einstein@home for now and will encounter download errors for some work units, so this bug concerns a lot of users :(.
To complete the experiment, I will keep work units running and see what happens when they are sent to the European server.
Can someone working for einstein@home look at the European server (einstein2.aei.uni-hannover.de) ?
EDIT : after testing, tasks downloaded from the american server succeed to upload results using the european server.
I found a number of other
)
I found a number of other files that seemed to be present at Syracuse, but missing from Hannover:
http://einstein-dl.syr.edu/download/293/skygrid_640Hz_m0.002.dat
http://einstein-dl.syr.edu/download/f8/h1_0639.80_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/234/l1_0639.80_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/269/h1_0640.00_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/a9/l1_0640.00_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/a4/h1_0640.20_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/38d/l1_0640.20_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/34b/h1_0640.40_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/3d8/l1_0640.40_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/21e/h1_0640.60_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/333/l1_0640.60_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/261/h1_0640.80_O3aC01Cl1In0
http://einstein-dl.syr.edu/download/3ae/l1_0640.80_O3aC01Cl1In0
Notice how all the download subdirectories are subtly different - that makes it extremely hard to guess the right location. There's a fiendishly complicated algorithm for working it out (involving selecting three characters from the MD5 of the name), but it's not worth it.
I'll give up now, and write up what we've found so far.
Richard Haselgrove wrote: I
)
I think the affected files cannot be determined by their name or URL.
But einstein@home staff could be able to find problematic files using their creation dates.
I think the affected files were created from a certain date but this created date is not visible in the URL.
Messages
)
Messages sent
https://github.com/BOINC/boinc/issues/4642
Thanks, Richard, for pointing
)
Thanks, Richard, for pointing me to this.
We are investigating, there seems to be something wrong with the webserver configuration of einstein2.
For now I removed einstein2 from the list of download servers, but this will only affect tasks that are delivered from now on.
We keep investigating and will fix this problem ASAP. But maybe over the weekend we eill have to do without the hannover mirrror - there's a serious hurricane warning for this evening, and we all are busy to protect our homes.
BM
Richard Haselgrove
)
Thank you
Bernd Machenschalk
)
Thank a lot for investigating on this issue.
I have a question, the server http://einstein-dl.syr.edu is unreachable from France, is this an expected behavior ?
Temporary fix in place -
)
Temporary fix in place - should hold over the weekend. Will fix it properly on Monday.
BM
Bernd Machenschalk
)
Thank you :-)