Bug 197814 - portsnap fetched snapshot integrity check failed
Summary: portsnap fetched snapshot integrity check failed
Status: Closed FIXED
Alias: None
Product: Services
Classification: Unclassified
Component: Portsnap (show other bugs)
Version: unspecified
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-02-19 08:49 UTC by Vladimir Gusev
Modified: 2015-03-09 10:07 UTC (History)
4 users (show)

See Also:

Attachments
effective portsnap configuration file (1.48 KB, text/plain)
2015-02-19 08:49 UTC, Vladimir Gusev 		no flags Details
portsnapp'ed bogus INDEX.new (976.81 KB, application/x-xz)
2015-02-25 09:11 UTC, Vladimir Gusev 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Vladimir Gusev 2015-02-19 08:49:57 UTC 
Created attachment 153169 [details]
effective portsnap configuration file

Trying to get the ports snapshot, a permanent error on "one bogus line" started to occur:

# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Thu Feb 19 03:09:43 MSK 2015:
3fbf7ca5c1aee4d361fb6e56e81b0de60c665bf8c74023100% of   71 MB 1744 kBps 00m42s
Extracting snapshot... done.

Verifying snapshot integrity... 
Portsnap metadata is correctly signed, but contains
at least one line which appears bogus.
Cowardly refusing to proceed any further.

# uname -a
FreeBSD portal 10.1-RELEASE-p5 FreeBSD 10.1-RELEASE-p5 #0: Tue Jan 27 08:55:07 UTC 2015     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Troubleshooting performed:

# rm -rf /usr/ports
# rm -rf /var/db/portsnap

didn't fix the issue.

Please help to find out the cure.
Thanks!
Comment 1 Sean Bruno freebsd_committer freebsd_triage 2015-02-19 18:12:54 UTC 
Adding Colin to take a look.
Comment 2 Colin Percival freebsd_committer freebsd_triage 2015-02-19 23:17:59 UTC 
Everything looks fine to me.  If this is still happening, can you run

# sh -x `which portsnap` fetch

and past the last few lines?
Comment 3 Vladimir Gusev 2015-02-20 08:24:25 UTC 
Hi Colin,

this is still happening, please see below.

# sh -x `which portsnap` fetch
[...skipped...]
Verifying snapshot integrity... + cut -f 2 -d '|' tINDEX.new
+ fetch_snapshot_verify
+ read F
+ gunzip -c snap/a16d96955d8951eb51bb636375a633b799a4bb9717eaef2bdfaede4bf405742c
+ /sbin/sha256 -q
+ [ a16d96955d8951eb51bb636375a633b799a4bb9717eaef2bdfaede4bf405742c != a16d96955d8951eb51bb636375a633b799a4bb9717eaef2bdfaede4bf405742c ]
+ read F
+ gunzip -c snap/2bafbd0d8edc7a7cfa7e19833986ae4032f82006fd0d65cba9c4a75b432b5c8e
+ /sbin/sha256 -q
+ [ 2bafbd0d8edc7a7cfa7e19833986ae4032f82006fd0d65cba9c4a75b432b5c8e != 2bafbd0d8edc7a7cfa7e19833986ae4032f82006fd0d65cba9c4a75b432b5c8e ]
+ read F
+ gunzip -c snap/4954f84544ccd39dc4020f92ffbcb3b2508c59aa3c07d35a8315ce71a734f841
+ /sbin/sha256 -q
+ [ 4954f84544ccd39dc4020f92ffbcb3b2508c59aa3c07d35a8315ce71a734f841 != 4954f84544ccd39dc4020f92ffbcb3b2508c59aa3c07d35a8315ce71a734f841 ]
+ read F
+ gunzip -c snap/5d78a0d5b86049ed7f8cea2e6f4ea7a00216a3574db5072f6168bcdaef2bb5b4
+ /sbin/sha256 -q
+ [ 5d78a0d5b86049ed7f8cea2e6f4ea7a00216a3574db5072f6168bcdaef2bb5b4 != 5d78a0d5b86049ed7f8cea2e6f4ea7a00216a3574db5072f6168bcdaef2bb5b4 ]
+ read F
+ gunzip -c snap/09f65f8a730283fd31d068a5927ed46d95e37540f89090c257d7809b75116293
+ /sbin/sha256 -q
+ [ 09f65f8a730283fd31d068a5927ed46d95e37540f89090c257d7809b75116293 != 09f65f8a730283fd31d068a5927ed46d95e37540f89090c257d7809b75116293 ]
+ read F
+ gunzip -c snap/526948705269fff43993c0e7396f2a8d3cba3f17cd7b20e67e25199f67cbe1c9
+ /sbin/sha256 -q
+ [ 526948705269fff43993c0e7396f2a8d3cba3f17cd7b20e67e25199f67cbe1c9 != 526948705269fff43993c0e7396f2a8d3cba3f17cd7b20e67e25199f67cbe1c9 ]
+ read F
+ gunzip -c snap/62e76a2dd406d7e245abd47170ccaeda648d851921b8bb41f3e579f9284cde2d
+ /sbin/sha256 -q
+ [ 62e76a2dd406d7e245abd47170ccaeda648d851921b8bb41f3e579f9284cde2d != 62e76a2dd406d7e245abd47170ccaeda648d851921b8bb41f3e579f9284cde2d ]
+ read F
+ gunzip -c snap/72014c143dcce6c9c00cbc94541218069e41eb706b8ebe1fcc44516b1d126c53
+ /sbin/sha256 -q
+ [ 72014c143dcce6c9c00cbc94541218069e41eb706b8ebe1fcc44516b1d126c53 != 72014c143dcce6c9c00cbc94541218069e41eb706b8ebe1fcc44516b1d126c53 ]
+ read F
+ return 0
+ rm -f INDEX.new
+ look INDEX tINDEX.new
+ cut -f 2 -d '|'
+ gunzip -c snap/72014c143dcce6c9c00cbc94541218069e41eb706b8ebe1fcc44516b1d126c53.gz
+ fetch_index_sanity
+ grep -qvE '^[-_+./@0-9A-Za-z]+\|[0-9a-f]{64}$' INDEX.new
+ fgrep -q ./ INDEX.new
+ fetch_metadata_freakout
+ echo

+ echo 'Portsnap metadata is correctly signed, but contains'
Portsnap metadata is correctly signed, but contains
+ echo 'at least one line which appears bogus.'
at least one line which appears bogus.
+ echo 'Cowardly refusing to proceed any further.'
Cowardly refusing to proceed any further.
+ return 1
+ return 1
+ return 1
+ exit 1

if anything else needed, please let me know.

Thank you.
Comment 4 Colin Percival freebsd_committer freebsd_triage 2015-02-20 22:34:33 UTC 
Very strange.  Can you email the INDEX.new file to cperciva@FreeBSD.org?
Comment 5 Vladimir Gusev 2015-02-25 09:11:47 UTC 
Created attachment 153512 [details]
portsnapp'ed bogus INDEX.new
Comment 6 Colin Percival freebsd_committer freebsd_triage 2015-02-26 21:16:39 UTC 
That file looks fine to me.  Can you run

# grep -vE '^[-_+./@0-9A-Za-z]+\|[0-9a-f]{64}$' INDEX.new

# fgrep -q ./ INDEX.new

and tell me if there is any output?  I'm wondering if there's a problem in the grep you're using...
Comment 7 Vladimir Gusev 2015-02-28 09:06:45 UTC 
(In reply to Colin Percival from comment #6)

all the commands entered haven't given any output

# grep -vE '^[-_+./@0-9A-Za-z]+\|[0-9a-f]{64}$' INDEX.new
# fgrep -q ./ INDEX.new
# /usr/bin/grep -vE '^[-_+./@0-9A-Za-z]+\|[0-9a-f]{64}$' INDEX.new
# /usr/bin/fgrep -q ./ INDEX.new

BTW, just for sure:

# sha256 /usr/bin/grep /usr/bin/fgrep
SHA256 (/usr/bin/grep) = 2a943de091b82187b0840436a2bf65e9f9bfc33a8b40ba9a7ed06e4156d1c827
SHA256 (/usr/bin/fgrep) = 2a943de091b82187b0840436a2bf65e9f9bfc33a8b40ba9a7ed06e4156d1c827

# uname -a
FreeBSD portal 10.1-RELEASE-p5 FreeBSD 10.1-RELEASE-p5 #0: Tue Jan 27 08:55:07 UTC 2015     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Thanks!
Comment 8 Vladimir Gusev 2015-03-09 10:07:38 UTC 
Ok, the cause of the failure was in $GREP_OPTIONS non-empty exported variable.

Well, the workaround is just unset/specify empty $GREP_OPTIONS:

# export GREP_OPTIONS=''
# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Mon Mar  9 03:13:10 MSK 2015:
1e55f8a65a5e3b11b856497828d9a24036a29780729437100% of   71 MB 2147 kBps 00m35s
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Mon Mar  9 03:13:10 MSK 2015 to Mon Mar  9 12:27:53 MSK 2015.
Fetching 4 metadata patches... done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
Fetching 33 patches. 
(33/33) 100.00%  done.                               
done.
Applying patches... 
done.
Fetching 3 new ports or files... done.


####### Troubleshooting performed as below

# export GREP_OPTIONS='--color=always'
# portsnap fetch
portsnap: Invalid key fingerprint: 9b5feee6d69f170e3dd0a2c8e469ddbd64f13f978f2f3aede40c98633216c330

# export GREP_OPTIONS='--color=always '
# portsnap fetch
portsnap: Invalid key fingerprint: 9b5feee6d69f170e3dd0a2c8e469ddbd64f13f978f2f3aede40c98633216c330

# export GREP_OPTIONS=' --color=always '
# portsnap fetch
portsnap: Invalid key fingerprint: 9b5feee6d69f170e3dd0a2c8e469ddbd64f13f978f2f3aede40c98633216c330

# export GREP_OPTIONS=' --color=always'
# portsnap fetch
portsnap: Invalid key fingerprint: 9b5feee6d69f170e3dd0a2c8e469ddbd64f13f978f2f3aede40c98633216c330

# export GREP_OPTIONS='-q'
# portsnap fetch
portsnap: SERVERNAME must be given via command line or configuration file.

# export GREP_OPTIONS='-E'
# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
Fetching public key from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Mon Mar  9 03:13:10 MSK 2015:
1e55f8a65a5e3b11b856497828d9a24036a29780729437100% of   71 MB 2204 kBps 00m34s
Extracting snapshot... done.
Verifying snapshot integrity... 
Portsnap metadata is correctly signed, but contains
at least one line which appears bogus.
Cowardly refusing to proceed any further.