update crawler to ensure urls aren't added twice to a snapshot

update crawler to ensure urls aren't added twice to a snapshot #100