yacy:rc1.git
3 years agorelease 1.7 Release_1.7
Michael Peter Christen [Fri, 4 Apr 2014 15:05:03 +0000 (17:05 +0200)]
release 1.7

3 years agoupdate of seed list
Michael Peter Christen [Fri, 4 Apr 2014 15:04:49 +0000 (17:04 +0200)]
update of seed list

3 years agoMerge branch 'master' of gitorious.org:yacy/icewindxs-rc1
Michael Peter Christen [Fri, 4 Apr 2014 14:42:50 +0000 (16:42 +0200)]
Merge branch 'master' of gitorious.org:yacy/icewindxs-rc1

Conflicts:
locales/ru.lng

3 years agoUpdate russian translation 42
malykhin.dmitry [Fri, 4 Apr 2014 14:28:57 +0000 (18:28 +0400)]
Update russian translation

3 years agoupdate to german locale
Michael Peter Christen [Fri, 4 Apr 2014 13:45:08 +0000 (15:45 +0200)]
update to german locale

3 years agofixed a time-out problem in the default servlet which is also a logging
Michael Peter Christen [Fri, 4 Apr 2014 13:27:29 +0000 (15:27 +0200)]
fixed a time-out problem in the default servlet which is also a logging
problem because the error log showed the wrong reason (file not found)
instead the actual reason (time-out).

3 years agofixed a problem with resource observer; probably coming from uncatched
Michael Peter Christen [Fri, 4 Apr 2014 13:26:39 +0000 (15:26 +0200)]
fixed a problem with resource observer; probably coming from uncatched
exceptions within the apache library which appear only in concurrency
environments.

3 years agoeliminating the word 'Facet' from the interface because it is ugly. If
Michael Peter Christen [Fri, 4 Apr 2014 13:25:37 +0000 (15:25 +0200)]
eliminating the word 'Facet' from the interface because it is ugly. If
people do not know what search navigation is, then they also do not know
what a 'facet' is.

3 years agoenhanced the new link structure graph
Michael Peter Christen [Fri, 4 Apr 2014 12:43:54 +0000 (14:43 +0200)]
enhanced the new link structure graph

3 years agofixed a bug which prevented crawl starts after a network switch
Michael Peter Christen [Fri, 4 Apr 2014 12:43:35 +0000 (14:43 +0200)]
fixed a bug which prevented crawl starts after a network switch

3 years agobetter link structure limit calibration
Michael Peter Christen [Fri, 4 Apr 2014 10:48:55 +0000 (12:48 +0200)]
better link structure limit calibration

3 years agofix for linkstructure computation: now also detecting dead links
Michael Peter Christen [Fri, 4 Apr 2014 10:47:29 +0000 (12:47 +0200)]
fix for linkstructure computation: now also detecting dead links

3 years agoanother fix for inbound/outbound detection
Michael Peter Christen [Fri, 4 Apr 2014 10:41:59 +0000 (12:41 +0200)]
another fix for inbound/outbound detection

3 years agoNPE fix
Michael Peter Christen [Fri, 4 Apr 2014 10:34:34 +0000 (12:34 +0200)]
NPE fix

3 years agofix for parsing of numeric value in case that boolean values are given
Michael Peter Christen [Fri, 4 Apr 2014 09:59:51 +0000 (11:59 +0200)]
fix for parsing of numeric value in case that boolean values are given

3 years agobetter logging in template engine: shows filename of servlets where
orbiter [Fri, 4 Apr 2014 08:55:46 +0000 (10:55 +0200)]
better logging in template engine: shows filename of servlets where
errors in templates occur

3 years agofixed thread dump in case of wrong seeds
orbiter [Fri, 4 Apr 2014 08:54:56 +0000 (10:54 +0200)]
fixed thread dump in case of wrong seeds

3 years agomoved Edge class out of linkstructure servlet as this does not work on
orbiter [Fri, 4 Apr 2014 08:54:11 +0000 (10:54 +0200)]
moved Edge class out of linkstructure servlet as this does not work on
non-eclipse driven environments (all non-dev cases)

3 years agoreduced search activity corona strength in network image
orbiter [Fri, 4 Apr 2014 08:08:44 +0000 (10:08 +0200)]
reduced search activity corona strength in network image

3 years agomove baseurl after parsing orig source in urlproxyservlet
reger [Fri, 4 Apr 2014 01:11:16 +0000 (03:11 +0200)]
move baseurl after parsing orig source in urlproxyservlet
to calculate absolute href links for rewrite from unmodified source.

3 years agofix: remove obsolete ref to yacy.home
reger [Fri, 4 Apr 2014 00:45:04 +0000 (02:45 +0200)]
fix: remove obsolete ref to yacy.home
(use Switchboard instead)

3 years ago- added d3.js for visualizations using embedded svg
Michael Peter Christen [Thu, 3 Apr 2014 12:51:19 +0000 (14:51 +0200)]
- added d3.js for visualizations using embedded svg
- added a servlet api/linkstructure.json which generates a link graph
information in json
- added a javascript link graph renderer hypertree.js using d3 and the
new servlet linkstructure.json
- embedded the new link graph in the crawler monitor and the host
browser

3 years agofix for solr query size; too many documents had been retrieved in case
Michael Peter Christen [Thu, 3 Apr 2014 11:42:10 +0000 (13:42 +0200)]
fix for solr query size; too many documents had been retrieved in case
that less than _pagesize_ had been requested.

3 years agoMerge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Wed, 2 Apr 2014 23:58:06 +0000 (01:58 +0200)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git

3 years agobugfix to inbound/outbound identification
Michael Peter Christen [Wed, 2 Apr 2014 23:21:43 +0000 (01:21 +0200)]
bugfix to inbound/outbound identification

3 years agoeleminate obsolete URIMetaDataRow class
reger [Wed, 2 Apr 2014 22:35:15 +0000 (00:35 +0200)]
eleminate obsolete URIMetaDataRow class
by joining it with/into URIMetaDataNode.

3 years agointroduced new solr field crawldepth_i which records the crawl depth of
Michael Peter Christen [Wed, 2 Apr 2014 21:37:01 +0000 (23:37 +0200)]
introduced new solr field crawldepth_i which records the crawl depth of
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.

3 years agoadded missing servlet html
Michael Peter Christen [Wed, 2 Apr 2014 15:37:21 +0000 (17:37 +0200)]
added missing servlet html

3 years agofix for first start options and added german translation for popup texts
orbiter [Wed, 2 Apr 2014 15:10:59 +0000 (17:10 +0200)]
fix for first start options and added german translation for popup texts

3 years agofixed crawl start path for cloned crawls
orbiter [Wed, 2 Apr 2014 14:06:29 +0000 (16:06 +0200)]
fixed crawl start path for cloned crawls

3 years ago- added COLS field in GSA response (non-gsa standard by customer
orbiter [Wed, 2 Apr 2014 14:05:44 +0000 (16:05 +0200)]
- added COLS field in GSA response (non-gsa standard by customer
request)
- updated document link in GSA response writer

3 years ago- added popover to p2p/stealth mode button to explain the peer mode and
Michael Peter Christen [Wed, 2 Apr 2014 11:33:43 +0000 (13:33 +0200)]
- added popover to p2p/stealth mode button to explain the peer mode and
privacy issues.
- added popover to first-time use case to explain that specific servlets
are only visible after customization and/or crawl starts

3 years agotransition.js is also included in bootstrap.js
Michael Peter Christen [Wed, 2 Apr 2014 10:19:26 +0000 (12:19 +0200)]
transition.js is also included in bootstrap.js

3 years agoremoved alert.js, modal.js and tooltip.js as these libraries are all
Michael Peter Christen [Wed, 2 Apr 2014 10:18:33 +0000 (12:18 +0200)]
removed alert.js, modal.js and tooltip.js as these libraries are all
included in bootstrap.js

3 years agofix for naming bug
Michael Peter Christen [Wed, 2 Apr 2014 02:06:35 +0000 (04:06 +0200)]
fix for naming bug

3 years agoMerge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Wed, 2 Apr 2014 01:27:08 +0000 (03:27 +0200)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git

3 years agofix: application link count
reger [Wed, 2 Apr 2014 01:21:51 +0000 (03:21 +0200)]
fix: application link count
(URIMetadataNode)

3 years agoMerge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Wed, 2 Apr 2014 00:35:58 +0000 (02:35 +0200)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git

3 years agonpe fix
Michael Peter Christen [Wed, 2 Apr 2014 00:34:55 +0000 (02:34 +0200)]
npe fix

3 years agofree configuration of postprocessing clickdepth maximum depth and time
Michael Peter Christen [Wed, 2 Apr 2014 00:34:39 +0000 (02:34 +0200)]
free configuration of postprocessing clickdepth maximum depth and time

3 years agoadded tutorial mode - some menu items will only appear if you 'qualify'
Michael Peter Christen [Wed, 2 Apr 2014 00:33:17 +0000 (02:33 +0200)]
added tutorial mode - some menu items will only appear if you 'qualify'
for them. Thus, the first-time user will only see four menu items. The
other items will unfold as the user interacts.

3 years agofix receiving DHT / parse pultipart
sixcooler [Tue, 1 Apr 2014 23:24:15 +0000 (01:24 +0200)]
fix receiving DHT / parse pultipart
+ another close to fix possible resource leak warning

3 years agofix for crawler column width
Michael Peter Christen [Tue, 1 Apr 2014 23:16:34 +0000 (01:16 +0200)]
fix for crawler column width

3 years agoMerge branch 'master' of gitorious.org:yacy/icewindxs-rc1
Michael Peter Christen [Tue, 1 Apr 2014 19:37:30 +0000 (21:37 +0200)]
Merge branch 'master' of gitorious.org:yacy/icewindxs-rc1

Conflicts:
locales/ru.lng

3 years agoUpdate russian translation 41
malykhin.dmitry [Tue, 1 Apr 2014 18:01:22 +0000 (22:01 +0400)]
Update russian translation

3 years agoadded more bootstrap libraries
Michael Peter Christen [Tue, 1 Apr 2014 15:18:26 +0000 (17:18 +0200)]
added more bootstrap libraries

3 years agoshow host list by default in host browser
Michael Peter Christen [Tue, 1 Apr 2014 14:55:22 +0000 (16:55 +0200)]
show host list by default in host browser

3 years agofix for paths in locale files
Michael Peter Christen [Tue, 1 Apr 2014 13:55:20 +0000 (15:55 +0200)]
fix for paths in locale files

3 years agoMerge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Tue, 1 Apr 2014 11:20:03 +0000 (13:20 +0200)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git

3 years agomake use of detected charset in htmlParser if none is given.
reger [Tue, 1 Apr 2014 02:02:34 +0000 (04:02 +0200)]
make use of detected charset in htmlParser if none is given.

3 years agoadd test case htmlParser.parse - getCharset
reger [Tue, 1 Apr 2014 00:55:22 +0000 (02:55 +0200)]
add test case htmlParser.parse  - getCharset
(which fails)

3 years agofixed crawl start links after renaming them in latest commit
Michael Peter Christen [Mon, 31 Mar 2014 22:35:58 +0000 (00:35 +0200)]
fixed crawl start links after renaming them in latest commit

3 years agomade crawl start pages public since they do not reveal individual
orbiter [Mon, 31 Mar 2014 18:42:39 +0000 (20:42 +0200)]
made crawl start pages public since they do not reveal individual
information and they are also not used as servlet to actually start the
crawl (which is Crawler_p.html).

3 years agoadded (again) underline to a tags
Michael Peter Christen [Mon, 31 Mar 2014 16:25:11 +0000 (18:25 +0200)]
added (again) underline to a tags

3 years agoIntegrated HostBrowser back to administration interface; it can appear
Michael Peter Christen [Mon, 31 Mar 2014 16:19:24 +0000 (18:19 +0200)]
Integrated HostBrowser back to administration interface; it can appear
with and without navigation bar.

3 years agoadded template for latest commit
Michael Peter Christen [Mon, 31 Mar 2014 14:00:13 +0000 (16:00 +0200)]
added template for latest commit

3 years agoComplete redesign of administration top-level menu. This follows two
Michael Peter Christen [Mon, 31 Mar 2014 13:47:58 +0000 (15:47 +0200)]
Complete redesign of administration top-level menu. This follows two
principles:
- provide an easy tutorial-like "what should I do first" menu
- provide all elements which are subject to most first questions to YaCy
exibition people on top level: Resource limitation, Parser and Ranking
settings
I apologize to everyone who are used to the old style and need to find
the menu items (again) after this change. I hope that this will make the
interface more usable for new users who see a web indexer/crawler the
first time.

3 years agoMerge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Mon, 31 Mar 2014 01:21:02 +0000 (03:21 +0200)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git

3 years agoredesign of search page integration menu structure
Michael Peter Christen [Mon, 31 Mar 2014 01:18:38 +0000 (03:18 +0200)]
redesign of search page integration menu structure

3 years agofix: 3 more links pointing to old /xml path
reger [Mon, 31 Mar 2014 00:58:43 +0000 (02:58 +0200)]
fix: 3 more links pointing to old /xml path

3 years agofix: blacklist xml export path to xml template
reger [Mon, 31 Mar 2014 00:48:28 +0000 (02:48 +0200)]
fix: blacklist xml export path to xml template

3 years agoreplacing direct html table cellspacing with css set-up for cellspacing
Michael Peter Christen [Sun, 30 Mar 2014 23:15:35 +0000 (01:15 +0200)]
replacing direct html table cellspacing with css set-up for cellspacing

3 years agoadding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p...
reger [Sun, 30 Mar 2014 03:11:42 +0000 (05:11 +0200)]
adding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p.html)

3 years agoalternative UrlProxyServlet (for /proxy.html) using different url rewrite rules
reger [Sun, 30 Mar 2014 02:04:02 +0000 (04:04 +0200)]
alternative UrlProxyServlet (for /proxy.html) using different url rewrite rules
- use JSoup parser for selective rewrite of html body <a href=  links only,
instead of regex which rewrites also header href/src links
- this improves display of pages which use header <base> tag
- tags with src attribute are taken from original location (like css) improving display and are not routed trough the indexer
Disadvantage: scripting links will drop out of proxy

Setting of the servlet through web.xml exclusivly (in case one would like to quickly switch back to the YaCyProxyServlet,
leaving the existing code of YaCyProxyServlet untouched available)

3 years agofix: cut-off button in WatchWebStructure_p.html
reger [Sat, 29 Mar 2014 23:23:54 +0000 (00:23 +0100)]
fix: cut-off button in WatchWebStructure_p.html
(by header css dd  hight/line-hight)

3 years agofix: port in local target adress
reger [Fri, 28 Mar 2014 23:34:01 +0000 (00:34 +0100)]
fix: port in local target adress
& button style

3 years agofixed visibility of custom icon
Michael Peter Christen [Fri, 28 Mar 2014 16:25:39 +0000 (17:25 +0100)]
fixed visibility of custom icon

3 years agoMerge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Fri, 28 Mar 2014 13:48:32 +0000 (14:48 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git

3 years agofix for relative sitemap urls
Michael Peter Christen [Fri, 28 Mar 2014 13:44:52 +0000 (14:44 +0100)]
fix for relative sitemap urls

3 years agoadded close to fix possible resource leak warning
Michael Peter Christen [Fri, 28 Mar 2014 13:34:49 +0000 (14:34 +0100)]
added close to fix possible resource leak warning

3 years agoprevent opening of new files as that could be a cause for the latest
Michael Peter Christen [Fri, 28 Mar 2014 13:31:43 +0000 (14:31 +0100)]
prevent opening of new files as that could be a cause for the latest
too-many-open-files exception. The old file is just truncated if the
table is cleaned.

3 years agoadded missing @Override annotation
Michael Peter Christen [Fri, 28 Mar 2014 12:48:37 +0000 (13:48 +0100)]
added missing @Override annotation

3 years agofix: button style IndexReIndexMonitor_p.html
reger [Fri, 28 Mar 2014 02:08:43 +0000 (03:08 +0100)]
fix: button style IndexReIndexMonitor_p.html

3 years agofix: proxyservlet path header setting
reger [Fri, 28 Mar 2014 01:05:58 +0000 (02:05 +0100)]
fix: proxyservlet path header setting

3 years agoadded linkscount_i and linksnofollowcount_i to the default solr schema
Michael Peter Christen [Thu, 27 Mar 2014 22:36:08 +0000 (23:36 +0100)]
added linkscount_i and linksnofollowcount_i to the default solr schema

3 years agoone more Thread Dump fix for new bootstrap css style
Michael Peter Christen [Thu, 27 Mar 2014 22:01:28 +0000 (23:01 +0100)]
one more Thread Dump fix for new bootstrap css style

3 years agofix for thread dump line spacing
Michael Peter Christen [Thu, 27 Mar 2014 21:13:37 +0000 (22:13 +0100)]
fix for thread dump line spacing

3 years agoremoved performance settings
Michael Peter Christen [Thu, 27 Mar 2014 21:00:57 +0000 (22:00 +0100)]
removed performance settings
- they are incomplete and buggy
- it was not easy to explain
- it did not comply with a KISS strategy
- setting a performance of low priority actually caused crashing of a
peer
- there was nobody who would maintain that functionality

3 years agochanged the protocol navigator: turned into switch buttons
Michael Peter Christen [Thu, 27 Mar 2014 12:50:13 +0000 (13:50 +0100)]
changed the protocol navigator: turned into switch buttons

3 years agofixed auto-activation of search field
Michael Peter Christen [Thu, 27 Mar 2014 12:36:56 +0000 (13:36 +0100)]
fixed auto-activation of search field

3 years agochanged protocol facet to show also all other counts if one facet is
Michael Peter Christen [Thu, 27 Mar 2014 12:29:14 +0000 (13:29 +0100)]
changed protocol facet to show also all other counts if one facet is
selected

3 years agomoved host facet up
Michael Peter Christen [Thu, 27 Mar 2014 11:03:51 +0000 (12:03 +0100)]
moved host facet up

3 years agoMerge branch 'master' of gitorious.org:yacy/icewindxs-rc1
Michael Peter Christen [Thu, 27 Mar 2014 11:00:10 +0000 (12:00 +0100)]
Merge branch 'master' of gitorious.org:yacy/icewindxs-rc1

3 years agoadded typeahead library to re-implement search suggestions for bootstrap
Michael Peter Christen [Thu, 27 Mar 2014 10:56:33 +0000 (11:56 +0100)]
added typeahead library to re-implement search suggestions for bootstrap
framework.

3 years agoremove unused private header entries (HeaderFramework)
reger [Wed, 26 Mar 2014 22:28:19 +0000 (23:28 +0100)]
remove unused private header entries (HeaderFramework)
X_YACY_ORIGINAL_REQUEST_LINE
X_YACY_KEEP_ALIVE_REQUEST_COUNT
CONNECTION_PROP_REQUESTLINE

3 years agomake transparent proxy handle https-connections:
sixcooler [Wed, 26 Mar 2014 19:01:15 +0000 (20:01 +0100)]
make transparent proxy handle https-connections:
the implemented handle for connect did not work for me - so lets try the
connectHandler

3 years agoUpdate russian translation
malykhin.dmitry [Wed, 26 Mar 2014 15:39:09 +0000 (19:39 +0400)]
Update russian translation

3 years agofix for source and target clickdepth in webgraph index
Michael Peter Christen [Wed, 26 Mar 2014 15:00:05 +0000 (16:00 +0100)]
fix for source and target clickdepth in webgraph index

3 years ago- added (previously removed) autocomplete
Michael Peter Christen [Wed, 26 Mar 2014 09:54:23 +0000 (10:54 +0100)]
- added (previously removed) autocomplete
- removed autocomplete from yacyinteractive.html
- todo: replace old jquery-1.3.2.min.js with latest which is already a
part of bootstrap.css

3 years agoTeam added support for URLs with unicode characters in host part to
Marc Nause [Tue, 25 Mar 2014 21:14:54 +0000 (22:14 +0100)]
Team added support for URLs with unicode characters in host part to
blacklist. Punycode is used to handle unicode characters.

3 years agoMerge branch 'master' of gitorious.org:yacy/icewindxs-rc1
Michael Peter Christen [Tue, 25 Mar 2014 09:35:57 +0000 (10:35 +0100)]
Merge branch 'master' of gitorious.org:yacy/icewindxs-rc1

Conflicts:
locales/ru.lng

3 years agoMerge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Mon, 24 Mar 2014 08:57:04 +0000 (09:57 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git

3 years agocorrected line-height in tagcloud
Michael Peter Christen [Mon, 24 Mar 2014 08:52:27 +0000 (09:52 +0100)]
corrected line-height in tagcloud

3 years agofixed search field
Michael Peter Christen [Mon, 24 Mar 2014 08:12:42 +0000 (09:12 +0100)]
fixed search field

3 years agoadd some InputFileStream close at end of reads
reger [Mon, 24 Mar 2014 01:32:17 +0000 (02:32 +0100)]
add some InputFileStream close at end of reads
to make sure file is released

3 years agolimit filetype nav to known extension also on image/media search
reger [Sun, 23 Mar 2014 22:10:29 +0000 (23:10 +0100)]
limit filetype nav to known extension also on image/media search
- on text search we limit filetype nav already to known extension, apply filter to image search

3 years agofix for embedded header: the new css framework always need a headline,
orbiter [Sun, 23 Mar 2014 16:28:34 +0000 (17:28 +0100)]
fix for embedded header: the new css framework always need a headline,
which is the location for the search input frame. The new embedded
header is a clone of the simpleHeader with removed menus and without a
link to the index administration.

3 years agoUpdate russian translation 39
malykhin.dmitry [Sun, 23 Mar 2014 15:32:47 +0000 (19:32 +0400)]
Update russian translation

3 years agoadded /date -ranking button
Michael Peter Christen [Sun, 23 Mar 2014 12:01:25 +0000 (13:01 +0100)]
added /date -ranking button

3 years ago- added text/image/audio/video/app search option to new navigation bar
Michael Peter Christen [Sun, 23 Mar 2014 11:29:46 +0000 (12:29 +0100)]
- added text/image/audio/video/app search option to new navigation bar
- changed colors of privacy selector