Michael Peter Christen [Sun, 9 Feb 2014 16:37:35 +0000 (17:37 +0100)]
Release 1.68
Michael Peter Christen [Sun, 9 Feb 2014 14:04:46 +0000 (15:04 +0100)]
catch solr close exceptions
Michael Peter Christen [Sun, 9 Feb 2014 13:57:17 +0000 (14:57 +0100)]
extra sleep for remote searches enhances search results because there is
more time for more remote peers to contribute on the first result page
Michael Peter Christen [Sun, 9 Feb 2014 13:45:12 +0000 (14:45 +0100)]
more efficient wordcount
Michael Peter Christen [Sun, 9 Feb 2014 11:36:56 +0000 (12:36 +0100)]
added memory protection for postprocessing
Michael Peter Christen [Sun, 9 Feb 2014 11:36:14 +0000 (12:36 +0100)]
enhanced memory protection and OOM exception handling in Solr connector
Michael Peter Christen [Sun, 9 Feb 2014 11:26:58 +0000 (12:26 +0100)]
removed warnings and superfluous logging
Michael Peter Christen [Sun, 9 Feb 2014 11:25:55 +0000 (12:25 +0100)]
fixed setting of shortMemoryStatus in MemoryControl
Michael Peter Christen [Sun, 9 Feb 2014 11:25:07 +0000 (12:25 +0100)]
removed interaction fragments
Michael Peter Christen [Fri, 7 Feb 2014 16:19:08 +0000 (17:19 +0100)]
fixed "Size in KBytes" calculation in PerformanceQueues_p.html,
see http://bugs.yacy.net/view.php?id=362
Michael Peter Christen [Fri, 7 Feb 2014 00:47:10 +0000 (01:47 +0100)]
removed unused classes and servlets
Michael Peter Christen [Fri, 7 Feb 2014 00:38:58 +0000 (01:38 +0100)]
removed libraries which had been used by jena and were added together
with jena in commit
5d20cd324af387c72fedec88a8059442c090c769, see:
https://gitorious.org/yacy/rc1/commit/
5d20cd324af387c72fedec88a8059442c090c769
Michael Peter Christen [Fri, 7 Feb 2014 00:20:06 +0000 (01:20 +0100)]
removed jena library and all code that depended on jena. When jena was
introduced, it was also used for search facets. The generic search
facets are now deduced from generic solr fields which makes jena as tool
for facet semantics superfluous.
Michael Peter Christen [Thu, 6 Feb 2014 23:23:50 +0000 (00:23 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Thu, 6 Feb 2014 23:22:31 +0000 (00:22 +0100)]
enrichment of PerformanceMemory display of SolrInfoMBean table
Michael Peter Christen [Thu, 6 Feb 2014 23:02:19 +0000 (00:02 +0100)]
added statistic information for solr resources in PerformanceMemory
reger [Thu, 6 Feb 2014 22:15:00 +0000 (23:15 +0100)]
fix: select all checkbox Tables_p
(needs form name attribute)
Michael Peter Christen [Thu, 6 Feb 2014 21:39:19 +0000 (22:39 +0100)]
re-inserted default value re-set button to performance queues and
patched missing values for recent new queues
Michael Peter Christen [Thu, 6 Feb 2014 20:25:03 +0000 (21:25 +0100)]
Merge branch 'master' of gitorious.org:yacy/heronchatoyants-rc1
Alex Vouilloz [Thu, 6 Feb 2014 19:57:07 +0000 (20:57 +0100)]
updated and completed translation of Steering.html and
ConfigUpdate_p.html into French
Michael Peter Christen [Thu, 6 Feb 2014 18:13:29 +0000 (19:13 +0100)]
enhanced strategy to clear solr caches
- redesigned the instance mirror class (which was a mess)
- added final method to close a searcher (which otherwise keeps a cache)
- changed cache clear method which iterates over resources and calls
clear to all caches in the searcher resources
Alex Vouilloz [Wed, 5 Feb 2014 23:46:14 +0000 (00:46 +0100)]
updated and completed translation of Network.html into French
Alex Vouilloz [Wed, 5 Feb 2014 18:22:59 +0000 (19:22 +0100)]
translated templates into French
Michael Peter Christen [Wed, 5 Feb 2014 14:35:53 +0000 (15:35 +0100)]
Merge branch 'master' of ssh://gitorious.org/yacy/rc1
reger [Wed, 5 Feb 2014 02:45:02 +0000 (03:45 +0100)]
fix not needed getFileExtension().toLower (double)
add missing .getFileExtension
Alexandre Vouilloz [Tue, 4 Feb 2014 23:56:49 +0000 (00:56 +0100)]
completed fr.lng for Status.html
Alexandre Vouilloz [Tue, 4 Feb 2014 20:41:12 +0000 (21:41 +0100)]
completed fr.lng for BasicConfig.html (+modif in de.lng)
orbiter [Mon, 3 Feb 2014 23:18:11 +0000 (00:18 +0100)]
enhanced didyoumean
Michael Peter Christen [Mon, 3 Feb 2014 17:49:03 +0000 (18:49 +0100)]
enhanced didyoumean
Michael Peter Christen [Mon, 3 Feb 2014 16:39:59 +0000 (17:39 +0100)]
reduced color depth of star icons (for smaller file sizes)
Michael Peter Christen [Mon, 3 Feb 2014 16:39:26 +0000 (17:39 +0100)]
small change to start/stop/pause icon style
Michael Peter Christen [Mon, 3 Feb 2014 12:38:26 +0000 (13:38 +0100)]
fixed 'resource leak' warning
orbiter [Mon, 3 Feb 2014 11:45:35 +0000 (12:45 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
orbiter [Mon, 3 Feb 2014 11:44:52 +0000 (12:44 +0100)]
added new suggest method which replaces more-than-one suggestions:
instead of computing suggest permutations of the given words, the
completion of a phrase using the given words is searched in the fulltext
index.
reger [Sun, 2 Feb 2014 02:47:12 +0000 (03:47 +0100)]
remove unused static var from HTTPDProxyHandler
reger [Fri, 31 Jan 2014 22:03:01 +0000 (23:03 +0100)]
fix vocabulary navigator checkbox selection (from last commit)
reger [Thu, 30 Jan 2014 21:57:27 +0000 (22:57 +0100)]
allow combining selection of different search nav's (facets)
- selecting more than one nav combines the 2 selections (with AND)
- unselecting one nav clears all selected
(e.g. select filetype:pdf and /language/fr shows ~ french pdf's only)
reger [Thu, 30 Jan 2014 21:00:32 +0000 (22:00 +0100)]
fix page nav, to keeping modifier
(was new issue)
orbiter [Thu, 30 Jan 2014 09:51:08 +0000 (10:51 +0100)]
added a boost on appearance of combined words (in the same order the
user submitted that) when searching for more than one word
orbiter [Wed, 29 Jan 2014 21:01:40 +0000 (22:01 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
reger [Tue, 28 Jan 2014 21:39:45 +0000 (22:39 +0100)]
URLproxy access check too tight
respect config ip pattern (was own ip)
Michael Peter Christen [Tue, 28 Jan 2014 11:39:58 +0000 (12:39 +0100)]
- faster initialization of index files
- removal of not used space if index files shrink (rare, but possible)
orbiter [Mon, 27 Jan 2014 20:54:52 +0000 (21:54 +0100)]
less logging
Michael Peter Christen [Mon, 27 Jan 2014 14:56:02 +0000 (15:56 +0100)]
fix auth for forced ping
Michael Peter Christen [Mon, 27 Jan 2014 14:43:08 +0000 (15:43 +0100)]
fixed missing corona in network picture
Michael Peter Christen [Mon, 27 Jan 2014 12:55:06 +0000 (13:55 +0100)]
updated bootstrap seedlist list
Michael Peter Christen [Mon, 27 Jan 2014 12:48:20 +0000 (13:48 +0100)]
enhancements for staticIP and ipv6 handling
reger [Mon, 27 Jan 2014 00:09:56 +0000 (01:09 +0100)]
add config value for ssl/https port (default=8443)
adjust server routines to use config
reger [Sun, 26 Jan 2014 20:50:42 +0000 (21:50 +0100)]
disable wrong forward to https on port change
reger [Sun, 26 Jan 2014 00:00:52 +0000 (01:00 +0100)]
improve isOlder check to not overwrite node index with metadata on equal load date
reger [Sat, 25 Jan 2014 23:58:54 +0000 (00:58 +0100)]
remove double jquery-1.3.1.js and adjust header links to jquery-1.3.2
reger [Sat, 25 Jan 2014 21:59:23 +0000 (22:59 +0100)]
add language navigation filter clause to rwi results
reger [Fri, 24 Jan 2014 02:11:25 +0000 (03:11 +0100)]
prepare for a language navigator
works fine to restrict language for local solrSearches.
More work needs to be done to make rwi/remote searches respect the modifier.language restriction.
reger [Thu, 23 Jan 2014 22:56:45 +0000 (23:56 +0100)]
added hint to web.xml and for completeness the full set of hardcoded mappings
orbiter [Thu, 23 Jan 2014 21:48:31 +0000 (22:48 +0100)]
Added a write-enabled checkbox for remote solr servers.
It is now possible to assign every peer other YaCy peers as remote solr
server which are only used for read operations during search. This also
affects crawling: it will exclude urls from crawls which exist on remote
solr/remote YaCy peers.
orbiter [Thu, 23 Jan 2014 18:21:07 +0000 (19:21 +0100)]
changed solr paths according to new default paths
Michael Peter Christen [Thu, 23 Jan 2014 18:01:31 +0000 (19:01 +0100)]
first try to add a generic solr servlet as luke request servlet
Michael Peter Christen [Thu, 23 Jan 2014 17:41:45 +0000 (18:41 +0100)]
changed the web.xml as well to migrate the solr servlet
Michael Peter Christen [Thu, 23 Jan 2014 16:20:49 +0000 (17:20 +0100)]
refactoring: renamed the SolrServlet to SolrSelectServlet for better
naming of more Solr Servlets
Michael Peter Christen [Thu, 23 Jan 2014 16:14:13 +0000 (17:14 +0100)]
added two virtual instances of solr for the both cores: collection1 and
webgraph. These cores are now accessible at
/solr/collection1/select instead /solr/select?core=collection1
and
/solr/webgraph/select instead /solr/select?core=webgraph
in addition to the old behavior to support compatibility to the old
peers. These new paths are fully solr standard-conform and will allow
the cross-linking between YaCy peers using their public solr API.
Michael Peter Christen [Thu, 23 Jan 2014 14:56:36 +0000 (15:56 +0100)]
added button in host browser to re-load 404/failed documents
reger [Thu, 23 Jan 2014 00:52:51 +0000 (01:52 +0100)]
fix search navigator not showing activated nav
introduced with https://gitorious.org/yacy/rc1/commit/
97e84439fb749a624c00ec01a6caa076c9c1c32f
orbiter [Wed, 22 Jan 2014 20:53:27 +0000 (21:53 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
orbiter [Wed, 22 Jan 2014 20:33:11 +0000 (21:33 +0100)]
fix for bad-formatted lonlat
reger [Wed, 22 Jan 2014 20:23:32 +0000 (21:23 +0100)]
optimize parse ServerSideInclude
orbiter [Wed, 22 Jan 2014 19:59:53 +0000 (20:59 +0100)]
fix for http://bugs.yacy.net/view.php?id=354
orbiter [Wed, 22 Jan 2014 19:54:03 +0000 (20:54 +0100)]
fix for OOM during remote search and too high load protection
sixcooler [Wed, 22 Jan 2014 16:52:07 +0000 (17:52 +0100)]
use less ram on dht-URL transfer by not using a URIMetadataNode[]
Michael Peter Christen [Wed, 22 Jan 2014 16:18:53 +0000 (17:18 +0100)]
do not set a load prerequisite that prevents the start of one-time-jobs
sixcooler [Wed, 22 Jan 2014 00:31:22 +0000 (01:31 +0100)]
bump to httpclient-4.3.2
sixcooler [Wed, 22 Jan 2014 00:26:44 +0000 (01:26 +0100)]
workaround to let dht-out run on smal Systems like a Pi
reger [Tue, 21 Jan 2014 20:35:48 +0000 (21:35 +0100)]
put list quick navigator buttons belowon BlackList_p editor
replacing the dropdown -> go navigation
Michael Peter Christen [Tue, 21 Jan 2014 18:28:00 +0000 (19:28 +0100)]
making latency_factor and maximum number of same hosts in loader queue
settings available in Crawler_p.html servlet for steering.
Michael Peter Christen [Tue, 21 Jan 2014 16:53:39 +0000 (17:53 +0100)]
code simplifications / removed warnings
Michael Peter Christen [Tue, 21 Jan 2014 16:52:16 +0000 (17:52 +0100)]
new crawling factors can now be changed during runtime
Michael Peter Christen [Tue, 21 Jan 2014 16:48:45 +0000 (17:48 +0100)]
- removed hardcoded load-test which is now handled in BusyQueues
steering, see /PerformanceQueues_p.html
- changed default values for crawler queue load limit (high, because
these jobs are started upon user request)
sixcooler [Tue, 21 Jan 2014 16:04:22 +0000 (17:04 +0100)]
configurable max-load values for YaCy-Threads:
try lower values on smal systems like a Pi
sixcooler [Tue, 21 Jan 2014 15:56:44 +0000 (16:56 +0100)]
lower client-connection for single-cpu-systems
Michael Peter Christen [Tue, 21 Jan 2014 15:05:55 +0000 (16:05 +0100)]
reverted crawling strategy from latest commit.
Michael Peter Christen [Mon, 20 Jan 2014 20:46:40 +0000 (21:46 +0100)]
enhanced crawler speed
Michael Peter Christen [Mon, 20 Jan 2014 20:09:03 +0000 (21:09 +0100)]
added synchronization to exists() call bacause the concurrent call to
that method showed in thread dump close to deadlock situations. Its also
better to synchronize IO operations because they become faster then.
Michael Peter Christen [Mon, 20 Jan 2014 17:39:36 +0000 (18:39 +0100)]
if a http client thread is still running but we do not wait for it any
more, call an interrupt
Michael Peter Christen [Mon, 20 Jan 2014 17:36:00 +0000 (18:36 +0100)]
cleanup crawl loader jobs
Michael Peter Christen [Mon, 20 Jan 2014 17:31:46 +0000 (18:31 +0100)]
- removed old metadata database and all migration code
- refactored all code which uses URIMetadataRow as standard for word
hash length and word hash ordering and moved that to the class 'Word',
becuase the class URIMetadataRow defined the old metadata data structure
and should be superfluous in the future
- removed unused methods from URIMetadataRow as preparation for further
removal of that class
reger [Mon, 20 Jan 2014 07:12:35 +0000 (08:12 +0100)]
fix IOexception logging issue in DefaultServlet
reason not sure but .logException triggers another exception
reger [Sun, 19 Jan 2014 23:58:17 +0000 (00:58 +0100)]
adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString
- since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic,
adjusted ConfigHeuristic to use OpensearchHeuristic settings only.
For this the default OSD search target list is made available (copied) by default and the other configs are removed.
- the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object,
but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns
just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers)
- started to adjust internal html href references from absolute to relative (currently it is mixed).
For future development we should prefer relative href targets (less trouble with context aware servlets)
reger [Sun, 19 Jan 2014 23:03:55 +0000 (00:03 +0100)]
upd heuristic default list (heuristicopensearch.conf)
- Faroo Web taken out (requires api key) http://www.faroo.com/hp/api/api.html#description
- update Faroo News to new url
- Twitter taken out (change to Api 1.1 not supporting rss) https://dev.twitter.com/discussions/24239
Michael Peter Christen [Sun, 19 Jan 2014 14:21:23 +0000 (15:21 +0100)]
do YaCy p2p connections using a timeout-request which covers the http
request into a separate thread and ignores the furthure result of a
request if that does not answer within the requested time-out. This is a
try to solve a problem with the peer-ping, which hangs whenever a peer
appears to be dead or blocked.
Michael Peter Christen [Sun, 19 Jan 2014 13:47:24 +0000 (14:47 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
Michael Peter Christen [Sun, 19 Jan 2014 13:47:11 +0000 (14:47 +0100)]
experimental removal of synchronization. This should work for all cases
where the size() and isEmpty() method is used only for statistics, which
happens at many locations in YaCy. If these methods are used for
structual reasons (like accessing the last element in an array) then it
may fail or cause other problems. As far as visible, this is not the
case.
Michael Peter Christen [Sun, 19 Jan 2014 13:44:58 +0000 (14:44 +0100)]
removed superfluous synchronization
Michael Peter Christen [Sun, 19 Jan 2014 13:42:26 +0000 (14:42 +0100)]
removed debug code
orbiter [Sun, 19 Jan 2014 00:50:55 +0000 (01:50 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
orbiter [Sun, 19 Jan 2014 00:50:42 +0000 (01:50 +0100)]
added load limit for DHT transfer: RWI acceptance only if local load is
not too high
reger [Sat, 18 Jan 2014 22:27:16 +0000 (23:27 +0100)]
fix bug: Crawl Start (Expert) crawls "?-URLs" even if told not to do so
http://bugs.yacy.net/view.php?id=329
reger [Sat, 18 Jan 2014 17:48:52 +0000 (18:48 +0100)]
Skip remote Solr search if last try showed error
As the solr servlet may not be available (e.g. no public search page, old version, individual access setting) a /solr/select error is
remembered in the seed.dna of the remote peer.
This is not permanent, as flag is not stored and the seed is reloaded on several occasions, it is just a memory of the recent past status.
Might also be set to "not available" on time-out of last try.
orbiter [Sat, 18 Jan 2014 11:55:05 +0000 (12:55 +0100)]
concurrency-solid version of transmission limitation
orbiter [Sat, 18 Jan 2014 11:54:14 +0000 (12:54 +0100)]
removed -d64 jvm option because that causes problems on non-64 bit
linux, see http://bugs.yacy.net/view.php?id=349 and
http://bugs.yacy.net/view.php?id=339
orbiter [Sat, 18 Jan 2014 09:50:36 +0000 (10:50 +0100)]
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
orbiter [Sat, 18 Jan 2014 09:50:24 +0000 (10:50 +0100)]
reduce load during dht transfer