SETI@home Technical News ReportsThis page contains the latest technical news on the astronomical and computer-system sides of SETI@home. We'll try to update it frequently. August 18, 1999 Added audio files (.wav, .au) to web site. August 17, 1999 Added a histogram of number of results returned per user. We're having delays in setting up Informix on the 450 with the RAID. Informix is sending out someone later this week. Temporarily put the Enterprise 450 on splitter duty, hence the rise in the splitting rate. August 15, 1999 Put the poll mechanism online at about 1:30 AM. Within 10 minutes there were a dozen responses. August 12, 1999 User-account creation hasn't worked for the last 24+ hours. (We added a field to the database but forgot to recompile all programs.) August 11, 1999 The Informix server software has been installed on the 450 server and is working OK using a single disk. Tomorrow we'll try it out with the RAID. Added an alphabetical list of groups overall and within categories. It will be available soon. Implemented a new "poll" mechanism. Will test it for a day or two, then go online. Changed time display from "hours (years)" to simply years if it's more than one year. August 6, 1999 The replacement of the circuit breaker went smoothly and SETI@home is back online. August 5, 1999 More on the power outage: We've just learned that the power to the Space Sciences Laboratory will be taken down tommorrow August the 6th, at noon PDT. The power will be down for approximately two hours. During those hours, SETI@home's servers will be offline and unavailable. August 4, 1999 We have just received word that power to the Space Sciences Laboratory building will be down for a period of two hours sometime this Thursday or Friday in order to replace a faulty elevator circuit breaker. We are currently working to find out the exact time and date which we will post to this page once we know the specifics. During this power outage all of SETI@home's servers, including the data server and the web server, will be unreachable.
July 30, 1999 Server upgrade update: we have received the RAID controller board for the new Sun 450, and have set up about 150 GB of disks as a RAID-5 array. This will be used for the science part of the database. There have been intermittent hardware problems with a couple of the machines running the splitter; that's why our rate of splitting has been going up and down. Fiddled with gnuplot to make nicer-looking graphs. A few ISPs were rejecting mail sent from SETI@home, so some users were unable to get their passwords. We reconfigured our mail server, so these users should now be able to receive our e-mails. July 27, 1999 We have added a new section to the web site showing a sky map of work units generated along with line graphs of some of the project's interesting statistics. July 23, 1999 A power outage on the Berkeley campus brought down our network between the hours of 2:30pm and 5:00pm. July 21, 1999 Added "Last 24 Hours" column to Totals page under statistics; added FLOPs row (based on 2 TeraFLOPs per result). Added top 100 users under Locations (home, work etc.). July 17, 1999 We are awaiting the arrival of a RAID controller card and database server software for our Sun 450 server. When these items arrive we'll do a major (and hopefully final) upgrade on our server architecture. July 15, 1999 The SETI@home/SERENDIP receiver at Arecibo Observatory, which was struck by lightning, has been repaired by the Observatory staff. The receiver is back on line and we are collecting data. July 12, 1999 We recalculated the scores for each group, so each group now has the credit of all its current members. We did this to fix the problem created by the Mac bug that processed workunits in 10 seconds. We corrected the scores of the users that were affected by this bug, but because people can switch from one group to another there is no record of which group they were currently on when the bug was inflating their scores. So, we just started over. This is a one-time event, however. From now on, group scores will be calculated as they were before. Fixed the group creation problem and removed directory browsing from the /stats and /stats/team directories. Also, the box that contains the group name in the "Top 100" is now only 100 pixels wide. July 9, 1999 We modified the personal statistics CGI so that it now states your rank based upon percentage, in addition to place. July 8, 1999 One of the splitters has died a horrible death, putting us back down to two. Tech support has been called... We modified the "Top 100" Group listings so that HTML tags in the Group's name are no longer interpreted as HTML, but are printed as-is. You can still add HTML to your Groups' page, however. July 7, 1999 We've found and fixed the bug in the Mac 1.05 version. We're testing a new version and will release it probably tomorrow. The bug introduced a lot of bad data into our accounting database; some people got credit for thousands of results that took 1 or 2 CPU seconds to compute. In the interest of fairness, we're going to attempt to "undo" these credits, including the totals under teams, CPUs, countries, etc. So some people may see their totals go down. July 6, 1999 The personal statistics look-up now has your credit, your group, groups you have founded and shows your overall SETI@home rank. You can accesss these stats via either the "User Account Area" or the "Current Statistics" pages. July 4, 1999 Someone broke into our web server and replaced our home page with a picture of Alf. This understandably caused concern about the security of our FTP servers (separate machines from the web server). These are highly protected, but we will double-check our security mechanisms. July 3, 1999 The SETI@home clients for MIPS/Irix (5.3, 6.2 and 6.4) had a problem affecting the data analysis due to a bug in the source code. They have been removed from the download page, and results from these versions are being discarded. July 2, 1999 At 1 AM this morning we noticed a bug in the new Mac client (1.05); it sometimes returns results after only a few CPU seconds. The problem apparently goes away if you reboot. We are trying to fix this bug, and will hopefully have a new Mac version soon. In the meantime, we have reverted to the previous version (1.0) for downloading, and we changed the server to ignore results generated in less than 10 CPU minutes. July 1, 1999 Due to a burst steam pipe in the vicinity of the campus-wide domain servers and routers, the entire Berkeley campus spent most of the day disconnected from the Internet. Sorry for any inconvenience. June 30, 1999 The garbage collector, the server, and the splitters have reached an equilibrium state. Work units are being generated as fast as they are being deleted, and they are being deleted about as fast as results are coming back. June 29, 1999 The third fast splitter machine is now online. We've taken the slow one offline, and transferred the tape it's working on to the new machine. June 28, 1999 We added a mechanism that allows the founding member of a group to edit the groups' information, such as the name, description, url and type. To edit a group, go to the groups' page and click on "Edit this Group". We now have 750,000 work units on disk. The splitters are running at about 140,000 work units per day, a bit slower than our first estimate. A third fast splitting machine should come on-line this week. We're approaching the disk free-space threshold where the garbage collector will kick in and start deleting work units for which results have been received, or which have been sent to multiple users. The SETI@home server has been moved to a new, faster machine. The 360 GB of work unit storage, which was previously divided between two machines, is now consolidated here. The server was down for a couple of hours today while we made these changes.
June 27, 1999 We modified the SETI@home server to user a fixed set of processes, each with a persistent database connection. This should make the server somewhat more efficient, bacause it eliminates a process fork and the creation of a database connection. It also eliminates the need for a separate process to handle user accounting updates. June 26, 1999 The fast splitters finished tapes 09ja99aa and 02mr99aa. The fast splitters started on tapes 10ja99aa and 01mr99ab (not to be confused with 01mr99aa). The slow splitter is still chugging away on 01mr99aa. June 24, 1999 As a result of a server bug (now fixed) credits for the last 24 hours were lost. We modified the stats pages so that HTML tags in the User's name are no longer interpreted as HTML, but are printed as-is. Also, the mailto tag was added to the User's email address when displayed in stats. June 23, 1999 Two new fast machines are splitting data now. The effective rate should be near 180,000 work units per day. Modified the SETI@home server to detect when a user is repeatedly sending us the same result. June 22, 1999 We cross-check interesting results by analyzing the work unit ourselves. We did this for the work unit containing the highest-score Gaussian, and didn't get the same result, so we removed that result from the database. New hardware from Sun (3 Ultra-10 workstations and 6 DLT drives) has arrived. We are configuring the machines. June 21, 1999 We are adding mechanisms for group leaders to edit their group information and to merge their group with another group. June 20, 1999 The modified accounting scheme is in place and seems to be working; credits will now appear synchronously in the screensaver. Writing work units from WS4 to WS3 (via NFS) caused WS3 to get NFS timeouts and run out of processes. Splitting is turned off until tomorrow. June 17, 1999 We changed the way accounting is done so that the User Info fields in the screensaver will be updated immediately after a result is sent (this has become our #1 user complaint). When a result is received, the user record is updated immediately (not by the process that handles the request, but by a dedicated process using a shared-memory scheme similar to that used for work units). The offline accounting program (which has now almost caught up, by the way) still handles other credits. June 16, 1999 Due to popular demand, we reset the group credits (CPU time, #results) to the sum of the credits of all people belonging to the group. June 15, 1999 Accounting for groups has not been working correctly. Credit was being added when people originally joined, but not for their work thereafter. This is now fixed, but we can't restore the missing credits. We've decided to stop doing accounting of "work units sent" for anything other than individual users. This will be removed from the Statistics pages soon. The work unit pipeline is now fully operational again. Users will be seeing data recorded on a variety of dates over the last few months. June 14, 1999 As a result of the inode problem we've made some changes to the server architecture. The idea is a sort of triage: 1) time-critical things: do them right away; 2) necessary but not time-critical things: queue them up; increase efficiency by merging database updates; 3) other things: apply all remaining resources, but do not queue. Here's a summary of our server functions and how we're handling them:
June 12, 1999 Result files have been building up and we've been running out of inodes on that file system. This tends to bring down the data server. June 11, 1999 We are adding a mechanism for users to edit their accounts (email address, name, etc.) through a web interface. June 9, 1999 We now have a fourth Sun workstation, WS4, a Sun Sparc 20. We moved the DLT drive to WS4 and are running the Splitter again. We plan to modify the SETI@home server to use the database more efficiently, querying it for a thousand work or so units at a time to send. June 7, 1999 We have formulated a plan to use the new Sun hardware when it arrives later this month. In a nutshell:
June 4, 1999 We learned that the SETI@home server has been repeatedly issuing the same 115 work units for the last few days. This was fixed immediately, and we put a notice on the web site. We're back to issuing our existing 31,415 work units. The problem was due to a subtle operating system bug, but we feel bad that we didn't catch it sooner. June 1, 1999 Sun Microsystems has offered a substantial donation of new hardware: 4 workstations, 4 DLT drives, a large amount of disk storage, and a 450 4-CPU Enterprise Server. The workstations and tape drives are schedule to arrive 6/15, and the 450 is slated for end of June. May 29, 1999 The receiver front end at Arecibo is not working; engineers are working to fix. Fortunately we still have a large backlog of tapes starting from December 1998. May 28, 1999 Quantum has donated about 360 GB of disk drives. May 27, 1999 To address performance problems, we have changed the way our server works. Before, when a user requests a work unit, the SETI@home server queried the database server to find the best work unit to send, and it updated the "# work units sent" field of the user, their country, their team, etc. This was overloading the database. In our new scheme, the SETI@home server avoids using the database as much as possible. It generates "flat files" that record work units sent and results received. An off-line program processes these files and updates the accounting records in our database. As a result of this "delayed accounting", users won't see credit for their results immediately on the web site or in their screensaver. There will a delay, depending on how far behind the off-line processing gets. We changed the word "Team" to "Group" on the web site because the SETI Institute has a fundraising program called "Team SETI", and there was concern about confusion. May 25, 1999 Our fix to the team mechanism has ANOTHER loophole. We changed the mechanism again (this time for sure??). We didn't zero out group membership this time. May 24, 1999 Our Team mechanism has a problem: it's possible to add any user to your team. Some teams have used this loophole to "hijack" users with lots of CPU time. We fixed this (we thought) and emptied out all teams. May 23, 1999 We added the "Team" mechanism. May 22, 1999 Because of the unexpectedly large number of users, our server has become overloaded. Clients are unable to connect and get error messages. We're getting thousands of email messages every day about this. To reduce the CPU load on WS3 we have temporarily stopped running the Splitter. We have 31,415 work units on disk right now, and we'll issue these until we can start generating new ones. May 17, 1999 The official project launch. Our server system consists of three Sun workstations:
The Splitter, which is CPU-intensive, can generate work units only at about 35% of the rate at which data arrives. Once we get more tape drives and more CPUs, we'll be able to keep up with the flow of data. Until then, we'll accumulate a backlog of tapes. An important component of SETI@home, Phase 2 Processing, is under development. This phase involves sifting through our database of Spike and Gaussian records, searching for events that are identical in space and frequency by different in time (typically by several months). May 14, 1999 We added www.cdrom.com as an FTP mirror site (thanks, guys!). May 13, 1999 The Windows and Mac versions of the client were put online today. This happened to coincide with a national TV news story, and immediately our FTP server was overwhelmed. We hastily borrowed several old Sparc IPCs and set them up as FTP servers, but this didn't solve the problem. |