Linux
Tap the Usenet


The Usenet offers lots of information - I wouldn't like it to miss some of them.

A powerful package is Rich $alz's "InterNetNews", say INN. It isn't easy to master the INN. Who made the sendmail may be happy: it becomes more difficult...!

INN would be able to fetch and send news by itself, but we will get some cooperation from a more intelligent package: newsflash and postit.

Again I'll describe it depending on the S.u.S.E. 4.2 Linux distribution. Here INN is easily installable in the /usr/lib/news directory (with the "YaST" tool). By the way you can find here a (sorry, not very useful) overview: /usr/lib/news/README.linux, and INN is pre-installed (if marked in YaST), but all the Documentation is missing. You can get the documentation, if you send an email to majordomo@warren.mentorg.com

  Subject: empty
  sendme faq-inn-1
  sendme faq-inn-2
  sendme faq-inn-3
Then you will receive a long FAQ describing how INN works and how to fix many problems. You also should read the tutorial (in German language) news-0.3.ps by Markus Dickebohm. It describes in a very good way how to install a INN/newsflash/postit system. Have a look at this server: Maybe there are newer tutorials available. And you could also load down the packages newsflash and postit (see below: "Give and Take").

InterNetNews

The first thing we will install is INN. A properly installed sendmail is required.

Preworking

The INN daemon innd does not like any other program listening at the news port, and there may be a line in /etc/inetd.conf causing a fatal error:
  nntp stream tcp  nowait root  /usr/sbin/tcpd /usr/sbin/in.nntpd
You must add a "#" in the first column to make it a comment, otherwise innd will not start (I needed several days to notice that). INN does not run together with C News or any other news daemon, too. It isn't necessary, by the way.

There is one thing you also should do. If INN does not like something it writes an email to "usenet@localhost". You must tell your system that "usenet" is a user of your system - some other mail addresses too. You do it by editing /etc/aliases. Add the following lines (perhaps you have to create a new file):

/etc/aliases  
  news: root
  newsadm: news
  newsadmin: news
  usenet: news
Type in to tell sendmail the new aliases:
  root:# newaliases
  root:# _
Now all emails concerning your Usenet system come to root.

But before we start configuring INN we have to edit the file /etc/hosts and to insert one entry for the news server. Take the address from the database or call the provider's hotline. Add:

/etc/hosts (cutting)
  <IP address>  <name_of_mail_server>    getmail

INN configuration files

There are some configuration files required.
/var/lib/news/inn.conf  (Slackware: /usr/lib/news/inn.conf)
  # Basic installation of INN
  server:	your.computer
  organization:	as you like
  fromhost:     Email_adresse@Host.Computer # for posting header
Thomas Schoepf <schoepf@informatik.tu-muenchen.de> wrote me: if there appears a error message like ME: cant GetConfigValue pathhost... while starting INN, you should add the line pathhost: <Host.Fuer.News.Path>.

/var/lib/news/hosts.nntp (Slackware: /usr/lib/news/hosts.nntp) 
  # Which site feeds our newsreader?
  localhost:
  Rechner.name:

/var/lib/news/nnrp.access (Slackware: /usr/lib/news/nnrp.access)
  # Which computers may tap us?
  localhost:Read Post:::*
  (none):Read Post:::*
  your.computer:Read Post:::*

/var/lib/news/newsfeeds (Slackware: /usr/lib/news/newsfeeds)
  # Where do the news go?
  # We don't feed our news to anywhere.
  ME\
          :*::
  # for NOV database
  @overview:*:Tc,WO:/usr/lib/news/bin/overchan
  # Feed the LRZ with all we have:
  get/getlist\
          :*:Ap,Tf,Wnm:get
Please refer to the INN documentation for more information. providername is the name the provider writes into the news header in the "Path:" line. You must exacltly write it the way your provider does. If you don't the article negotiation will take much more time.

There are some files missing, which can be fetched from the news server. Therefore you could use the script getlist, which should be part of the INN package. If not, load it via ftp:

unix.hensa.ac.uk/mirrors/yggdrasil.fall95.disk1/usr/lib/news/bin/getlist.
Store it in /usr/lib/news/bin and make it executable: "chmod news getlist".

So you can load two files from lrznews using getlist:

  root:# /usr/lib/news/bin/getlist -h getnews active >/var/lib/news/active.raw
  root:# /usr/lib/news/bin/getlist -h getnews newsgroups >/var/lib/news/newsgroups
  root:# _
If you have the Slackware distribution change /var/lib/news against /usr/lib/news.

You even could manage this manually (without "getlist") using telnet:
  root: # telnet getnews 119 >/var/lib/news/active.raw
now wait 10 sec. - *no* prompt appears!
  list active
wait again a few seconds
  quit
wait until your own prompt appears (takes some time):
  root: # telnet getnews 119 >/var/lib/news/newsgroups
now wait 10 sec. - *no* prompt appears!
  list newsgroups
wait again a few seconds
  quit
wait until your own prompt appears (takes some time):
  root: # _
The fresh file active must be edited now. In it you find all the subsribable Usenet groups and the actual posting numbers. Warning: this file is sensitive. It does not like any changes (words per line, number of white spaces or number of digits)! You may not change anything except the things we discuss here!!!

You should do now two things. First, set all posting numbers zo zero. Do it this way:

  root:# cd /var/lib/news           # Slackware: /usr/lib/news
  root:# sed 's/ [0-9]* [0-9]* / 0000000000 0000000001 /' active.raw > active
  root:# _
Second, look if there are entries for the following newsgroups. If they are missing add them now.
  to 0000000000 0000000001 y
  junk 0000000000 0000000001 y
  control 0000000000 0000000001 y

Polish

I found some errors the hard way. INN needs total access to the whole directory /var/run:
  root:# chown root.root /var/run   # (not root.uucp!)
  root:# _
(perhaps you need to play something with the access bits)

For testing look into /var/spool/news whether the directories in.coming, news.archive, out.going and over.view are present - if not, create them using mkdir!

Finally you must give all files in /var/lib/news (Slackware: /usr/lib/news and in /var/spool/news the user news:

  root:# chown -R news.news /usr/lib/news/*    # for Slackware only this one
  root:# chown -R news.news /var/lib/news/*    # if present
  root:# chown -R news.news /var/spool/news/*  # if present
  root:# _
For basic installation call now some assisting programs: makehistory and news.daily. Do it as user news (important!):
  root# cd /
  root# su news -c "/usr/lib/news/bin/makehistory -o"
  root# su news -c "/usr/lib/news/bin/news.daily"
  root# _

The Big Question: Does it Work?

A test start:
  root:# /usr/lib/news/etc/rc.news
  Starting innd.
  root:# telnet localhost 119
  Trying 127.0.0.1...
  Connected to localhost.
  Escape character is '^]'.
  200 my.computer InterNetNews NNRP server INN version date ready (posting ok).
  no prompt appears!
If this message appears you may breath deeply and say loudly "uff!!!". You can escape here typing in quit.

If the message does not appear, sorry: you need to find the bug, and this may take some time. Let the INN-FAQ help you. Perhaps INN could send you an email to "usenet" - respectively "root", in which you may read what happened. If there is no mail perhaps you can find some hints in the syslog, in /var/log/messages or in the log files in /var/lib/news/log.

The INN installation made some gray hairs onto my head and took more than one month time. I tried to tell you all the mistake possibilities, but I feel I could not cover all of them. The INN is not very cooperative while debugging... But I think I could help you in most cases

Start-up of the System

The INN is started while system start in rc.news. You must include it in the start-up sequence.

Again we have the possibls cases sinpleinit or sysvinit.

simpleinit (e.g. Slackware)

In /etc/rc.d/rc.M we define a new file:
/etc/rc.d/rc.M; Sektion "# Initialize the NET subsystem."
  . /etc/rc.d/rc.inet1
  . /etc/rc.d/rc.isdn       
  . /etc/rc.d/rc.inet2
  . /usr/lib/news/etc/rc.news   # <-- add this line!
You also must find the file which is called while system reboot (often this is /etc/rc.d/rc.6). Here you should add before the shutdown commands:
/etc/rc.d/rc.6
  echo -n "Shutting down NNTP port:"
  /usr/lib/news/bin/ctlinnd throttle "shutdown"
  /usr/lib/news/bin/ctlinnd shutdown "shutdown"
  [more shutdown commands]
If you don't some (harmless) error messages appear while shutdown.

sysvinit (z.B. S.u.S.E, Caldera, Red Hat...)

If you have sysvinit (like S.u.S.E. 4.2) it is a bit more complicated. First, create a start/stop file:
/sbin/init.d/news # (Red Hat: /etc/rc.d/news)
  #! /bin/sh
  # Author: Bernhard Hailer <dl4mhk@lrz.uni-muenchen.de>, 24-Jun-96
  # /sbin/init.d/news
  case "$1" in
    start)  
      echo "Initializing NNTP port. (INN)"
      /usr/lib/news/etc/rc.news       #  rc.news wird beim INN mitgeliefert
      ;;
    stop) 
      echo -n "Shutting down NNTP port:"
      /usr/lib/news/bin/ctlinnd throttle "shutdown"
      /usr/lib/news/bin/ctlinnd shutdown "shutdown"
      ;;  
    *)
      echo "Usage: $0 {start|stop}"
      exit 1
  esac
  exit 0
You must set some soft links, because the system needs to know which files have to be executed while starting or stopping:
  root:# cd /sbin/init.d/rc2.d  #  There are other path definitions
  root:# ln -s ../news K20news  #  in other Linux distributions!
  root:# ln -s ../news S20news
  root:# _
Now while booting the INN start-up should appear on the screen - together with the sendmail daemon. Also while shutdown some messages should appear.

Daily Work

The file news.daily we used above should be called daily - if not, INN will mail some reproaches to the user usenet. And therefore we create a small crontab entry for the user news, who should be the only user allowed to execute this script:
/var/cron/tabs/news # (perhaps /var/spool/cron/crontabs/news)
  PATH=/usr/lib/news/bin:/usr/lib/news/etc:/usr/lib/news:/usr/bin:/usr/sbin
  10 07 * * * /usr/lib/news/bin/news.daily 
We tell this now crond:
  root:# chown news.news /var/cron/tabs/news
  root:# crontab -u news /var/cron/tabs/news
  root:# _
Now the script should be called every day at 07:10 a.m.

news.daily does the cleaning and garbage collecting. Old news postings will be erased, and some old log files and so on will be deleted, too. There are two control files which may be interesting:

/var/lib/news/expire.ctl  # Slackware: /usr/lib/news/expire.ctl
  # If posting is erased we will remember its ID 14 days, because
  # we don't want to receive it again:
  /remember/:14
  # (A)ll postings will be kept min. 3, regularly 10 and max. 30 days
  # (min. and max., if an expire date is set in the posting)
  *:A:3:10:30
  # some special newsgroups may remain much shorter:
  junk:A:1:1:2
  control:A:1:1:2

/var/lib/news/control.ctl  # Slackware: /usr/lib/news/control.ctl
  # preset: if INN wants to tell something it has to mail it to admin.
  all:*:*:mail

Give and Take

To operate posting and polling some software packages exist, which are able to cooperate with INN, like newsflash+postit by Chris Blum or suck+rpost by Tim Smith. INN also has some scripts on board which could do it. But the NNTP handshake is very complicated, and the telephone connection should be used very effectively, it is a good choice to take a more intelligent package.

The program suck is more known, but I choosed the (very compact) programs by Chris Blum, because they are highly configurable. They use the telephone line very efficiently, because they open more than one connection to the news server. So, while the news server may create a news packet, there may take place some discussions about the next packet at the same time.

Load these programs down using ftp at

  pople.pc.uni-koeln.de/pub/newsflash-0.99.tgz
  pople.pc.uni-koeln.de/pub/postit-jun96.tar.gz 
(please look if there are newer versions!) Store it in the directory /usr/src and extract them:
  root:# tar zxfv newsflash-0.99.tgz
  root:# tar zxfv postit-jun96.tar.gz
  root:# _

Installation of newsflash

Newsflash fetches some choosed newsgroups (the ones not existing yet at home).

Before compilation you should edit /usr/src/newsflash/config.h and set the path variables correctly for your system (/var/lib/news or /usr/lib/news/newsflash for Slackware). Then:

  root:# cd /usr/src/newsflash
  root:# make dep
  ...
  root:# make
  ...
  root:# cp newsflash /usr/lib/news/bin
  root:# cp printdate /usr/lib/news/bin
  root:# _  
newsflash needs to know which newsgroups it shall retrieve, so you must tell this in a file: it contains the files listing the desired groups:
/var/lib/news/serverlist # or path defined in config.h!
  get1 getnews 119 /var/lib/news/packet1 timestamp
  get2 getnews 119 /var/lib/news/packet2 timestamp
  get3 getnews 119 /var/lib/news/packet3 timestamp
  get4 getnews 119 /var/lib/news/packet4 timestamp
Here four packets are executed, all aim at the news server of the provider. The first word is an ID string, the second is the server defined in /etc/hosts, 119 is the standard news port. The packet description follows, and the last number is a timestamp. You can calculate an active timestamp using the program printdate (included in the newsflash package); the produced 9 digit number tells us how many seconds are gone since January 1st at midnight. To retrieve the new news since the last two days you should subtract about 150000 or so (one day are 86400 seconds) and type in the result as timestamp.

Now we must prepare the packet files we defined above. Newsflash can optimize very good the data transmission parallelizing more than one packet. Here we tell it how to execute the transmission:

/var/lib/news/packet1  # Slackware: /usr/lib/news/newsflash/packet1
  de.comm.isdn
  fido.ger.isdn
  de.comp.os.linux.networking

/var/lib/news/packet2  # Slackware: /usr/lib/news/newsflash/packet2
  de.comp.os.linux.networking
  fido.ger.isdn
  de.comm.isdn

/var/lib/news/packet3  # Slackware: /usr/lib/news/newsflash/packet3
  de.comp.os.linux.misc
  de.comp.os.linux.hardware
  de.comp.os.linux.x
  lrz.*

/var/lib/news/packet4  # Slackware: /usr/lib/news/newsflash/packet4
  lrz.*
  de.comp.os.linux.x
  de.comp.os.linux.hardware
  de.comp.os.linux.misc
As you see the packets 1/2 and 3/4 have the same contents, but in reverse order. So one packet is executed from both ends, and the transmission is optimized. The packets should have the same estimated number of postings.

Installation of postit

Again here: before compilation you should edit /usr/src/postit/config.h and correct the defined paths there (/var/spool/news). Most times this file should be ok.
  root:# cd /usr/src/postit
  root:# make dep
  ...
  root:# make
  ...
  root:# cp postit /usr/lib/news/bin
  root:# cp samples/sendnews /usr/lib/news/bin
  root:# _  
The last file we copied here (/usr/lib/news/bin/sendnews ) must be edited:
/usr/lib/news/bin/sendnews
  SPOOLPATH=/var/spool/news
  OUTGOING=${SPOOLPATH}/out.going
  BINPATH=/usr/lib/news/bin

  ${BINPATH}/ctlinnd flush get
  mv ${OUTGOING}/get ${OUTGOING}/get.postit
  ${BINPATH}/ctlinnd begin get
  ${BINPATH}/postit get.postit getnews

Another Big Question: Working Anyway?

If you call now newsflash the telephone wire should get hot now for some minutes. You should be able to see what's going on on your console. The messages may seem to be mis-ordered, because the process is parallelized.

If it does not work, you must look for the bug. Read the README. In most cases you need to edit again one of the files hosts.nntp, nnrp.access, newsfeeds etc. and play with the entries.

If it works you can read now the news groups with a news reader like nn, tin or Netscape).

To fetch the news automatically you may use the following script, which is very similar to the email fetcher script.

/sbin/transnews
  #!/bin/sh 
  # transnews: gets and sends usenet news.
  # Author:    Bernhard Hailer <dl4mhk@lrz.uni-muenchen.de>
  # Version:   4.0 (27-Jan-97)

  PATH=/sbin:/usr/sbin:/bin:/usr/bin

  . /etc/i4l.provider
  . /etc/i4l.secrets

  cd /                                # against PATH=. effect 

  echo -----------------------------------------------------------------------
  echo -e "Starting news transfer: \c"; date

  . /sbin/connect on

  echo "Checking news server..."
  set `ping -qc5 -i1 getnews | grep transmitted`
  if [ $4 -gt 0 ];
  then
    echo "8-) News server online - starting news poll"  
    echo "Sending news..."
    su news -c "/usr/lib/news/bin/sendnews"
    echo "Again sleeping 1 sec for PPP handshaking"
    sleep 1
    echo "Fetching news..."
    su news -c "/usr/lib/news/bin/newsflash -v"    
    echo "Again sleeping 1 sec for PPP handshaking"
    sleep 1
  else
    echo ":-[ Sorry - news server is down. Try later..."
  fi

  . /sbin/isdn off

  echo -e "Finished news transfer: \c"; date 

Does Your Computer Run All the Time?

Yes? Then you should edit the crontab of root and add the following:
  # Transfer news every night at 04:29 a.m.
  29 04 * * *       /sbin/transnews 1>> /var/log/news-log 2>> /var/log/news-log
As described in the email chapter cron must be called with the new crontab now. The script scould be started one minute before the email transfer starts (not at the same time!!), so perhaps one tariff unit can be saved. If you would start the transfer at the same time, one of the applications would see a open line where no one is (opening a line takes some time). The connection would be interrupted.


Have Fun in the Usenet!!


Copyright © (GPL V 2) 1996 Bernhard Hailer
Last modification: 20-Feb-97