Month: February 2014

Home / Month: February 2014

Shell

I’ve been playing with the idea of converting to Arch Linux, at least on a trial basis, for some time now.  A couple weeks ago I spun up a VM and installed it without a fuss, so it seemed like it would be okay to install on my laptop, but not having a lot of free time I put it off.  The popularity of Arch Linux in my What’s your favorite Linux Distro poll pushed me over the top and I managed to find some time this weekend to sit down and do the install.

It took much longer than my install on the VM for a couple of reasons and would have gone a lot smoother had I been a bit more prepared.  However, now that I’ve been through it and made my mistakes, I’m quite confident I could get through the process in a reasonable amount of time as most of the difficulty came right at the start of the install.   This is some of what I learned from the process of installing Arch Linux: 

  1. UEFI is a pain in the … you know what I am talking about
  2. If using UEFI, disable secure boot.  Not as straight forward as it would seem, at least not on my Acer Aspire.  In order to disable the secure boot, you have to first set a supervisor password, then disable the secure boot, then remove the supervisor password.  It seems like backwards logic to me, in my world the supervisor password shouldn’t become an option unless secure boot is turned on, and turning off secure boot should disable the supervisor password.  But they didn’t ask me when they wrote the UEFI so why should it make sense?
  3. Your Boot partition needs to be flagged for UEFI-EF00. The Arch Linux Beginners guide is excellent and walks you through the install process almost flawlessly with the exception of this.  It tells you how to make the partition, but not how to flag it.  I spent way to much time trying to figure out why my install wouldn’t boot, and scouring the guide for an answer.  The solution? install gdisk and add the EF00 type flag.
  4. If you have multiple partitions or multiple drives, make sure you write down which slice is being mounted where as you create them and make sure you mount them in the correct order before running pacstrap, your boot partition probably doesn’t have enough free space for the whole operating system.
  5. The Arch Linux Wiki is excellent.  Unlike the Ubuntu Answers pages, it actually has answers.  The Arch Linux Wiki is very comprehensive, and covers pretty much everything I had to look up very well, and at the very least, gave me an idea of what I needed to look for to identify the source of the problem.
  6. UbuntuOne client doesn’t like to run unless you have installed extra fonts or a Desktop Environment.
  7. Chrome currently isn’t compatible with the installed libcrypt, so I am relegated to chromium, which is pretty much the same thing.
  8. Installing base-devel is not optional

And that is basically it.  

After the initial UEFI / partitioning battle, everything went pretty well as advertised and has really refreshed my knowledge of the underline workings of Linux.  I’ve been using Ubuntu for my desktop for two long (read: when they first started shipping install disks for free) and it’s fogged my memory of configuration on the command line for desktop services such as X11(I spend most of my working life on server consoles).

Probably the best part for me(so far) is with breaking away from Ubuntu, I’ve been forced to take a serious look at the various window managers and desktop environments out there. Which has brought me to i3.wm, on it’s own its excellent although there are a lot of new key bindings for me to learn and a almost crazy amount of customization to play with, and it is missing some of the standard functionality I want in a desktop environment, such as wall papers, lock screen etc.

Luckily it can replace the window manager in XFCE4 easily and makes for an excellent combination.

Update: After playing with i3 configuration, I’ve dropped XFCE and am using straight i3.wm and loving it.  Also, I’ve seen a lot of comments about this article on reddit.  If you are visiting from reddit and have a comment, please post it below so I see it and can respond. Thanks!

Note: This page was recovered from an old, now defunct, blog.

Python 3: Send SMTP Authenticated Email

February 18, 2014 | Coding | No Comments

programming code

Write enough shell scripts and eventually you will have to write one that sends a email report, in fact now a day’s I find that most shell scripts I write (the scheduled ones anyway) require some kind of notification after they have run. Likewise, if you set up enough mail servers — usually one is enough — you will need a quick easy way to test SMTP.

Below is a python 3 script with smtplib and argparse, which sends a test email using smtp authentication.  It’s short, to the point, and I hope pretty self explanatory for anyone needing to send authenticated email via python.

#!/usr/bin/env python3

import smtplib
import argparse

def testmail(username, password, server, deliveryaddr):
    header = "From: %s\r\nTo: %s\r\nSubject: SMTP Test\r\n\r\nTest Successful" % (username, deliveryaddr)
    
    try:
        server = smtplib.SMTP(server)
        server.login(username, password)
        server.sendmail(username, deliveryaddr, header)
    except Exception as e:
        print("An Error Occurred! %s" % e)
    else:
        print("Test Successful")

if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument('-u', '--user', required=True,
            help="Username for SMTP Login")
    parser.add_argument('-p', '--password', required=True,
            help="Password for SMTP Login")
    parser.add_argument('-s', '--server', required=True,
            help="SMTP Server url")
    parser.add_argument('-d', '--dest', required=True,
            help='Destination Email Address')

    args = parser.parse_args()

    testmail(args.user, args.password, args.server, args.dest)

Note: This post was recovered from an old, now defunct, blog.

How-To: Bond Multiple DSL connections

February 14, 2014 | Networking | No Comments

Shell

…with MLPPP and a MikroTik RouterBoard

Multi Link Point to Point Protocol is a nifty protocol which allows for the bonding of multiple lines.  With a MLPPP enabled DSL account, two or more lines, DSL modems in bridged mode and a bonding router such as a Mikrotik routerboard, you can effectively quadruple the speed (minus a little overhead) of a DSL connection. 

Here’s how! connect to the Mikrotik via the Winbox application and follow along. 

PPP -> “Interface” Tab:       
– Hit the red plus sign       
– pick “PPPoE Client”           
– under the “General” tab:               
1. add the interfaces to be bonded at the bottom.           
– under the “Dial Out” tab:               
1. add the PPPoE authentication information to the “User” and “Password” fields. 

Interfaces -> “Interface” Tab:       
– There should be a pppoe-out1 interface listed       
– Double click each interface you added to the bonded connection in the previous step.           
1. Click the dropdown “Master Port” field, and select “none” (rather than ether2-master).  
This step is absolutely essential.  If you don’t do it, the bonding will not work.

 IP -> DHCP Client:       
– Delete the default DHCP client set up on ether1-gateway, as it will interfere with the PPPoE connection otherwise. 

IP -> DHCP Server:       
– Disable or delete the default DHCP server set up on ether2-master if something else on the customer network is handling DHCP.  If you do not disable this, the routerboard will hand out addresses on the 192.168.88.0/24 network, possibly interfering with the existing LAN configuration. IF YOU NEED TO ADD A BLOCK OF IP ADDRESSES (eg: 172.165.216.232/29 totally made up of course)

IP -> Addresses:       
– Hit the red plus sign       
– Fill in the “Address” field: Give the routerboard the first “usable” IP address in the range (in this example, it is 172.165.216.233), but with the proper netmask.  Don’t worry about filling in the “Network” and “Broadcast” fields, since these are populated automatically when you fill in a netmask in the Address field and hit “OK” or “Apply”.       
– EG: 172.165.216.233/29, and hit Apply.       
– Choose “ether2-master” from the dropdown.  Hit “OK”. 

IP -> Firewall -> “Filter Rules” Tab:       
– Change all default rules from triggering on all -> ether1-gateway, to pppoe-out1, or they won’t take effect.       
– Add a rule for the Routerboard management port so we can connect remotely and make changes:           
General Tab -> Chain: Input, Protocol: 6 (tcp), Dst. Port: 8291, In. Interface: pppoe-out1           
NOTE: Ensure the rule is ABOVE the final drop rule in the default configuration, or attempts to connect to the Routerboard on 8291 will not succeed, and we will not have remote access. 

IP -> Firewall -> “NAT” Tab:       
– Change default rule to do masquerading via pppoe-out1 rather than ether1-gateway (as per default).       
– IF THERE IS A BLOCK OF ROUTED IPs ADDED TO THE ROUTERBOARD:            -> add an additional condition to the NAT rule:  Add the IP range to the “Src. Address” field, and then click the box that appears to the left of the “Src. Address” field.  An exclamation mark will appear, indicating that the rule is INVERSE.  ie: the rule effectively now says that IF a packet comes from an IP address that IS NOT in the range of specified IPs, it should be masqueraded with the external public IP address of the Routerboard.  If a packet comes from a normal LAN private IP address, it will still be masqueraded.  This allows you to have both a LAN of firewalled workstations AND several network appliances that have their own public IP addresses on the Internet.

– Hit the red plus sign       
– pick “PPPoE Client”           
– under the “General” tab:               
1. add the interfaces to be bonded at the bottom.           
– under the “Dial Out” tab:               
1. add the PPPoE authentication information to the “User” and “Password” fields. 

Interfaces -> “Interface” Tab:       
– There should be a pppoe-out1 interface listed       
– Double click each interface you added to the bonded connection in the previous step.           
1. Click the dropdown “Master Port” field, and select “none” (rather than ether2-master).  
This step is absolutely essential.  If you don’t do it, the bonding will not work. 

IP -> DHCP Client:       
– Delete the default DHCP client set up on ether1-gateway, as it will interfere with the PPPoE connection otherwise.

 IP -> DHCP Server:       
– Disable or delete the default DHCP server set up on ether2-master if something else on the customer network is handling DHCP.  If you do not disable this, the routerboard will hand out addresses on the 192.168.88.0/24 network, possibly interfering with the existing LAN configuration. IF YOU NEED TO ADD A BLOCK OF IP ADDRESSES (eg: 172.165.216.232/29 totally made up of course)

IP -> Addresses:       
– Hit the red plus sign       
– Fill in the “Address” field: Give the routerboard the first “usable” IP address in the range (in this example, it is 172.165.216.233), but with the proper netmask.  Don’t worry about filling in the “Network” and “Broadcast” fields, since these are populated automatically when you fill in a netmask in the Address field and hit “OK” or “Apply”.       
– EG: 172.165.216.233/29, and hit Apply.       
– Choose “ether2-master” from the dropdown.  Hit “OK”. 

IP -> Firewall -> “Filter Rules” Tab:       
– Change all default rules from triggering on all -> ether1-gateway, to pppoe-out1, or they won’t take effect.       
– Add a rule for the Routerboard management port so we can connect remotely and make changes:           
General Tab -> Chain: Input, Protocol: 6 (tcp), Dst. Port: 8291, In. Interface: pppoe-out1           
NOTE: Ensure the rule is ABOVE the final drop rule in the default configuration, or attempts to connect to the Routerboard on 8291 will not succeed, and we will not have remote access. 

IP -> Firewall -> “NAT” Tab:       
– Change default rule to do masquerading via pppoe-out1 rather than ether1-gateway (as per default).       
– IF THERE IS A BLOCK OF ROUTED IPs ADDED TO THE ROUTERBOARD:            -> add an additional condition to the NAT rule:  Add the IP range to the “Src. Address” field, and then click the box that appears to the left of the “Src. Address” field.  An exclamation mark will appear, indicating that the rule is INVERSE.  ie: the rule effectively now says that IF a packet comes from an IP address that IS NOT in the range of specified IPs, it should be masqueraded with the external public IP address of the Routerboard.  If a packet comes from a normal LAN private IP address, it will still be masqueraded.  This allows you to have both a LAN of fire-walled workstations AND several network appliances that have their own public IP addresses on the Internet.

Note: This post was recovered from an old, now defunct, blog.

Shell

If you are able to connect to the devices web console (default user/password is Admin/Admin) you can reset from the web page.  You may need to do some detective work to find out what IP it is pulling from DHCP if it wasn’t statically assigned. If not follow the steps below for a manual reset.

  1. Disconnect the Ethernet cable.
  2. Use a paperclip to press the reset button (located on the back of the AudioCodes), the FAIL light on the front of the AudioCodes should come on. Hold the pin down for 6 to 10 seconds.
  3. When you release the reset button all LEDs will glow brighter than while the button was held, and will switch on/off a couple times while the system resets to factory defaults.

You may want to let it sit for a minute to make sure all the configuration has been applied and the system is ready to go. Reconnect the Ethernet cable, and connect to the web terminal to set it up.

Note: This post was recovered from a old, now defunct, blog.

Learn Python by Building an IRC Logging Bot!

February 6, 2014 | Coding | No Comments

programming code

When I first sat down to learn Python, I was instantly disappointed with all the the available resources, and the fact that almost none of the tutorials I found seemed to work.  In fact I got so frustrated with the language that I shelved the language and went to Ruby for several months.  It wasn’t until I paid a visit to the Windsor Hackforge and met 3 Python fanatics that I decided to take another look.

What I found during my second look at Python was much different, now knowing a couple avid Python programmers I had people who could answer questions and my problems began to become clear.  The source of most of my frustration was caused by the incompatibilities between Python 2 and 3.  I always want to keep my software up to date so it meant using Python 3, but pretty much every example, tutorial, and book I looked at was written in Python 2.

Its incredibly frustrating when you cant even print text to the console!

The other side of the frustration was the boring tutorials,  I mean how many different ways can you generate or manipulate the Fibonacci sequence anyway?  Sure they were full of good information, but when you’ve learned several languages over the years all these tutorials are just a repetition.  I needed something different, interesting, and potentially useful to spend my time working on.

So I set my sights on writing a IRC Bot — How 1995 is that? — but it was fun.  It dawned on me after that, wouldn’t it be great if programming tutorials actually walked you through writing something useful or entertaining? I mean, its one thing to learn that “This is how you declare a variable” or “This is how you write a loop” and completely another thing to see how to implement them in the structure of the program.

I’ve gone back and re-written my original bot, took a lot of it out and tried to smooth out some of the rough edges, and updated it so it runs properly in Python 3.  Without further Ado…

How to write a IRC Logging Bot in Python 3

First things first… lets skip those.  I’ll assume you have Python 3 already installed on your system and know how to use a text editor to create your script.  This is a small program and fits nicely in a single file, so open up your editor of choice and follow along.

At the top of our file we are going to put in our shebang line and import a few packages.

#!/usr/bin/env python3

import sys
import socket
import string
import argparse 

Now in order to handle an instance of an ircbot, I’m going to create an object.  If your not familiar with OOP (Object Oriented Programming) this may be a bit tricky for you, but Python makes this extremely simple.

We define a new class with the class keyword and the class name as such:

class ircbot():

Make a note of that colon. Everything will need to be indented based on it after this.  The next thing we are going to do is create a constructor, which will populate some variables and establish the connection to the IRC server.

 class ircbot():
    
        def __init__(self, host, port, channel, name, log):
                self.HOST = host
                self.PORT = port
                self.CHANNEL = channel
                self.NICK = name
                self.IDENT = name
                self.REALNAME = name
                self.LOG = log 

The def __init__ you are seeing is a function which is automatically called when the object is created (we will do that later), make note of the “self” in the definition.  Its needed by all class functions and global variables, in order for other functions to have access to them.

This isn’t the full constructor either, the next thing we are going to put in is the connection to the server.  And just so that we don’t get any messy errors thrown at us, I’m also going to introduce exception handling via try-except.  This isn’t in depth, but serves as a good example for your future programs.

   def __init__(self, host, port, channel, name, log):
                # Everything above, after self is a variable we are going
                # to expect every time a instance of ircbot is created
                self.HOST = host
                self.PORT = port
                self.CHANNEL = channel
                self.NICK = name
                self.IDENT = name
                self.REALNAME = name
                self.LOG = log

                try:
                        # Any exception thrown here will halt the program, jumping
                        # down to the except block below
                        self.irc = socket.socket()
                        self.irc.connect((self.HOST, self.PORT))

                        # If you have played with Python 2 you will notice a difference
                        # in the below print statement, print is now a function
                        print("Connected to %s\n" % self.HOST)

                        self.irc.send(("NICK %s\r\n" % self.NICK).encode())

                        self.irc.send(("USER %s %s bla :%s\r\n" % (self.IDENT,
                                                                   self.HOST,
                                                                   self.REALNAME)).encode())
                        self.irc.send(("JOIN %s\r\n" % self.CHANNEL).encode())
                        print("Joined %s as %s\n" % (self.CHANNEL, self.NICK))
                except Exception as e:
                        # Exception is generic all built-in, non-system-exiting 
                        # exceptions are derived from this class.
                        # which makes it a simple way to catch whatever and print a 
                        # clean message and exit with a system 
                        # error code
                        print(e)
                        sys.exit(1)

There, if we created a instance of ircbot, with the correct info in host, port, channel, name and log it would connect to the irc server.  It won’t do us much good through, its not reading responses or handling the PING/PONG that the IRC server will send to see if we are still connected and it certainly isn’t going to write all the activity to a log file for us either.  Lets finish this off the ircbot class by creating a “run” function,  it doesn’t have to be run.  In fact it doesn’t even have to be a separate function, we could just as easily write this into the __init__, but I don’t like to put to much into the constructor if I don’t have to.

   def run(self):
                """
                        Main Application Loop
                """

                # creating a tring variable to  dump irc data into
                readbuffer = ""

                # Everything the program does is going to happen here
                # an infinit loop of catching data and parsing it
                while True:
                        readbuffer = readbuffer + self.irc.recv(1024).decode()

                        # We don't need everything the server sends to us
                        # so we split it into a list and then drop the first part
                        temp = readbuffer.split("\n")
                        readbuffer = temp.pop()

                        # Now we are going to step through the rest of the
                        # lines in the list
                        for line in temp:
                                # Here I am going to strip off line endings
                                # and split the string into a list using
                                # white space as the separator. It's not
                                # necessary, but useful for parsing commands
                                # if you grow the bot that way
                                linex = line.rstrip()
                                linex = linex.split()

                                # PING PONG!
                                # When the IRC Server sends us a ping we
                                # best respond, or it will drop us
                                if (linex[0] == "PING"):
                                        self.irc.send(bytes("PONG %s\r\n" % linex[1]))
                                else:
                                        # And here we handle printing to the screen
                                        # or to a file
                                        if self.LOG == 'stdout':
                                                print(line) 
                                        else:
                                                with open(self.LOG, "a") as log:
                                                        log.write(line)

Now the IRC Logger Bot is complete.  You could import this and use it in another program, but lets make it runnable as one file.  I’m going to add another function that will use the argparse module to give us some Command line power, as well as give some default settings.

def parseargs():

        parser = argparse.ArgumentParser()
        parser.add_argument('-s', '--server', default='irc.freenode.net',
                             help='DNS address of the IRC server. 
                            default=irc.freenode.net')
        parser.add_argument('-p', '--port', type=int, default=6667,
                            help='port number of irc server.default=6667')
        parser.add_argument('-c', '--channel', default='#python-unregistered',
                                help='IRC channel to join. default=#python-unregistered')
        parser.add_argument('-n', '--name', default='irc_logger_bot',
                                help='how the bot will be identified in the channel. default=irc_logger_bot')
        parser.add_argument('-o', '--output', default='stdout',
                                help='file to write log to. default=stdout')

Most of these parser lines are the same, but take an extra look at the -p/–port option. It has one more configuration flag than the others.  I’m sure you can figure that out on your own, Python makes coding easy.

Once we add our final lines to this project, you will be able to run this script with a -h or –help and get a shiny helpful output like:

usage: tutorial_bot.py [-h] [-s SERVER] [-p PORT] [-c CHANNEL] [-n NAME]
                       [-o OUTPUT]

optional arguments:
  -h, --help            show this help message and exit
  -s SERVER, --server SERVER
                        dns address of the irc server.
                        default=irc.freenode.net
  -p PORT, --port PORT  port number of irc server. default=6667
  -c CHANNEL, --channel CHANNEL
                        irc channel to join. default=#python-unregistered
  -n NAME, --name NAME  how the bot will be identified in the channel.
                        default=irc_logger_bot
  -o OUTPUT, --output OUTPUT
                        file to write log to. default=stdout

Finally, we are going to make this run if executed from the command line by using if __name__ == ‘__main__’:

if __name__ == '__main__':

        # Get the options from the parser
        opt = parseargs()

        # Create bot object and run it!
        bot = ircbot(opt.server, opt.port, opt.channel, opt.name, 
                     opt.output)
        bot.run() 

What does if __name__ == ‘__main__’ do?

Well when the python interpreter runs it creates some special variables such as __name__, if a file is executed it is assigned the name __main__.  using __name__ == ‘__main__’ in this way allows us to cleanly either import the file as a module, or execute specific code.

And that it, program complete.  I hope you have found this post both helpful and useful.  

The full code can be found at https://github.com/wsimmerson/ircbot/blob/master/tutorial_bot.py

Thanks for reading!

Note: This post was recovered from an old, now defunct, blog.

Shell

     For those that administer postfix email servers that see high volumes of email, dealing with Spam can be quite a challenge. Initially I was relying on the server to do its own Spam checking with tools such as amavisd-new, spamassassin, etc.

Which was working really well for the most part, but when the Spam volume gets high, performing all that processing, in the hundreds of thousands of emails, was putting unnecessary load on the server.      

Enter Real-Time Black-hole lists or DNS Black-hole lists — among other names — which allow us to block suspected Spam before it is even processed by the server. I was originally against RBLs having had servers land on them and having to go through the annoyance of getting unlisted.

There are even a few that I would consider outright extortion; asking you to pay a fee to expedite the un-listing process or a one time fee to never be listed again. That being said, the more reputable ones — at the time I’m writing this, I’m thinking spamhaus, sorbs, spamcop. — will remove you on request within 24 hours without demanding payment, and are quite effective. 

How does a email server get listed? 

There are several reasons a email server will get listed on a Real-time Black-hole List, some of which are: Sending copious amounts of Spam. (Duh)Not having a valid reverse DNS entry. Not having a proper SPF record.Server is in a Dynamic IP range.Server is in the same subnet as a known and aggressive spammer.The Server is new.      

These are all pretty easy to fix, with the exception of the first one, Sending copious amounts of Spam. The only way to truly put an end to this is to make sure your system is secure, an ever ongoing process at best, and when compromised it can be hard to catch it in time to prevent getting listed.

I suggest requesting AOL white list your servers; when it is approved any report from an AOL customer of Spam from your server will generate a email report, which will help catch any problems early as AOL is a prime target for Spam. 

How do RBLs work? 

Real-time Black-hole Lists are basically a Reverse DNS pointer which look something like this. 

129.12.222.192.dnsbl.example.com.  IN A  127.0.0.3                                                       IN TXT “SPAMMER, Banish to Oblivion” 

That should be a obvious enough of a fake IP, but thats it. The email server performs a reverse lookup with the appropriate RBLs info attached, and acts accordingly. Pretty simple! 

How do I make Postfix use a RBL?      

Configuring Postfix to use a Real-Time Black-hole List is extremely simple, once you have decided on which RBLs you wish to use.

In your Postfix main.cf you should already have smtpd_recipient_restrictions configured, if you don’t you may want to look into that before you go any further.

Assuming you already have it set up, you can simply add a reject_rbl_client dnsrbl.example.net to the end of it. 

Here are a few real entries you can add to the end of your smtpd_recipient_restrictions: reject_rbl_client,
sbl-xbl.spamhaus.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client dnsbl.sorbs.net,
reject_rbl_client list.dsbl.org

And that’s sudo postfix reload Your good to go!

Note: This post was recovered from an old, now defunct, blog.