Month: June 2019

Home / Month: June 2019

How to train SpamAssassin

June 28, 2019 | System Administration | No Comments

Shell

In order for this to be effective you need to have a collection of good email (HAM) and a collection of bad email (SPAM). These collections of ham and spam should be 1000+ messages each, and you should probably have more HAM than SPAM.

You should keep updating these folders with new messages as time goes on as spam practices change and you don’t want to run the risk of SpamAssassin thinking a specific year or month is spam.

I like to create a mailbox for this purpose. Lets call it spamtrap@example.net and through IMAP create folders called HAM and SPAM to keep things organized.

If you are using Ubuntu Server, the default bayes path for your SpamAssassin DB is /var/lib/amavis/.spamassassin so that is where we will do our work. Otherwise, check your distro package for details.

$ sudo su -
$ cd /var/lib/amavis/.spamassassin

Next lets check that status of the bayes db.

$  sa-learn --dbpath . --dump magic
0.000          0          3          0  non-token data: bayes db version
0.000          0       1207          0  non-token data: nspam
0.000          0       3784          0  non-token data: nham
0.000          0     177278          0  non-token data: ntokens
0.000          0 1079041431          0  non-token data: oldest atime
0.000          0 1561554929          0  non-token data: newest atime
0.000          0 1561558640          0  non-token data: last journal sync atime
0.000          0 1561526651          0  non-token data: last expiry atime
0.000          0          0          0  non-token data: last expire atime delta
0.000          0          0          0  non-token data: last expire reduction count

Now we will train it, but not SYNC it. Syncing makes any new data live, and you may not want that until you’ve built a sufficiently detailed database.

$ sa-learn --no-sync --dbpath . --progress --ham /Path/To/Mailbox/example.net/SpamTrap/.HAM/{cur,new}
96% [=========================================  ]  25.00 msgs/sec 02m31s DONE
Learned tokens from 33 message(s) (3783 message(s) examined)
$ sa-learn --no-sync --dbpath . --progress --spam /Path/To/Mailbox/example.net/SpamTrap/.SPAM/{cur,new}
98% [========================================== ]  26.23 msgs/sec 00m47s DONE
Learned tokens from 355 message(s) (1242 message(s) examined)

The {cur,new} tell sa-learn to look into both the cur and new sub directories of the HAM AND SPAM folders.

Run the dump magic command again, and if satisfied with the number of tokens, sync the database.

$  sa-learn --dbpath . --sync

That’s all. SpamAssassin is trained up and live.

How to test bind9 config

June 27, 2019 | System Administration | No Comments

Shell

Note: Depending on your install file names and locations may be different

After making changes to the main named.conf it is advised to check the config before restarting the service.

$named-checkconf /etc/bind/named.conf.local

If running in a chrooted environment

$ named-checkconf -t /var/named/chroot /etc/bind/named.conf.local

Like the main config, after editing a zone file it is advised to test it directly.

#$amed-checkzone example.com /var/cache/bind/zones/example.com

programming code

A while back I had to write an app to parse data from an old billing system to be used with a new banks credit card processing system via batch upload. Most of what I do is either on Linux systems and only need to be run directly by me, or are web applications of one form or another, so a standalone application for our billing department was something really new — well not really new, but something I haven’t done in a very, very long time and not with web technologies — and exciting.

At the time I was completely obsessed with Angular and wanted to use it for everything, so of course, I chose it to write the app and after some research, decided on ElectronJS to do the stand alone part of it.

Today we got notice from the bank that the format was going to be changing due to changes in how credit cards have to be handled, so I opened up the code to take a look and refresh my memory only to find that I had completely forgotten how everything worked with Electron. So I fired up a new Angular app and reverse engineered my own code, to record here for posterity.

Start an Angular project and install Electron.

$ ng new electron-app
$ cd electron-app
$ npm install --save electron
$ npm install -D electron-packager

Electron’s main configuration.

// main.js

const { app, BrowserWindow } = require('electron');

let win;

function createWindow () {
  // Create the browser window.
  win = new BrowserWindow({
    width: 900, 
    height: 900,
    backgroundColor: '#ffffff',
    icon: `file://${__dirname}/dist/electron-app/assets/logo.png`
  });

  win.loadURL(`file://${__dirname}/dist/electron-app/index.html`);

  // uncomment below to open the DevTools.
  // win.webContents.openDevTools()

  // Event when the window is closed.
  win.on('closed', function () {
    win = null;
  });
}

// Create window on electron initialization
app.on('ready', createWindow);

// Quit when all windows are closed.
app.on('window-all-closed', function () {
  // On macOS specific close process
  if (process.platform !== 'darwin') {
    app.quit();
  }
});

app.on('activate', function () {
  // macOS specific close process
  if (win === null) {
    createWindow();
  }
});

Update files to use Electron

// package.json

{
  "name": "electron-app",
  "version": "0.1.0",
  "main": "main.js", // <-- ADD THIS
// package.json
  "scripts": {
    ....
    "electron": "electron .",                       // <-- ADD THIS
    "package": "electron-packager .",               // <-- ADD THIS
<!-- index.hmtl -->
<base href="./">

Build and run the application.

$ ng build
$ npm run electron

Build the final, distributable package.

$ ng build --production
$ npm run package

This will create a directory name like electron-app-win32-x64 (or similar depending on your platform), containing all the necessary files to run and an electron-app.exe

Windows Terminal Preview

June 22, 2019 | System Administration | No Comments

Shell

Well, Microsoft released a preview version of its new “Windows Terminal” It’s available in the Windows Store. I’ve been looking forward to checking it out since it was announced. The slow march of Windows conversion to Linux, but they wont call it that.

But, alas. The laptop from which I am typing this is, unfortunately, not sufficient for its install. Being, apparently, behind on updates. Which are now downloading.

So long as this antiquated hardware doesn’t overhead and die, as it is prone to do. Ill get to check out the Terminal tomorrow. I’m crossing my fingers…

Hopefully, this survives long enough to update.

Django: Extending the User Model

June 22, 2019 | Coding | No Comments

programming code

I love Django. It’s a wonderful framework and ORM. Whether you’re building a full website or a restful api, its got you covered. But if you are like me, you HATE the default user model. This generally comes down to it having both a username & e-mail field; and the e-mail field isn’t even unique.

In my world, the e-mail address is the perfect, unique username. So how do you make this happen?

If your app is out, live in the world all ready, you’re out of luck. But if you are firing up a new one, there are a couple of options. Just DON’T RUN THE MIGRATIONS until the new user model is ready.

First thing after starting the new project is starting the users app.

(ENV) $: python manage.py startapp users

Next we are going to create our new manager and user model in users. DO NOT RUN MIGRATIONS! Did you get that? I’ll tell you when.

#users/managers.py

from django.contrib.auth.base_user import BaseUserManager
from django.utils.translation import ugettext_lazy as _


class UserManager(BaseUserManager):
    """
    Our custom user model that uses email as the unique username
    """
    def create_user(self, email, password, **extra_fields):
        """
        Create user.
        """
        if not email:
            raise ValueError(_('The Email must be set'))
        email = self.normalize_email(email)
        user = self.model(email=email, **extra_fields)
        user.set_password(password)
        user.save()
        return user

    def create_superuser(self, email, password, **extra_fields):
        """
        Create superuser.
        """
        extra_fields.setdefault('is_staff', True)
        extra_fields.setdefault('is_superuser', True)
        extra_fields.setdefault('is_active', True)

        if extra_fields.get('is_staff') is not True:
            raise ValueError(_('Superuser must have is_staff=True.'))
        if extra_fields.get('is_superuser') is not True:
            raise ValueError(_('Superuser must have is_superuser=True.'))
        return self.create_user(email, password, **extra_fields)

When we create our user model, we are going to extend AbstractUser. We can extend BaseAbstractUser as well, but that is a lot more work and I really don’t see a need to ever do that at all.

# users/models.py

from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils.translation import ugettext_lazy as _

from .managers import UserManager

class User(AbstractUser):
    username = None
    email = models.EmailField(_('email address'), unique=True)

    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = []

    objects = UserManager()

    def __str__(self):
        return self.email

Notice the username = None and USERNAME_FIELD = ’email’. We can also add any other fields we want here now. Instead of adding a separate 1 to 1 profile model, we can put things right on the user. Such as phone number, or company name, customer id #, etc.

Now we can add it to our settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'users',
]

AUTH_USER_MODEL = 'users.User'

You can go ahead and create and run the migrations now and run python manage.py createsuperuser to set yourself up.

Now we need Forms and Admin integration.

# users/forms.py
from django.contrib.auth.forms import UserCreationForm, UserChangeForm

from .models import User

class UserCreationForm(UserCreationForm):
    class Meta(UserCreationForm):
        model = User
        fields = ('email',)

class UserChangeForm(UserChangeForm):
    class Meta:
        model = User
        fields = ('email',)
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin

from .forms import UserCreationForm, UserChangeForm
from .models import User

class UserAdmin(UserAdmin):
    add_form = UserCreationForm
    form = UserChangeForm
    model = User
    list_display = ('email', 'is_staff', 'is_active',)
    list_filter = ('email', 'is_staff', 'is_active',)
    fieldsets = (
        (None, {'fields': ('email', 'password')}),
        ('Permissions', {'fields': ('is_staff', 'is_active')}),
    )
    add_fieldsets = (
        (None, {
            'classes': ('wide',),
            'fields': ('email', 'password1', 'password2', 'is_staff', 'is_active')}
        ),
    )
    search_fields = ('email',)
    ordering = ('email',)


admin.site.register(User, UserAdmin)

That’s it. You have your own customized user model that you can extend further at will.

Referencing the User model:

There are two ways to get the user model now. You can’t simply import AbstractUser anymore. The preferred method is to use get_user_model() but you can also use settings.AUTH_USER_MODEL. For instance, if you want a foreign key relationship to the user.

# Example
User = get_user_model()
## or User = settings.AUTH_USER_MODEL ## but get_user_model() is preferred

class OurTestModel(models.Model):
     customer = models.ForeignKey(User, on_delete=models.CASCADE)

Edit:

To make this a simpler process I’ve created a GitHub repo that you can drop into your Django project and get started right away.
https://github.com/wsimmerson/DjangoExtendedUser

Shell

Congratulations on using a Version Control System for your code! Believe it or not, not everyone is, well, let’s say, “informed” enough to do it. You’re probably also using GitHub, which is great too. Especially since they now offer private repositories for free.

It’s not always the case, however, that you want to use a solution like GitHub, BitBucket, etc. Sometimes you’re work requirements are such that you’re code is all kept internally to your work network and you need to set up your own remotes. Here is how you do it.

On the server that will host your repos you will need to create a directory to contain them all, create the bare repository for your project then initialize the empty repo. Lets say your repository directory will live off of root and be named git.

$ mkdir /git
$ mkdir /git/my-repo.git && cd /git/my-repo.git
$ git --bare init

Your empty repository is ready to go, so go back to your workstation. Replace username and host with your username and the dns name or ip address of the remote server.

$ git remote add origin username@host:/git/my-repo.git
$ git push origin master

And that’s it. You have have a remote repository under your control.

Django: Unique Slug for Urls

June 21, 2019 | Coding | No Comments

programming code

Let’s imagine a situation where you want to create, lets say, a blog post or a news article in Django. You want to use take the Title and turn it into a slug that can be used in the URL. Meaning, it needs to be unique, or at least, unique for the date. Depending on the URL parameters you with you use.

You could make it a editable field with a unique property, but its probably better to abstract it away and check recursively for existing matches and increment a number on the end. Simplifying the process.

First, lets define the model as such.

class Article(models.Model):
    title = models.CharField(max_length=255)
    slug = models.SlugField(unique=True, blank=True)
    body = models.TextField()
    published = models.BooleanField(default=True)
    published_on = models.DateTimeField(blank=True)
    updated_on = models.DateTimeField(blank=True)
    author = models.ForeignKey(User, on_delete=models.CASCADE)

Next we will need to override the models save method.

def save(self, *args, **kwargs):
        self.updated_on = timezone.now()
        if not self.id:
            self.published_on = timezone.now()

            # find unique slug
            cnt = 0
            while not self.slug:
                if cnt > 0:
                    # Create a new slug string with the counter
                    slug = "{}-{}".format(slugify(self.title), cnt)
                else:
                    slug = slugify(self.title)

                # check to see if a article exists with that slug
                a = Article.objects.filter(slug=slug)
                if not a:
                    # Assign the slug, this will end the loop.
                    self.slug = slug
                else:
                    cnt += 1
        super(Article, self).save(*args, **kwargs)

You could easily abstract that loop out into a function.

Hello world!

June 20, 2019 | Thoughts | No Comments

I’m Back Baby!!!

–Bender

If you’re reading this you probably stumbled on it by accident and have no idea who I am, and that’s perfectly okay. This page is mainly for me anyway.

In the past I’ve had several blogs on various topics, the longest was a technical blog. Little more than notes on various things programming or Linux related. At some point I forgot about it, then thought I had taken it down, then actually did take it down.

Ever since I took it down, its been on my mind. It was up for years, and full of useful information. Something that I hope to accomplish here, again. It’s good to be back.

Before I move on to writing other posts — there are a lot floating through my head at the moment — you probably want to know what this Kodiċi Systems is, yes?

Well, it’s literally nothing. Kodiċi is a Maltese word for code, that I am most likely using completely out of context and improperly. Years ago I had the idea of spinning off a side business, providing support to local businesses in both IT, Web and App development. It was biting off more than I could chew. A demanding full time job and a family. There is only so much I’m willing to sacrifice and time with my wife and kids aren’t in that pool. Ever since I’ve been sitting on this domain, why not put it to use?

So that’s it, my reintroduction to the web is complete. A few more configuration items to complete, then blogging time.