Tag Archive : Python

/ Python


July 10, 2019 | Coding | No Comments

python programming language

Here I am, apparently late to the party again. Set in my ways, using the same old — trustworthy — tools and along comes another one, seemingly better.

Pipenv will eventually be included in pip itself, but for now we have to pip install it. Hopefully you’re using Python 3, because that’s where you need to get it, although once you have it you can use it to create Python 2 environments.

Pipenv gets rid of the necessity for a requirements.txt file — but can both use and generate them — instead using two files a Pipfile and Pipfile.lock. The pipfile to me, is very reminiscent of a npm package.json while the Pipfile.lock “locks” the specific package versions, dependencies and hashs.

Instead of creating the virtual env specifically pipenv creates one in your home directory based on the current folder name. You can get around this by creating a .venv directory in your current folder, which pipenv will prioritize.

C:\code\myproject>pipenv shell
Creating a virtualenv for this project…
Pipfile: C:\code\myproject\Pipfile                                                                                      Using c:\users\wayne\appdata\local\programs\python\python37-32\python.exe (3.7.0) to create virtualenv…
[ ===] Creating virtual environment...Already using interpreter 
Installing setuptools, pip, wheel...

Successfully created virtual environment!                                                                               Virtualenv location: C:\Users\Wayne\.virtualenvs\myproject--MmAU6cW
Creating a Pipfile for this project…                                                                                    Launching subshell in virtual environment…

You will see that a .virtualenvs with the current folder name was created in my home directory. But if I create a .venv directory and run pipenv shell again…

C:\code\myproject>pipenv shell
Creating a virtualenv for this project…
Pipfile: C:\code\myproject\Pipfile                                                                                      Using c:\users\wayne\appdata\local\programs\python\python37-32\python.exe (3.7.0) to create virtualenv…
[ ===] Creating virtual environment...Already using interpreter 
Installing setuptools, pip, wheel...

Successfully created virtual environment!                                                                               Virtualenv location: C:\Users\Wayne\.virtualenvs\myproject--MmAU6cW
Creating a Pipfile for this project…                                                                                    Launching subshell in virtual environment…

Now we have a new one in the .venv

We can also specify a specific python version by pipenv –python=version.number

We now have a Pipfile in our project directory that looks like:

name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true



python_version = "3.7"

Notice the seperation of dev-packages and packages. We can now deploy our projects and not include test runners that found their way into requirements.txt in production. Lets install something.

$ pipenv install flask

Pipenv has been updated to:

flask = “*”

We also now have a Pipfile.lock file that looks like:

    "_meta": {
        "hash": {
            "sha256": "a82b674d67d29678775ff6b773de1686a9593749ec14483b0d8e05131b662286"
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        "sources": [
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
    "default": {
        "click": {
            "hashes": [
            "version": "==7.0"
        "flask": {
            "hashes": [
            "index": "pypi",
            "version": "==1.1.1"
        "itsdangerous": {
            "hashes": [
            "version": "==1.1.0"
        "jinja2": {
            "hashes": [
            "version": "==2.10.1"
        "markupsafe": {
            "hashes": [
            "version": "==1.1.1"
        "werkzeug": {
            "hashes": [
            "version": "==0.15.4"
    "develop": {}

This file includes the specific version of flask and all of its dependencies. This file will be updated every time we install, upgrade or remove a package.

This would be a incredibly long post if I continued to post demonstrations for everything this can do, so below is a descriptive cheat sheet that should be self explanatory.

Cheat Sheet

Activate the shell
pipenv shell

Initialize environment with a specific python version
pipenv –python=2.7.3

Install from Pipfiles
pipenv install

Install in production
pipenv install –deploy

Install a development dependancy
pipenv install nose –dev

Install all packages specified in Pipfile.lock
pipenv sync

Install from a requirements.txt file
pipenv install -r ./requirements.txt

Check for known security issues in your installed packages
pipenv check

View a graph of all your dependencies
pipenv graph

Update the Lock file before deploying
pipenv lock

Ignore the pip file
pipenv install –ignore-pipfile

Installs all packages specified in Pipfile.lock
pipenv –rm

Show the path to the virtual env
pipenv –venv

Create a requirements.txt file (if your team refuses to change to pipenv)
pipenv run pip freeze > requirements.txt

Deactivate the shell

PyCharm Save Actions

July 9, 2019 | Coding | No Comments

programming code

I think I’m late to the party on this, but wanted to throw out my gleeful support of this plugin. Automatically cleanup and/or format your code when you save a file, or run against several files.

This should be part of PyCharm itself and not just a plugin, its that useful. https://plugins.jetbrains.com/plugin/7642-save-actions

If you use PyCharm on a daily basis like me, just do yourself the favor and install it.


July 9, 2019 | Coding | No Comments


I’ve been a bit bored, haven’t been able to come up with a new personal side project for a while. Last night I realized I hadn’t done any browser automation stuff, and what better way of learning it than building a Twitter bot?

Caveat: This is probably a violation of Twitters terms of service so use for educational purposes only!

So what I’ve done is taken Python and with Selenium automated the process of signing into Twitter, searching a specific hashtag, get a list of tweets and like them. All this interspersed with some short, random weight times to mimic more human like behavior.

The repo can be found at https://github.com/wsimmerson/twitterbot but it is only two files, the bot itself and a config file with variables such as username and password, excluded from git of course.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import random
import time
import sys

import config

class TwitterBot():
    def __init__(self, username, password):
        self.username = username
        self.password = password

            self.hashtag = config.HASHTAG
            print("config.HASHTAG must be defined")

            self.pages_to_scroll = config.PAGES
            self.pages_to_scroll = 10

        self.bot = webdriver.Chrome()

    def activate(self):

        usernameBox = self.bot.find_element_by_class_name("js-username-field")

        passwordBox = self.bot.find_element_by_class_name(


        for x in range(1, self.pages_to_scroll):
                "window.scrollTo(0, document.body.scrollHeight);")
            # for more humanesque scrolling behavior
            time.sleep(random.randrange(10, 30))

        links = []
        for tweet in self.bot.find_elements_by_class_name("tweet"):

        for link in links:
            favBtn = self.bot.find_element_by_class_name("js-actionFavorite")
            except Exception as e:
                # sometimes the clicks fail, not sure why yet.
                print("Error: ", e)
            # How long does it take for a human to read a tweet?
            time.sleep(random.randrange(20, 60))

if __name__ == '__main__':
    bot = TwitterBot(config.USERNAME, config.PASSWORD)

The config.py file requires:
HASHTAG= (don’t actually add the #, its in the search string by default)
and an optional
PAGES= (this defaults to 10 if not present; determines how many tweets you will find on the page)

It took me about an hour or so to get this working and I may extend it out further. I.E. retweet, if the number of likes is over a specific threshold or exclude tweets that contain certain words, etc. Otherwise, this was a very short project and my quest to find something better continues.

Convert strings to HEX Colors

July 4, 2019 | Coding | No Comments

programming code

I’ve been working with charts.js and data being generated from the python backed of the API. Since the data is changing all the time, and the nature of the different queries, I never know how many data sets will be generated to plot.

This makes it difficult to define a specific set of colors to go with the unknown quantity of labels. At first I wrote a function to generate a random color code, but then, on every page reload, everything is a different color.

This of course, makes the experience rough and kinda annoying the longer you’re on the page, pulling different queries. A different approach was in order.

The individual data sets have unique names, which get used for labels. So I thought, why not try to use those names to create hex color codes? Well it turns out its really simple to do.

def convertToHexColor(word):
    word = "AB" + word.upper()
    buff = ""
    for c in word:
        buff += "{}".format(ord(c))
    word = "000000" + hex(int(buff)).lstrip("0x")

    return "#" + word[-6:]

That’s it. It’s small, but I created a github repo for it. If you need it, enjoy.

Django: Extending the User Model

June 22, 2019 | Coding | No Comments

programming code

I love Django. It’s a wonderful framework and ORM. Whether you’re building a full website or a restful api, its got you covered. But if you are like me, you HATE the default user model. This generally comes down to it having both a username & e-mail field; and the e-mail field isn’t even unique.

In my world, the e-mail address is the perfect, unique username. So how do you make this happen?

If your app is out, live in the world all ready, you’re out of luck. But if you are firing up a new one, there are a couple of options. Just DON’T RUN THE MIGRATIONS until the new user model is ready.

First thing after starting the new project is starting the users app.

(ENV) $: python manage.py startapp users

Next we are going to create our new manager and user model in users. DO NOT RUN MIGRATIONS! Did you get that? I’ll tell you when.


from django.contrib.auth.base_user import BaseUserManager
from django.utils.translation import ugettext_lazy as _

class UserManager(BaseUserManager):
    Our custom user model that uses email as the unique username
    def create_user(self, email, password, **extra_fields):
        Create user.
        if not email:
            raise ValueError(_('The Email must be set'))
        email = self.normalize_email(email)
        user = self.model(email=email, **extra_fields)
        return user

    def create_superuser(self, email, password, **extra_fields):
        Create superuser.
        extra_fields.setdefault('is_staff', True)
        extra_fields.setdefault('is_superuser', True)
        extra_fields.setdefault('is_active', True)

        if extra_fields.get('is_staff') is not True:
            raise ValueError(_('Superuser must have is_staff=True.'))
        if extra_fields.get('is_superuser') is not True:
            raise ValueError(_('Superuser must have is_superuser=True.'))
        return self.create_user(email, password, **extra_fields)

When we create our user model, we are going to extend AbstractUser. We can extend BaseAbstractUser as well, but that is a lot more work and I really don’t see a need to ever do that at all.

# users/models.py

from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils.translation import ugettext_lazy as _

from .managers import UserManager

class User(AbstractUser):
    username = None
    email = models.EmailField(_('email address'), unique=True)

    USERNAME_FIELD = 'email'

    objects = UserManager()

    def __str__(self):
        return self.email

Notice the username = None and USERNAME_FIELD = ’email’. We can also add any other fields we want here now. Instead of adding a separate 1 to 1 profile model, we can put things right on the user. Such as phone number, or company name, customer id #, etc.

Now we can add it to our settings.py



AUTH_USER_MODEL = 'users.User'

You can go ahead and create and run the migrations now and run python manage.py createsuperuser to set yourself up.

Now we need Forms and Admin integration.

# users/forms.py
from django.contrib.auth.forms import UserCreationForm, UserChangeForm

from .models import User

class UserCreationForm(UserCreationForm):
    class Meta(UserCreationForm):
        model = User
        fields = ('email',)

class UserChangeForm(UserChangeForm):
    class Meta:
        model = User
        fields = ('email',)
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin

from .forms import UserCreationForm, UserChangeForm
from .models import User

class UserAdmin(UserAdmin):
    add_form = UserCreationForm
    form = UserChangeForm
    model = User
    list_display = ('email', 'is_staff', 'is_active',)
    list_filter = ('email', 'is_staff', 'is_active',)
    fieldsets = (
        (None, {'fields': ('email', 'password')}),
        ('Permissions', {'fields': ('is_staff', 'is_active')}),
    add_fieldsets = (
        (None, {
            'classes': ('wide',),
            'fields': ('email', 'password1', 'password2', 'is_staff', 'is_active')}
    search_fields = ('email',)
    ordering = ('email',)

admin.site.register(User, UserAdmin)

That’s it. You have your own customized user model that you can extend further at will.

Referencing the User model:

There are two ways to get the user model now. You can’t simply import AbstractUser anymore. The preferred method is to use get_user_model() but you can also use settings.AUTH_USER_MODEL. For instance, if you want a foreign key relationship to the user.

# Example
User = get_user_model()
## or User = settings.AUTH_USER_MODEL ## but get_user_model() is preferred

class OurTestModel(models.Model):
     customer = models.ForeignKey(User, on_delete=models.CASCADE)


To make this a simpler process I’ve created a GitHub repo that you can drop into your Django project and get started right away.

Django: Unique Slug for Urls

June 21, 2019 | Coding | No Comments

programming code

Let’s imagine a situation where you want to create, lets say, a blog post or a news article in Django. You want to use take the Title and turn it into a slug that can be used in the URL. Meaning, it needs to be unique, or at least, unique for the date. Depending on the URL parameters you with you use.

You could make it a editable field with a unique property, but its probably better to abstract it away and check recursively for existing matches and increment a number on the end. Simplifying the process.

First, lets define the model as such.

class Article(models.Model):
    title = models.CharField(max_length=255)
    slug = models.SlugField(unique=True, blank=True)
    body = models.TextField()
    published = models.BooleanField(default=True)
    published_on = models.DateTimeField(blank=True)
    updated_on = models.DateTimeField(blank=True)
    author = models.ForeignKey(User, on_delete=models.CASCADE)

Next we will need to override the models save method.

def save(self, *args, **kwargs):
        self.updated_on = timezone.now()
        if not self.id:
            self.published_on = timezone.now()

            # find unique slug
            cnt = 0
            while not self.slug:
                if cnt > 0:
                    # Create a new slug string with the counter
                    slug = "{}-{}".format(slugify(self.title), cnt)
                    slug = slugify(self.title)

                # check to see if a article exists with that slug
                a = Article.objects.filter(slug=slug)
                if not a:
                    # Assign the slug, this will end the loop.
                    self.slug = slug
                    cnt += 1
        super(Article, self).save(*args, **kwargs)

You could easily abstract that loop out into a function.

Python 3: Send SMTP Authenticated Email

February 18, 2014 | Coding | No Comments

programming code

Write enough shell scripts and eventually you will have to write one that sends a email report, in fact now a day’s I find that most shell scripts I write (the scheduled ones anyway) require some kind of notification after they have run. Likewise, if you set up enough mail servers — usually one is enough — you will need a quick easy way to test SMTP.

Below is a python 3 script with smtplib and argparse, which sends a test email using smtp authentication.  It’s short, to the point, and I hope pretty self explanatory for anyone needing to send authenticated email via python.

#!/usr/bin/env python3

import smtplib
import argparse

def testmail(username, password, server, deliveryaddr):
    header = "From: %s\r\nTo: %s\r\nSubject: SMTP Test\r\n\r\nTest Successful" % (username, deliveryaddr)
        server = smtplib.SMTP(server)
        server.login(username, password)
        server.sendmail(username, deliveryaddr, header)
    except Exception as e:
        print("An Error Occurred! %s" % e)
        print("Test Successful")

if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument('-u', '--user', required=True,
            help="Username for SMTP Login")
    parser.add_argument('-p', '--password', required=True,
            help="Password for SMTP Login")
    parser.add_argument('-s', '--server', required=True,
            help="SMTP Server url")
    parser.add_argument('-d', '--dest', required=True,
            help='Destination Email Address')

    args = parser.parse_args()

    testmail(args.user, args.password, args.server, args.dest)

Note: This post was recovered from an old, now defunct, blog.

Learn Python by Building an IRC Logging Bot!

February 6, 2014 | Coding | No Comments

programming code

When I first sat down to learn Python, I was instantly disappointed with all the the available resources, and the fact that almost none of the tutorials I found seemed to work.  In fact I got so frustrated with the language that I shelved the language and went to Ruby for several months.  It wasn’t until I paid a visit to the Windsor Hackforge and met 3 Python fanatics that I decided to take another look.

What I found during my second look at Python was much different, now knowing a couple avid Python programmers I had people who could answer questions and my problems began to become clear.  The source of most of my frustration was caused by the incompatibilities between Python 2 and 3.  I always want to keep my software up to date so it meant using Python 3, but pretty much every example, tutorial, and book I looked at was written in Python 2.

Its incredibly frustrating when you cant even print text to the console!

The other side of the frustration was the boring tutorials,  I mean how many different ways can you generate or manipulate the Fibonacci sequence anyway?  Sure they were full of good information, but when you’ve learned several languages over the years all these tutorials are just a repetition.  I needed something different, interesting, and potentially useful to spend my time working on.

So I set my sights on writing a IRC Bot — How 1995 is that? — but it was fun.  It dawned on me after that, wouldn’t it be great if programming tutorials actually walked you through writing something useful or entertaining? I mean, its one thing to learn that “This is how you declare a variable” or “This is how you write a loop” and completely another thing to see how to implement them in the structure of the program.

I’ve gone back and re-written my original bot, took a lot of it out and tried to smooth out some of the rough edges, and updated it so it runs properly in Python 3.  Without further Ado…

How to write a IRC Logging Bot in Python 3

First things first… lets skip those.  I’ll assume you have Python 3 already installed on your system and know how to use a text editor to create your script.  This is a small program and fits nicely in a single file, so open up your editor of choice and follow along.

At the top of our file we are going to put in our shebang line and import a few packages.

#!/usr/bin/env python3

import sys
import socket
import string
import argparse 

Now in order to handle an instance of an ircbot, I’m going to create an object.  If your not familiar with OOP (Object Oriented Programming) this may be a bit tricky for you, but Python makes this extremely simple.

We define a new class with the class keyword and the class name as such:

class ircbot():

Make a note of that colon. Everything will need to be indented based on it after this.  The next thing we are going to do is create a constructor, which will populate some variables and establish the connection to the IRC server.

 class ircbot():
        def __init__(self, host, port, channel, name, log):
                self.HOST = host
                self.PORT = port
                self.CHANNEL = channel
                self.NICK = name
                self.IDENT = name
                self.REALNAME = name
                self.LOG = log 

The def __init__ you are seeing is a function which is automatically called when the object is created (we will do that later), make note of the “self” in the definition.  Its needed by all class functions and global variables, in order for other functions to have access to them.

This isn’t the full constructor either, the next thing we are going to put in is the connection to the server.  And just so that we don’t get any messy errors thrown at us, I’m also going to introduce exception handling via try-except.  This isn’t in depth, but serves as a good example for your future programs.

   def __init__(self, host, port, channel, name, log):
                # Everything above, after self is a variable we are going
                # to expect every time a instance of ircbot is created
                self.HOST = host
                self.PORT = port
                self.CHANNEL = channel
                self.NICK = name
                self.IDENT = name
                self.REALNAME = name
                self.LOG = log

                        # Any exception thrown here will halt the program, jumping
                        # down to the except block below
                        self.irc = socket.socket()
                        self.irc.connect((self.HOST, self.PORT))

                        # If you have played with Python 2 you will notice a difference
                        # in the below print statement, print is now a function
                        print("Connected to %s\n" % self.HOST)

                        self.irc.send(("NICK %s\r\n" % self.NICK).encode())

                        self.irc.send(("USER %s %s bla :%s\r\n" % (self.IDENT,
                        self.irc.send(("JOIN %s\r\n" % self.CHANNEL).encode())
                        print("Joined %s as %s\n" % (self.CHANNEL, self.NICK))
                except Exception as e:
                        # Exception is generic all built-in, non-system-exiting 
                        # exceptions are derived from this class.
                        # which makes it a simple way to catch whatever and print a 
                        # clean message and exit with a system 
                        # error code

There, if we created a instance of ircbot, with the correct info in host, port, channel, name and log it would connect to the irc server.  It won’t do us much good through, its not reading responses or handling the PING/PONG that the IRC server will send to see if we are still connected and it certainly isn’t going to write all the activity to a log file for us either.  Lets finish this off the ircbot class by creating a “run” function,  it doesn’t have to be run.  In fact it doesn’t even have to be a separate function, we could just as easily write this into the __init__, but I don’t like to put to much into the constructor if I don’t have to.

   def run(self):
                        Main Application Loop

                # creating a tring variable to  dump irc data into
                readbuffer = ""

                # Everything the program does is going to happen here
                # an infinit loop of catching data and parsing it
                while True:
                        readbuffer = readbuffer + self.irc.recv(1024).decode()

                        # We don't need everything the server sends to us
                        # so we split it into a list and then drop the first part
                        temp = readbuffer.split("\n")
                        readbuffer = temp.pop()

                        # Now we are going to step through the rest of the
                        # lines in the list
                        for line in temp:
                                # Here I am going to strip off line endings
                                # and split the string into a list using
                                # white space as the separator. It's not
                                # necessary, but useful for parsing commands
                                # if you grow the bot that way
                                linex = line.rstrip()
                                linex = linex.split()

                                # PING PONG!
                                # When the IRC Server sends us a ping we
                                # best respond, or it will drop us
                                if (linex[0] == "PING"):
                                        self.irc.send(bytes("PONG %s\r\n" % linex[1]))
                                        # And here we handle printing to the screen
                                        # or to a file
                                        if self.LOG == 'stdout':
                                                with open(self.LOG, "a") as log:

Now the IRC Logger Bot is complete.  You could import this and use it in another program, but lets make it runnable as one file.  I’m going to add another function that will use the argparse module to give us some Command line power, as well as give some default settings.

def parseargs():

        parser = argparse.ArgumentParser()
        parser.add_argument('-s', '--server', default='irc.freenode.net',
                             help='DNS address of the IRC server. 
        parser.add_argument('-p', '--port', type=int, default=6667,
                            help='port number of irc server.default=6667')
        parser.add_argument('-c', '--channel', default='#python-unregistered',
                                help='IRC channel to join. default=#python-unregistered')
        parser.add_argument('-n', '--name', default='irc_logger_bot',
                                help='how the bot will be identified in the channel. default=irc_logger_bot')
        parser.add_argument('-o', '--output', default='stdout',
                                help='file to write log to. default=stdout')

Most of these parser lines are the same, but take an extra look at the -p/–port option. It has one more configuration flag than the others.  I’m sure you can figure that out on your own, Python makes coding easy.

Once we add our final lines to this project, you will be able to run this script with a -h or –help and get a shiny helpful output like:

usage: tutorial_bot.py [-h] [-s SERVER] [-p PORT] [-c CHANNEL] [-n NAME]
                       [-o OUTPUT]

optional arguments:
  -h, --help            show this help message and exit
  -s SERVER, --server SERVER
                        dns address of the irc server.
  -p PORT, --port PORT  port number of irc server. default=6667
  -c CHANNEL, --channel CHANNEL
                        irc channel to join. default=#python-unregistered
  -n NAME, --name NAME  how the bot will be identified in the channel.
  -o OUTPUT, --output OUTPUT
                        file to write log to. default=stdout

Finally, we are going to make this run if executed from the command line by using if __name__ == ‘__main__’:

if __name__ == '__main__':

        # Get the options from the parser
        opt = parseargs()

        # Create bot object and run it!
        bot = ircbot(opt.server, opt.port, opt.channel, opt.name, 

What does if __name__ == ‘__main__’ do?

Well when the python interpreter runs it creates some special variables such as __name__, if a file is executed it is assigned the name __main__.  using __name__ == ‘__main__’ in this way allows us to cleanly either import the file as a module, or execute specific code.

And that it, program complete.  I hope you have found this post both helpful and useful.  

The full code can be found at https://github.com/wsimmerson/ircbot/blob/master/tutorial_bot.py

Thanks for reading!

Note: This post was recovered from an old, now defunct, blog.